test(perf): add the idle-window T0 calibration reports

testdata/perf/t0-idle-window/README.md

@@ -0,0 +1,27 @@
+# Idle-window T0 calibration (issue #3294)
+
+The per-run wall-clock reports behind the 2026-11-07 idle-window rerun on
+issue #3294, committed at the maintainer's request as the calibration data
+for the benchmark baseline ("please do commit the 24 per-run JSON reports —
+they are the calibration data for the benchmark baseline").
+
+- 24 runs: `{cpu, cuda}` x N in {10, 30, 60, 120} x 4 runs per cell; run 0
+  is the cold run, discarded from the minimum per the maintainer's
+  2026-11-04 guidance; the minimum is taken over runs 1-3.
+- `summary.json` — per-cell min-of-3 walls, the matched run's `fps` and
+  `wall_fps`, per-cell nvidia-smi utilization gates, and the OLS fits
+  (primary over N >= 30, secondary over all four points).
+- `REPORT.md` — the environment statement, the min-of-3 table, both fits,
+  and the load-throttling discriminator verdict.
+- `measure.py` — the generator: rc.3 `vmafx.exe`, 1920x1080 yuv420p8 from
+  the t1_frames fixture pair, full-process wall via `time.perf_counter`,
+  every run's JSON report kept.
+
+Headline (RTX 5080, `CUDA_VISIBLE_DEVICES=1`): CPU T0 = 31.1 ms at
+21.305 ms/frame (R^2 = 1.0000); CUDA T0 = 403.1 ms at 6.300 ms/frame
+(R^2 = 0.9788), crossing near N = 21. On the quiet GPU the scoring window
+runs at 127-153 fps for CUDA against ~47 for CPU — the loaded-run parity
+(33.4 vs 33.1 fps) was external-load throttling, not a shared in-loop cost.
+The machine is quieter, not idle: utilization gates read ~22-26 % with
+occasional 44-58 % compositor spikes; even the worst timed CUDA run
+(92.7 fps, taken during a spike) stays ~2x CPU.

testdata/perf/t0-idle-window/REPORT.md

@@ -0,0 +1,114 @@
+# #3294 idle-window full matrix — RTX 5080, 2026-11-07
+
+## Environment
+
+- Machine: Windows 10.0.26300 x64. Two NVIDIA GPUs: device 0 = NVIDIA GeForce
+  RTX 4060 Laptop GPU, device 1 = **NVIDIA GeForce RTX 5080** (the device under
+  test; CUDA runs pinned via `CUDA_VISIBLE_DEVICES=1`).
+- Exe provenance: `C:\PLAN\VMAF\verify\vmafx-install\bin\vmafx.exe` — **rc.3**
+  (static default model), 291,737 bytes, dated Nov 3 15:21.
+- Input: `C:\PLAN\VMAF\baseline\t1_frames\ref_1080p8_{N}.yuv` /
+  `dis_1080p8_{N}.yuv` — 1920x1080 yuv420p 8-bit raw, N in {10, 30, 60, 120}.
+- Method: {cpu, cuda} x N, 4 runs per cell; run 0 cold discarded, min over
+  runs 1-3; full-process wall time via `time.perf_counter`; per-run JSON kept
+  in this directory. Harness: `C:\PLAN\VMAF\verify\tmp\t0_idle_matrix.py`,
+  console log `C:\PLAN\VMAF\verify\tmp\t0_idle_matrix_run.log`.
+
+### Utilization situation (nvidia-smi `-i 1`, 1 s apart, 4 samples per gate)
+
+Desktop-compositor load is present and visible; the baseline sits ~22-26 %
+with occasional spikes into the 40-60 % band. Gate samples verbatim:
+
+- pre-run gate: `[23, 23, 24, 23]`
+- cpu N=10: `[24, 56, 22, 22]`
+- cpu N=30: `[23, 23, 22, 23]`
+- cpu N=60: `[22, 58, 23, 23]`
+- cpu N=120: `[23, 24, 58, 23]`
+- cuda N=10: `[24, 24, 22, 44]`
+- cuda N=30: `[24, 22, 23, 58]`
+- cuda N=60: `[23, 26, 55, 26]`
+- cuda N=120: `[24, 24, 25, 23]`
+
+So: baseline ~23 %, spikes 44-58 % (7 of 36 samples). This is residual desktop
+compositor activity, not a dedicated compute job; it is quantified here so the
+numbers below can be read with that noise floor in mind.
+
+## Min-of-3 table (runs 1-3; run 0 cold discarded)
+
+"wall" = min wall ms over runs 1-3; "fps"/"wall_fps" are the matched (min-wall)
+run's in-window values from the JSON report; "wall/s" = N / wall_s
+(full-process rate).
+
+| backend | N   | wall ms | fps (in-window) | wall_fps (in-window) | wall/s (N/wall_s) |
+|---------|-----|---------|-----------------|----------------------|-------------------|
+| cpu     | 10  | 245.6   | 47.62           | 47.80                | 40.72             |
+| cpu     | 30  | 671.9   | 47.24           | 47.31                | 44.65             |
+| cpu     | 60  | 1307.1  | 47.36           | 47.35                | 45.90             |
+| cpu     | 120 | 2588.6  | 47.11           | 47.11                | 46.36             |
+| cuda    | 10  | 378.6   | 149.25          | 149.16               | 26.41             |
+| cuda    | 30  | 560.0   | 131.00          | 131.13               | 53.57             |
+| cuda    | 60  | 829.2   | 126.85          | 126.72               | 72.36             |
+| cuda    | 120 | 1143.0  | 137.46          | 137.59               | 104.99            |
+
+Per-run detail (wall ms): cpu N=10: 266.0/245.6/252.5; cpu N=30:
+675.0/688.8/671.9; cpu N=60: 1307.1/1318.9/1366.6; cpu N=120:
+2770.5/2588.6/2654.3. cuda N=10: 391.9/416.7/378.6; cuda N=30:
+629.1/632.0/560.0; cuda N=60: 1014.9/857.7/829.2; cuda N=120:
+1224.7/1487.0/1143.0.
+
+## OLS fits T(N) = T0 + N*t_frame (on min-of-3 wall ms)
+
+| backend | fit          | T0 ms  | t_frame ms | R²     |
+|---------|--------------|--------|------------|--------|
+| cpu     | primary N≥30 | 31.1   | 21.305     | 1.0000 |
+| cpu     | all 4 points | 32.0   | 21.296     | 1.0000 |
+| cuda    | primary N≥30 | 403.1  | 6.300      | 0.9788 |
+| cuda    | all 4 points | 349.8  | 6.870      | 0.9787 |
+
+CPU is almost perfectly linear with a ~31 ms fixed cost and 21.3 ms/frame.
+CUDA has a large fixed cost (~350-400 ms: CUDA context/extractor setup) but a
+6.3-6.9 ms/frame slope — i.e. in-window ~145-159 fps vs CPU's ~47 fps. The
+crossover (fit) is at N ≈ 21: below it the CUDA cell is setup-dominated.
+
+## Discriminating analysis
+
+The maintainer's two hypotheses for the earlier loaded-run result (CPU
+33.1 fps vs CUDA 33.4 fps in-window — statistically indistinguishable):
+
+- (a) external GPU load throttled the CUDA extractors down to the CPU rate
+  inside the scoring window;
+- (b) something shared (input read/convert) sits inside the loop and caps both
+  backends at the same rate.
+
+On the now-quiet GPU the CUDA in-window rates are 126-153 fps across
+N in {30, 60, 120} (min-of-3 matched-run fps 131.0 / 126.85 / 137.46), i.e.
+**~2.7-2.9x the CPU's ~47 fps** — nowhere near equal. If hypothesis (b) were
+true, a shared per-frame serial cost would keep the CUDA in-window rate
+pinned at the CPU rate regardless of load; instead the CUDA slope
+(6.3 ms/frame) is ~3.4x steeper on the fast side of the CPU's 21.3 ms/frame,
+and the gap widens with N exactly as a GPU-vs-CPU compute-rate difference
+predicts. The N=10 CUDA cell (26.4 wall/s < CPU 40.7 wall/s) is explained by
+the ~400 ms fixed setup, not by a per-frame cap.
+
+**Verdict: hypothesis (a) is supported — strongly.** Under prior load the CUDA
+extractors were being crowded down to ~33 fps in-window; with that load gone
+they run at ~127-153 fps in-window while CPU stays at its usual ~47 fps
+(47.1-47.8 in this window vs 33.1 under the loaded run — the CPU number itself
+moved too, so part of the earlier equality was load on the host as well, but
+the decisive ~3x CUDA recovery only happens on the GPU side).
+
+Honest caveats: the machine is *quieter*, not idle — the utilization gates
+above show a ~22-26 % compositor baseline with 44-58 % spikes (7 of 36
+samples), and CUDA run-to-run spread is visible (e.g. cuda N=60 runs
+1014.9/857.7/829.2 ms; cuda N=120 1224.7/1487.0/1143.0 ms), consistent with
+that residual load grazing the measurements. The R² of the CUDA fit (0.979)
+reflects the same noise. Even taking the *worst* timed run of every CUDA cell
+(92.74 fps at N=60), CUDA remains ~2x the CPU rate — the conclusion does not
+depend on cherry-picking.
+
+## Artifacts
+
+- `C:\PLAN\VMAF\verify\tmp\t0-idle-matrix\summary.json`
+- Per-run JSON reports: `C:\PLAN\VMAF\verify\tmp\t0-idle-matrix\n{N}-{backend}-{run}.json` (24 files)
+- Harness log: `C:\PLAN\VMAF\verify\tmp\t0_idle_matrix_run.log`
+- Harness: `C:\PLAN\VMAF\verify\tmp\t0_idle_matrix.py`

testdata/perf/t0-idle-window/measure.py

@@ -0,0 +1,148 @@
+# -*- coding: utf-8 -*-
+"""#3294 idle-window full matrix on the RTX 5080.
+
+Cells: {cpu, cuda} x N in {10, 30, 60, 120}. Four runs per cell: run 0 is the
+cold run (kept on disk, discarded from the minimum, per the maintainer's
+2026-11-04 guidance), the minimum is taken over runs 1-3. Before each cell a
+4-sample utilization gate is recorded (nvidia-smi, GPU 1 = 5080). Every run's
+JSON report is kept; per run we extract frames count (asserted == N), fps,
+wall_fps, backend_used, feature_backends. OLS fit T(N) = T0 + N*t_frame per
+backend on the min-of-3 wall times, primary fit over N>=30, secondary over all
+four points. Read-only wrt the repo; artifacts land next to this script.
+Usage: python -I t0_idle_matrix.py
+"""
+from __future__ import annotations
+
+import json
+import os
+import statistics
+import subprocess
+import time
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+OUT = HERE / "t0-idle-matrix"
+OUT.mkdir(exist_ok=True)
+
+EXE = r"C:\PLAN\VMAF\verify\vmafx-install\bin\vmafx.exe"
+FRAMES_DIR = Path(r"C:\PLAN\VMAF\baseline\t1_frames")
+NS = (10, 30, 60, 120)
+
+ENV_BASE = {
+    "SYSTEMROOT": os.environ["SYSTEMROOT"],
+    "PATH": r"C:\PLAN\VMAF\verify\vmafx-install\bin;C:\msys64\mingw64\bin;"
+            + os.environ["SYSTEMROOT"] + r"\System32",
+    "CUDA_DEVICE_ORDER": "PCI_BUS_ID",
+    "CUDA_VISIBLE_DEVICES": "1",
+}
+
+
+def util_sample(k: int = 4) -> list[int]:
+    vals = []
+    for _ in range(k):
+        r = subprocess.run(["nvidia-smi", "-i", "1", "--query-gpu=utilization.gpu",
+                            "--format=csv,noheader,nounits"],
+                           capture_output=True, text=True, timeout=30)
+        try:
+            vals.append(int(r.stdout.strip()))
+        except ValueError:
+            vals.append(-1)
+        time.sleep(1.0)
+    return vals
+
+
+def run_cell(backend: str, n: int) -> list[dict]:
+    ref = FRAMES_DIR / f"ref_1080p8_{n}.yuv"
+    dis = FRAMES_DIR / f"dis_1080p8_{n}.yuv"
+    runs = []
+    for i in range(4):
+        report = OUT / f"n{n}-{backend}-{i}.json"
+        cmd = [EXE, "-r", str(ref), "-d", str(dis), "-w", "1920", "-h", "1080",
+               "-p", "420", "-b", "8", "--backend", backend, "-o", str(report),
+               "--json"]
+        t0 = time.perf_counter()
+        proc = subprocess.run(cmd, capture_output=True, text=True, env=ENV_BASE,
+                              timeout=600)
+        wall_ms = (time.perf_counter() - t0) * 1000.0
+        if proc.returncode != 0:
+            raise RuntimeError(f"{backend} n={n} run{i}: EXIT={proc.returncode}\n"
+                               f"{proc.stderr[:400]}")
+        data = json.loads(report.read_text(encoding="utf-8"))
+        frames = data.get("frames", [])
+        assert len(frames) == n, f"expected {n} frames, got {len(frames)}"
+        rec = {
+            "run": i,
+            "wall_ms": round(wall_ms, 1),
+            "fps": data.get("fps"),
+            "wall_fps": data.get("wall_fps"),
+            "backend_used": data.get("backend_used"),
+            "feature_backends": [(f["extractor"], f["backend"])
+                                 for f in data.get("feature_backends", [])],
+            "n_frames": len(frames),
+        }
+        runs.append(rec)
+        print(f"  {backend} N={n} run{i}: wall={wall_ms:.0f}ms fps={rec['fps']} "
+              f"wall_fps={rec['wall_fps']} backend={rec['backend_used']}")
+        time.sleep(1.0)
+    return runs
+
+
+def ols(points: list[tuple[float, float]]) -> tuple[float, float, float]:
+    """Return (T0, t_frame, R2) for T(N) = T0 + N*t_frame."""
+    xs = [p[0] for p in points]
+    ys = [p[1] for p in points]
+    n = len(xs)
+    mx, my = statistics.fmean(xs), statistics.fmean(ys)
+    sxx = sum((x - mx) ** 2 for x in xs)
+    sxy = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
+    slope = sxy / sxx
+    intercept = my - slope * mx
+    ss_tot = sum((y - my) ** 2 for y in ys)
+    ss_res = sum((y - (intercept + slope * x)) ** 2 for x, y in zip(xs, ys))
+    r2 = 1.0 - ss_res / ss_tot if ss_tot else 1.0
+    return intercept, slope, r2
+
+
+def main() -> int:
+    gate = util_sample()
+    print(f"pre-run util gate (5080, 4 samples): {gate}")
+    results = {"pre_run_util": gate, "cells": {}, "notes": {
+        "machine": "Windows 10.0.26300, RTX 5080 (device 1 via CUDA_VISIBLE_DEVICES)",
+        "exe": "rc.3 vmafx.exe (static default model)",
+        "input": "1920x1080 yuv420p 8-bit raw (t1_frames)",
+        "method": "4 runs/cell: run0 cold discarded, min over runs 1-3; "
+                  "full-process wall via time.perf_counter",
+    }}
+    for backend in ("cpu", "cuda"):
+        for n in NS:
+            gate = util_sample()
+            print(f"cell {backend} N={n} util gate: {gate}")
+            runs = run_cell(backend, n)
+            results["cells"][f"{backend}-n{n}"] = {"util_gate": gate, "runs": runs}
+    # fits on min-of-3 (runs 1..3)
+    for backend in ("cpu", "cuda"):
+        pts = []
+        pts_all = []
+        for n in NS:
+            runs = results["cells"][f"{backend}-n{n}"]["runs"]
+            best = min(runs[1:], key=lambda r: r["wall_ms"])
+            pts_all.append((n, best["wall_ms"]))
+            if n >= 30:
+                pts.append((n, best["wall_ms"]))
+        t0, slope, r2 = ols(pts)
+        t0a, slopea, r2a = ols(pts_all)
+        results[f"fit_{backend}"] = {
+            "primary_Nge30": {"T0_ms": round(t0, 1), "t_frame_ms": round(slope, 3),
+                              "R2": round(r2, 4)},
+            "all_points": {"T0_ms": round(t0a, 1), "t_frame_ms": round(slopea, 3),
+                           "R2": round(r2a, 4)},
+        }
+        print(f"fit {backend}: N>=30 T0={t0:.1f}ms t={slope:.3f}ms R2={r2:.4f} | "
+              f"all T0={t0a:.1f}ms t={slopea:.3f}ms R2={r2a:.4f}")
+    (OUT / "summary.json").write_text(json.dumps(results, indent=2), encoding="utf-8")
+    print("saved", OUT / "summary.json")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())