diff --git a/clean/k/FORMATS.md b/clean/k/FORMATS.md index 8e9ff98e..c2916d3c 100644 --- a/clean/k/FORMATS.md +++ b/clean/k/FORMATS.md @@ -113,24 +113,33 @@ The per-frame runtime runs `KFrame`'s frame on its HIP stream (the interop's Fra | Step | Kernels | Notes | |---|---|---| | input copy (option `input_copy=1`) or the compact input pass (interop default for K) | none: the K kernels read the copies | the game's RGBA16F colour (`KIO_C16`, not divided by π: model K applies κ), R32F depth and FP16 motion of 2 or 4 channels (`KIO_M16`, m = ms · r + mo on load: FP32 multiply then add, as `input_convert` computes it) in their 256-byte pitched rows; `KFrameIn.mch`, `rmsx`, `rmsy`, `rmox`, `rmoy`; the same values as the FP32 slot of the input pass, so the same output bits (GPU test) | -| exposure | `exposure_auto_diet` (`exposure.co`, spec 07, when auto-exposure is asked; `exposure_auto_diet_c16` on the RGBA16F copy), `k_expo_scalars` (`k_io.co`) | the exposure pair `[e, e_prev, last e]` of spec 10 (value, the game's texel on the GPU, or spec 07), then `xs` | +| exposure | `exposure_auto_diet` (`exposure.co`, spec 07, HDR only, when the game asked for auto-exposure at creation or the front end asks this frame; `exposure_auto_diet_c16` on the RGBA16F copy), `k_expo_scalars` (`k_io.co`) | E_cur, E_prev, E_aux of spec 42 draft 3 section 4 (the game's value or texel as E_cur, else 1; the measured value as E_aux), then `xs` | | preparation | `k_out_color`, then `k_in_tiled` when supported; otherwise `k_in_color` plus `k_in_base`/`k_in_reset` | `KFrameIn.flags` = [HDR] from `WgRtConfig.hdr` (front end `WG_CREATE_HDR`), always [MVlo], [inv], reset | | network | the eleven `k_stage_*` / `k_stagew_*` launches of `k_net.py` variant "fused" | window offsets `k_stage_offsets(f, net/window_offsets)`; the G2 d0 also emits hidden history and, by default, the compact pixel head instead of the head map (`k_stage2_d0_hp`, `k_stage2h_d0_hp`) | | output stage | `k_out_pixel_c` (compact pixel head, default) or `k_out_pixel` (runtime option `k_head=full`) | game output omode 1 (FrameIo.out_half_pitch > 0) or 2 (interop dense RGB) | -`k_expo_scalars(ex, src, xs, KExpoArgs)` (one lane): with `pair` = 1 it first updates the pair as `exposure_pair` -(e = src[0] or `value`, e_prev = e on the first frame, else the previous e); then E_cur = e, E_prev = e_prev (E_cur on -a reset frame), E_aux = E_cur, each rounded to FP16 with 0 → 1, and the FP32 operations of `k_io.scalars()` with -e₁ = e₂ = 1 and `inv_pi` = RN32(1/π) (π: the pre-exposure, 1 if missing, ≤ 0 or not finite). The kernels then read -the same values the host would give for that exposure (bit-identical to `KFrame` given that E_cur). -`KExpoArgs` = {float value; int pair, use_src, first, hdr, reset; float e1, e2, inv_pi} (36 bytes). - -State: Hc, L, Hh double-buffered, zero at create; reset on the game's flag, the first frame and a render-size change; +`k_expo_scalars(st, src, measured, xs, KExpoArgs)` (one lane), spec 42 draft 3 section 4 (the host rule of +`clean/k/reference/exposure.py`): [HDR] E_cur = `value` (`src_mode` 0: the front end's exposure value, 1 when the +game gives none), src[0] (`src_mode` 1: the game's exposure texel) or 1 (`src_mode` 2: no exposure from the game, +auto-exposure asked); the auto-exposure value never takes its place. E_aux = measured[0] when auto-exposure ran this +frame (`aux` = 1), else E_cur. E_prev = st[3] (the previous frame's E_cur), E_cur on the first frame and on a reset. +[LDR] all 1. The raw values go to st = [E_cur, E_prev, E_aux, E_cur for the next frame]; then each is rounded to FP16 +with 0 → 1, and the FP32 operations of `k_io.scalars()` with e₁ = e₂ = 1 (the front-end interface carries no +exposure scale) and `inv_pi` = RN32(1/π) (π: the pre-exposure, 1 if missing, ≤ 0 or not finite): κ = e₁E_cur/π, +λ = e₂E_aux/(e₁E_cur), ρ = e₁E_cur/(e₁E_prev). With auto-exposure and no texture, E_cur = E_prev = 1 and the measured +value acts only through λ. The kernels then read the same values the host would give for those exposures +(bit-identical to `KFrame` given the same E_cur, E_aux). Runtime option `k_expo_rule=v051` (A/B only): the 0.5.1 rule +(the measured value as E_cur, E_aux = E_cur). `KExpoArgs` = {float value; int src_mode, aux, first, hdr, reset; float +e1, e2, inv_pi} (36 bytes). + +State: Hc, L, Hh double-buffered, zero at create; reset on the game's flag and the first frame only (a render-size +change keeps the history: the histories are at output and base resolution; model M's spec 10 size rule is not K's); frame index f = 0 on a reset frame, else the previous f + 1; `WgRtFrame.reset = 2` (a parked upscaler taken over by a new context) is a reset that also starts the host frame state over as at create (frame counter, so the first-frame exposure; f; history parity; last size; jitter), so the frames from it on equal a new runtime's; filter-shape sets of all eight levels uploaded once at create (the frame takes level `k_filter_level(rw, rh, W, H)`); `exp2tab` allocated and zero (the GPU uses v_exp_f16). -s = ½, base `k_base_size(W, H, ½)`. Buffers for `wg_rt_buffer`: `exposure.pair`, `k.scalars` (xs), `k.x0`, `k.N` +s = ½, base `k_base_size(W, H, ½)`. Buffers for `wg_rt_buffer`: `exposure.pair` ([E_cur, E_prev, E_aux] of the last +frame, raw), `exposure.measured` (the last auto-exposure value), `k.scalars` (xs), `k.x0`, `k.N` (network output, the 40-channel head map: only with the runtime option `k_head=full`; the default frame writes the compact pixel head instead, buffer `k.Np`, and asking for `k.N` fails with a message naming the option; `k.Np` in turn fails with `k_head=full`), diff --git a/clean/k/README.md b/clean/k/README.md index 1c59fa4d..8f104911 100644 --- a/clean/k/README.md +++ b/clean/k/README.md @@ -89,7 +89,10 @@ one rounding instead of two, and 46 of 3.7 M colour-history pixels differed by 1 ### Whole frame: `clean/k/runner/k_frame.py` `KFrame(dev, func, output_size, max_render_size, params, flags)` keeps the histories (double-buffered) and the frame -index of spec 44 (0 on a reset: the game's flag, the first frame, or a render-size change). `frame(x)` returns the +index of spec 44 (0 on a reset: the game's flag or the first frame; a render-size change keeps the history). The frame +dict gives E_cur, E_aux, e1, e2, or the inputs of the exposure rule of spec 42 draft 3 section 4 +(`clean/k/reference/exposure.py`: `game_exposure` (None: the game passes none), `measured` (the auto-exposure value +when it runs), `exposure_scale`), which the per-frame runtime applies on the GPU. `frame(x)` returns the image, and `time_frame(x)` the GPU ms of the input stage, network, output stage and total. `params` are the converter's tensors (`k_model.load_raw`) including `net/window_offsets`. @@ -184,6 +187,14 @@ Model K ships in the same `wronggpu_runtime.dll` as model M; nothing changes for compact one), `input_copy=1` (copies instead of the input pass where the formats allow), `timing_log=1`, `k_diag=input2|copyin` (timing diagnostics only), `k_head=full` (the 40-channel head map between d0 and the output stage instead of the compact pixel head; needed to read the buffer `k.N`). + The input stage is exact by default: the input-stage kernels from `k_io_exact.co` (every operation correctly + rounded: bit for bit spec 42 and `KFrame(exact_input=True)`); the output colour taps then come from `k_out_color`. + `k_in_exact=1` names the default; `k_in_exact=0` is the same as `k_fast_in=1`, which selects `k_io.co`'s GPU log2 / + 2^x / reciprocal estimates instead (not bit-exact, test use). `k_in_exact=1` is refused with `k_fast=1` or + `k_fast_in=1`. + Exposure (spec 42 draft 3 section 4): E_cur = the game's exposure texture or value, 1 if it passes none; the + auto-exposure value (spec 07, run when the game asks for it at creation, HDR only) is E_aux and acts only through + λ. `k_expo_rule=v051` (A/B only) restores the 0.5.1 rule (the measured value as E_cur and E_aux). In game (Cyberpunk 2077 benchmark, 1440p Quality, RX 9070 XT; the project's measurements): model K GPU time about 1.8 ms per frame (`fused`; 1.72 ms with `fused16`), the game at ~120 fps against ~117 fps with model M @@ -472,7 +483,7 @@ The 1.3 ms objective remains unmet. See ## Reset-frame input path -History resets (first frame, explicit reset, or render-size change) select `k_in_reset` +History resets (first frame or the game's reset flag; not a render-size change) select `k_in_reset` after the unchanged colour prepass. Spec 42 sections 6.1, 6.5 and 7 require each slot's channels to be `(current, current, 0, 0)` on a reset. The kernel therefore reads only the current own-sample channel; it skips history, motion/depth, colour diff --git a/clean/k/kernels/k_io.hip b/clean/k/kernels/k_io.hip index 6fe38265..7159dd9f 100644 --- a/clean/k/kernels/k_io.hip +++ b/clean/k/kernels/k_io.hip @@ -1874,45 +1874,49 @@ WG_KERNEL_N(k_in_tiled_color_uniform_f, 256)(KInColorUniformParams P0) { // Host-side arguments of k_expo_scalars (must match KExpoArgs in clean/k/runtime/runtime_k.cpp and k_io.py). struct KExpoArgs { - float value; // pair mode: this frame's exposure value (when use_src = 0) - int pair; // 1: update the exposure pair first (as kernels/exposure.hip exposure_pair: e = src[0] or value, - // e_prev = e on the first frame, else the previous e); 0: the pair is already written - // (auto-exposure: exposure_auto_diet) - int use_src, first; + float value; // E_cur when src_mode = 0: the exposure value the game passes + int src_mode; // where E_cur comes from: 0 value, 1 src[0] (the game's exposure texel), 2 none (E_cur = 1) + int aux; // 1: E_aux = measured[0] (spec 07 auto-exposure ran this frame); 0: E_aux = E_cur + int first; // the first frame of the runtime (E_prev = E_cur) int hdr, reset; - float e1, e2; // exposure factors of spec 42 section 4 (1 unless the game supplies an exposure scale) + float e1, e2; // exposure factors of spec 42 section 4 (the game's exposure scale, else 1) float inv_pi; // RN32(1 / pi), pi the pre-exposure (host) }; static_assert(sizeof(KExpoArgs) == 36, "KExpoArgs"); -// Spec 42 section 4 / spec 43 section 3.7 from the exposure pair ex = [e, e_prev, last e] (the model M pair buffer of -// the runtime, spec 07 / spec 10 section 3): E_cur = e, E_prev = e_prev (E_cur on a reset), E_aux = E_cur (the -// runtime has one exposure value per frame: spec 07's result, the game's texel or value), each read as FP16 (nearest -// even) with 0 -> 1; then exactly the FP32 operations of clean/k/runner/k_io.py scalars() and out_uniforms() -// (kinv = RN(1/kappa); FP32 division is correctly rounded on the GPU). Writes xs = (kappa, rho, lambda, kinv), which -// the stage kernels read through KFrameIn.xs. One lane. +// Spec 42 draft 3 section 4 / spec 43 section 3.7. Which value is which (HDR): E_cur = the game's exposure texel or +// value, 1 if the game passes none (the auto-exposure value never takes its place); E_aux = the auto-exposure value +// when it runs, else E_cur; E_prev = the previous frame's E_cur, E_cur itself on the first frame and on a reset. +// LDR: every value and factor 1. The state st = [E_cur, E_prev, E_aux, E_cur kept for the next frame] holds the raw +// values; each is then read as FP16 (nearest even) with 0 -> 1, and the scalars are exactly the FP32 operations of +// clean/k/runner/k_io.py scalars() and out_uniforms() (kinv = RN(1/kappa); FP32 division is correctly rounded on the +// GPU). Writes xs = (kappa, rho, lambda, kinv), which the stage kernels read through KFrameIn.xs. One lane. WG_FN float kx_ev(float e) { const float h = (float)kh_round(e); return h == 0.0f ? 1.0f : h; } -WG_KERNEL_N(k_expo_scalars, 32)(float* ex, const float* src, float* xs, KExpoArgs A) { +WG_KERNEL_N(k_expo_scalars, 32)(float* st, const float* src, const float* measured, float* xs, KExpoArgs A) { if (wg_lane_id() != 0) return; - if (A.pair) { - const float e = A.use_src ? src[0] : A.value; - const float e_prev = A.first ? e : ex[2]; - ex[0] = e; - ex[1] = e_prev; - ex[2] = e; + float Ecr = 1.0f, Epr = 1.0f, Ear = 1.0f; + if (A.hdr) { + Ecr = A.src_mode == 1 ? src[0] : A.src_mode == 0 ? A.value : 1.0f; + Ear = A.aux ? measured[0] : Ecr; + Epr = (A.first || A.reset) ? Ecr : st[3]; } - const float Ec = kx_ev(ex[0]), Ep = kx_ev(A.reset ? ex[0] : ex[1]), Ea = Ec; - const float ce = A.e1 * Ec; + st[0] = Ecr; + st[1] = Epr; + st[2] = Ear; + st[3] = Ecr; + const float e1 = A.hdr ? A.e1 : 1.0f, e2 = A.hdr ? A.e2 : 1.0f; + const float Ec = kx_ev(Ecr), Ep = kx_ev(Epr), Ea = kx_ev(Ear); + const float ce = e1 * Ec; float kappa = 1.0f, rho = 1.0f; if (A.hdr) { kappa = A.inv_pi * ce; if (kappa == 0.0f) kappa = 1.0f; - rho = ce / (A.e1 * Ep); + rho = ce / (e1 * Ep); } - const float lam = (A.e2 * Ea) / ce; + const float lam = (e2 * Ea) / ce; xs[0] = kappa; xs[1] = rho; xs[2] = lam; diff --git a/clean/k/runner/k_frame.py b/clean/k/runner/k_frame.py index 317779c0..65b6074b 100644 --- a/clean/k/runner/k_frame.py +++ b/clean/k/runner/k_frame.py @@ -2,11 +2,15 @@ """Model K, one whole frame on one device: input stage (spec 42) -> the eleven network stages (spec 41 / 44) -> output stage (spec 43), with the history images kept between frames (spec 42 / 43 inputs Hc, Hh, L, double-buffered) and the frame index of spec 44 (0 on every reset frame, then +1 per frame; it selects the window offsets of m, e4, d4 -from the converted model's table). +from the converted model's table). A reset frame: the game's reset flag or the first frame; a render-size change keeps +the history (spec 44; model M's spec 10 rule of a reset on a size change does not apply to K). E_prev: the previous +frame's E_cur, E_cur on a reset frame (spec 42 draft 3 section 4). kf = KFrame(dev, func, output_size, max_render_size, params, flags) # flags: hdr, mv_lowres, depth_inverted out = kf.frame(x) # x: color, motion, depth, render_size, jitter, mv_scale, mv_offset, pre_exposure, - # reset, E_cur, E_aux (FP16 values), e1, e2 -> {"out": [H, W, 4] RGBA float32} + # reset, E_cur, E_aux (FP16 values), e1, e2 -> {"out": [H, W, 4] RGBA float32}; + # or instead of E_cur, E_aux, e1, e2 the inputs of the exposure rule + # (clean/k/reference/exposure.py): game_exposure, measured, exposure_scale ms = kf.time_frame(x) # {"input", "network", "output", "total"}: median GPU ms reference_frame(state, x, params, flags) runs the same frame with the NumPy references (state: the histories and the @@ -17,6 +21,7 @@ import numpy as np from clean.k.reference import instage, network, outstage +from clean.k.reference.exposure import apply_rule from clean.k.reference.layout import base_size from clean.k.reference.outstage import ceil16 from clean.k.runner import k_io @@ -93,12 +98,12 @@ def _frame_dict(self, x): rw, rh = (int(v) for v in x["render_size"]) if rw > self.max_rw or rh > self.max_rh: raise ValueError(f"render size {(rw, rh)} above the maximum {(self.max_rw, self.max_rh)}") - reset = bool(x.get("reset")) or self.first or self.last_render != (rw, rh) - d = dict(x) + reset = bool(x.get("reset")) or self.first + d = apply_rule(x, self.flags["hdr"]) # E_prev: the previous frame's E_cur, unless the frame gives it (spec 49's test sequences); E_cur on a reset e_prev = x.get("E_prev") if x.get("E_prev") is not None else self.E_prev d.update(self.flags, output_size=(self.W, self.H), base=(self.Bw, self.Bh), reset=reset, s=self.s, - E_prev=x["E_cur"] if reset or e_prev is None else e_prev) + E_prev=d["E_cur"] if reset or e_prev is None else e_prev) return d def _launches(self, d, full_head=False): @@ -272,16 +277,16 @@ def new_reference_state(output_size, base=None): def reference_frame(state, x, params, flags): """The NumPy references for one frame; state (new_reference_state) is updated like KFrame's.""" rw, rh = (int(v) for v in x["render_size"]) - reset = bool(x.get("reset")) or state["first"] or state["last_render"] != (rw, rh) - d = dict(x) + reset = bool(x.get("reset")) or state["first"] # the game's flag or the first frame (not a render-size change) + d = apply_rule(x, bool(flags.get("hdr", False))) d.update({k: bool(flags.get(k, False)) for k in ("hdr", "mv_lowres", "depth_inverted")}, output_size=state["output_size"], base=state["base"], reset=reset, - E_prev=x["E_cur"] if reset or (x.get("E_prev") is None and state["E_prev"] is None) + E_prev=d["E_cur"] if reset or (x.get("E_prev") is None and state["E_prev"] is None) else (x["E_prev"] if x.get("E_prev") is not None else state["E_prev"])) f = 0 if reset else state["f"] x0 = instage.input_stage(dict(d, Hc=state["Hc"], Hh=state["Hh"])) outs = network.run_network(x0, params, f) N = outs["d0"]["head"] o = outstage.output_stage(dict(d, N=N, L=state["L"], Hc=state["Hc"])) - state.update(Hc=o["Hc"], L=o["L"], Hh=o["Hh"], f=f + 1, first=False, E_prev=x["E_cur"], last_render=(rw, rh)) + state.update(Hc=o["Hc"], L=o["L"], Hh=o["Hh"], f=f + 1, first=False, E_prev=d["E_cur"], last_render=(rw, rh)) return {"out": o["out"], "x0": x0, "N": N, "f": f, "reset": reset} diff --git a/clean/k/runner/k_io.py b/clean/k/runner/k_io.py index f3d736ac..43f9b5b0 100644 --- a/clean/k/runner/k_io.py +++ b/clean/k/runner/k_io.py @@ -56,7 +56,7 @@ class KInColorParams(ctypes.Structure): class KExpoArgs(ctypes.Structure): """Host arguments of k_expo_scalars (the per-frame runtime's GPU exposure scalars, clean/k/runtime/runtime_k.cpp).""" - _fields_ = [("value", ctypes.c_float)] + [(n, ctypes.c_int) for n in ("pair", "use_src", "first", "hdr", "reset")] + \ + _fields_ = [("value", ctypes.c_float)] + [(n, ctypes.c_int) for n in ("src_mode", "aux", "first", "hdr", "reset")] + \ [(n, ctypes.c_float) for n in ("e1", "e2", "inv_pi")] diff --git a/clean/k/runtime/runtime_k.cpp b/clean/k/runtime/runtime_k.cpp index 38b4feb5..9f0457d1 100644 --- a/clean/k/runtime/runtime_k.cpp +++ b/clean/k/runtime/runtime_k.cpp @@ -64,9 +64,9 @@ struct KInUniformParams { KInParams input; KiHost host; }; struct KInColorUniformParams { KInColorParams input; KiHost host; }; static_assert(sizeof(KInUniformParams) == 248 && sizeof(KInColorUniformParams) == 264, "input host arguments"); static_assert(offsetof(KInUniformParams, host) == 200 && offsetof(KInColorUniformParams, host) == 216, "input host offsets"); -struct KExpoArgs { +struct KExpoArgs { // k_io.hip KExpoArgs float value; - int pair, use_src, first, hdr, reset; + int src_mode, aux, first, hdr, reset; float e1, e2, inv_pi; }; static_assert(sizeof(KExpoArgs) == 36, "KExpoArgs"); @@ -264,7 +264,14 @@ int wide_ng(const std::string& s, bool f16, int nwin) { // own buffers first (hipMemcpyAsync, before the first mark); "input2+copyin" both), k_head (compact, the default: d0 writes // the compact pixel head for k_out_pixel_c; full: the 40-channel head map and k_out_pixel, same output bits, and the // buffer "k.N" for diagnostics), k_fast (0, the default: the exact stage kernels; 1: opt-in, not bit-exact, the "_f" -// variants of the input and output stage kernels, see k_io.hip KIO_FAST). +// variants of the input and output stage kernels, see k_io.hip KIO_FAST; it brings the fast input stage with it), +// k_fast_in (0, the default: the input-stage kernels from k_io_exact.co (WGK_IN_EXACT, every operation correctly +// rounded: bit for bit spec 42, as KFrame(exact_input=True)), with the output colour taps prepared by k_out_color as +// KFrame's exact plan does; 1: opt-in speed mode, the input stage from k_io.co, whose log2 / 2^x / reciprocals are the +// GPU's estimates (spec 42 section 9: within one FP16 step; not bit-exact)), k_in_exact (the older name: k_in_exact=1 is the default, k_in_exact=0 is k_fast_in=1; k_in_exact=1 +// is refused with k_fast=1 or k_fast_in=1), k_expo_rule (spec, the default: spec 42 draft 3 section 4, E_cur = the +// game's exposure or 1, E_aux = the auto-exposure value; v051: opt-in, for A/B only, the 0.5.1 rule E_cur = E_aux = +// the auto-exposure value when it runs). std::string option(const WgRtConfig& c, const std::string& key) { const char* env = std::getenv("WG_RUNTIME_OPTIONS"); const std::string text = std::string(c.runtime_options ? c.runtime_options : "") + ";" + (env ? env : ""); @@ -363,6 +370,19 @@ bool KRuntime::create(const WgRtConfig& c, Device* dev, AllocFn alloc_fn, bool e const std::string kfast = option(c, "k_fast"); if (!kfast.empty() && kfast != "0" && kfast != "1") return fail("model K: k_fast must be 0 or 1"); fast_ = kfast == "1"; + const std::string kexact = option(c, "k_in_exact"); + if (!kexact.empty() && kexact != "0" && kexact != "1") return fail("model K: k_in_exact must be 0 or 1"); + const std::string kfastin = option(c, "k_fast_in"); + if (!kfastin.empty() && kfastin != "0" && kfastin != "1") return fail("model K: k_fast_in must be 0 or 1"); + if (kexact == "1" && kfastin == "1") return fail("model K: k_in_exact=1 and k_fast_in=1 exclude each other"); + if (kexact == "1" && fast_) return fail("model K: k_in_exact=1 and k_fast=1 exclude each other"); + // the exact input stage is the default (spec 09 / 49 accuracy); k_fast_in=1 (or k_in_exact=0, or k_fast=1, whose + // "_f" kernels include the fast input stage) selects the fast one + in_exact_ = !(fast_ || kfastin == "1" || kexact == "0"); + const std::string krule = option(c, "k_expo_rule"); + if (!krule.empty() && krule != "spec" && krule != "v051") + return fail("model K: unknown k_expo_rule " + krule + " (spec or v051)"); + expo_v051_ = krule == "v051"; map_path_ = option(c, "model_k_map"); if (map_path_.empty()) { map_path_ = join(build_dir_, "preset-K.v1.map.json"); // next to the library, else built into it @@ -394,6 +414,9 @@ bool KRuntime::create(const WgRtConfig& c, Device* dev, AllocFn alloc_fn, bool e // exposure scalars (k_io.co), model M's exposure (spec 07) and frame helpers (timing_barrier) for (const char* co : {"k_block.co", "k_block_wide.co", "k_io.co", "exposure.co", "frame.co"}) if (!dev_->load_module(join(build_dir_, co))) return fail("model K: " + dev_->err()); + // k_in_exact=1: the same input kernels with exact arithmetic, addressed as "kx/" + if (in_exact_ && !dev_->load_module_tagged(join(build_dir_, "k_io_exact.co"), "kx")) + return fail("model K: " + dev_->err()); // buffers for (auto& kv : blobs_) { @@ -426,7 +449,7 @@ bool KRuntime::create(const WgRtConfig& c, Device* dev, AllocFn alloc_fn, bool e // exp2tab: read only by the host emulator's k_out_pixel (the GPU uses v_exp_f16); a valid, zeroed buffer !alloc(&exp2tab_, 65536 * 2, true) || !alloc(&tables_, 8 * kTableBytes) || !alloc(&expo_, 32, true) || !alloc(&expo_partials_, 4 * (size_t)cdiv(ny, 4)) || !alloc(&expo_computed_, 4, true) || - !alloc(&expo_sync_, 4, true) || !alloc(&xs_, 16, true)) + !alloc(&expo_sync_, 4, true) || !alloc(&xs_, 16, true) || !alloc(&kexpo_, 16, true)) return false; if (!external_io_ && !alloc(&out_, (size_t)W_ * H_ * 12, true)) return false; // the filter-shape table (spec 43 section 7) of every level, computed once; a frame reads its level's set @@ -565,7 +588,10 @@ bool KRuntime::frame(const WgRtFrame& fr, const FrameIo* io) { } const int rw = fr.rw, rh = fr.rh; if (rw <= 0 || rh <= 0 || rw > max_rw_ || rh > max_rh_) return fail("render size larger than the maximum"); - const bool reset = fr.reset || frames_ == 0 || rw != last_rw_ || rh != last_rh_; + // Model K resets on the game's flag and on the first frame only (spec 44: the reset restarts the frame index). A + // render-size change keeps the history: the histories are at output and base resolution, and the input and output + // stages reproject them with this frame's render size (model M's spec 10 rule does not apply to K). + const bool reset = fr.reset || frames_ == 0; const int f = reset ? 0 : fidx_; const int p = parity_, q = 1 - parity_; const int tb = io ? io->timing_base : -1; @@ -614,12 +640,17 @@ bool KRuntime::frame(const WgRtFrame& fr, const FrameIo* io) { min_ = motion_; } - // exposure (spec 10 section 5.1, spec 07) and the scalars of spec 42 section 4 on the GPU. Pre-exposure pi: 1 if - // missing, <= 0 or not finite (spec 10); model K does not divide the colour by pi, kappa = (1 / pi) e1 E does. + // exposure: spec 42 draft 3 section 4 (k_io.hip k_expo_scalars) on the GPU. Pre-exposure pi: 1 if missing, <= 0 + // or not finite (spec 10); model K does not divide the colour by pi, kappa = (1 / pi) e1 E_cur does. + // The front end's exposure source (FrameIo.exposure_mode): 0 a value (1 when the game gives nothing), 1 the game's + // exposure texel, 2 "compute it" (no texture, auto-exposure requested). E_cur is the game's value or texel, 1 with + // mode 2. Auto-exposure (spec 07, HDR only) runs when the game asked for it at creation (WgRtConfig.auto_exposure) + // or the front end asks this frame (mode 2), also next to a game texture; its value is E_aux only. const float pi = (std::isfinite(fr.pre_exposure) && fr.pre_exposure > 0.0f) ? fr.pre_exposure : 1.0f; const int mode = io && io->exposure_mode >= 0 ? io->exposure_mode : (cfg_.auto_exposure ? 2 : 0); + const bool auto_run = hdr_ && (mode == 2 || cfg_.auto_exposure); int first = frames_ == 0 ? 1 : 0; - if (mode == 2) { + if (auto_run) { ExpParams ep = {}; ep.rw = rw; ep.rh = rh; @@ -638,17 +669,20 @@ bool KRuntime::frame(const WgRtFrame& fr, const FrameIo* io) { } KExpoArgs xa = {}; xa.value = fr.exposure; - xa.pair = mode != 2; - xa.use_src = mode == 1; + xa.src_mode = mode == 1 ? 1 : mode == 2 ? 2 : 0; + xa.aux = auto_run ? 1 : 0; xa.first = first; xa.hdr = hdr_; xa.reset = reset; + // e1 = e2 = the game's exposure scale (spec 42 section 4); the front-end interface carries none, so 1 xa.e1 = 1.0f; xa.e2 = 1.0f; xa.inv_pi = 1.0f / pi; DevPtr src = mode == 1 && io ? io->expo_src : expo_computed_; if (mode == 1 && !src) return fail("exposure from the game's texture without a source buffer"); - if (!launch("k_expo_scalars", 1, 32, &expo_, &src, &xs_, &xa)) return false; + // k_expo_rule=v051 (A/B only): the 0.5.1 rule, the auto-exposure value as E_cur (so also as E_aux and E_prev) + if (expo_v051_ && mode == 2 && auto_run) xa.src_mode = 1, xa.aux = 0, src = expo_computed_; + if (!launch("k_expo_scalars", 1, 32, &kexpo_, &src, &expo_computed_, &xs_, &xa)) return false; // the frame's inputs (k_io.py frame_in): dense render-size images, motion in render pixels ([MVlo]; the interop // converts output-resolution motion to render pixels in its input pass) @@ -692,10 +726,14 @@ bool KRuntime::frame(const WgRtFrame& fr, const FrameIo* io) { ip.x0 = net_["X0"]; ip.cpre = cpre_; ip.cpre2 = cpre2_; - const char* input_kernel = reset ? "k_in_reset" : "k_in_base"; + const char* input_kernel = in_exact_ ? (reset ? "kx/k_in_reset" : "kx/k_in_base") : (reset ? "k_in_reset" : "k_in_base"); // k_fast=1: the "_f" variant of an input / output stage kernel (k_io.hip KIO_FAST), else the exact kernel itself std::string kname_buf; auto kname = [&](const char* n) -> const char* { + if (in_exact_ && std::strncmp(n, "k_in_", 5) == 0) { // input-stage kernels from k_io_exact.co + kname_buf = std::string("kx/") + n; + return kname_buf.c_str(); + } if (!fast_) return n; kname_buf = std::string(n) + "_f"; return kname_buf.c_str(); @@ -703,7 +741,8 @@ bool KRuntime::frame(const WgRtFrame& fr, const FrameIo* io) { const bool use_tiled = tiled_input_ && cfg_.backend == 0 && fin.s == 0.5f && W_ > 0 && W_ <= 16384 && H_ > 0 && H_ <= 16384 && Bw_ > 0 && Bw_ <= 4096 && Bh_ > 0 && Bh_ <= 4096 && rw > 0 && rw <= .68 * W_ && rh > 0 && rh <= .68 * H_ && std::isfinite(fin.jx) && std::isfinite(fin.jy) && std::fabs(fin.jx) <= .5f && std::fabs(fin.jy) <= .5f; - const bool use_tiled_color = use_tiled && !reset && 4LL * Bw_ >= W_ && 4LL * Bh_ >= H_; + // k_in_exact: the output colour taps by k_out_color (KFrame's exact plan), not by the tiled input kernel + const bool use_tiled_color = use_tiled && !in_exact_ && !reset && 4LL * Bw_ >= W_ && 4LL * Bh_ >= H_; KiHost input_host = {}; const bool use_uniforms = use_tiled && input_uniforms_safe(fin, &input_host); auto input_stage = [&]() { @@ -724,7 +763,8 @@ bool KRuntime::frame(const WgRtFrame& fr, const FrameIo* io) { if (use_uniforms) ++input_uniform_launches_; ++tiled_launches_; } else { - if (!launch("k_in_color", cdiv((long long)rw * rh, kThreads), kThreads, &ip)) return false; + if (!launch(in_exact_ ? "kx/k_in_color" : "k_in_color", cdiv((long long)rw * rh, kThreads), kThreads, &ip)) + return false; ++legacy_color_launches_; if (!launch(input_kernel, cdiv(4LL * Bw_ * Bh_, kThreads), kThreads, &ip)) return false; ++legacy_base_launches_; @@ -816,7 +856,7 @@ const char* KRuntime::kernels() { kernels_text_ = std::string("model=K network=") + (f8_ ? "fused8" : f16_ ? "fused16" : "fused") + (f8_ ? " fp8_stages=" + std::to_string(f8data_.size()) + (f8why_.empty() ? "" : " fp8_kept:" + f8why_) : std::string()) + - " input=k_io.co output=k_io.co exposure=k_expo_scalars" + + (in_exact_ ? " input=k_io_exact.co" : " input=k_io.co") + " output=k_io.co exposure=k_expo_scalars" + std::string(" input_tiled=") + (tiled_input_ ? "1" : "0") + " input_actual=" + (tiled_last_ ? "tiled" : "legacy") + " tiled_launches=" + std::to_string(tiled_launches_) + @@ -826,7 +866,7 @@ const char* KRuntime::kernels() { " tiled_color_launches=" + std::to_string(tiled_color_launches_) + " legacy_color_launches=" + std::to_string(legacy_color_launches_) + " legacy_base_launches=" + std::to_string(legacy_base_launches_) + - (fast_ ? " k_fast=1" : "") + + (fast_ ? " k_fast=1" : "") + (expo_v051_ ? " k_expo_rule=v051" : "") + " hidden=fused_d0" + std::string(" head=") + (compact_head_ ? "compact" : "full") + " base=" + std::to_string(Bw_) + "x" + std::to_string(Bh_) + " hdr=" + (hdr_ ? "1" : "0") + " mv=render" + @@ -850,7 +890,9 @@ bool KRuntime::buffer(const std::string& name, DevPtr* p, size_t* bytes) { size_t b; }; std::map m = { - {"exposure.pair", {expo_, 12}}, {"k.scalars", {xs_, 16}}, + // exposure.pair: [E_cur, E_prev, E_aux] of the last frame (raw values, k_expo_scalars); exposure.measured: the + // last auto-exposure value (spec 07) + {"exposure.pair", {kexpo_, 12}}, {"k.scalars", {xs_, 16}}, {"exposure.measured", {expo_computed_, 4}}, {"k.x0", {net_["X0"], net_bytes_["X0"]}}, {"k.N", {net_["OUT"], net_bytes_["OUT"]}}, {"k.Hc", {Hc_[lp], P16 * 8}}, {"k.L", {L_[lp], P16 * 8}}, {"k.Hh", {Hh_[lp], (size_t)Bw_ * Bh_ * 8}}, {"filter.out", {out_, (size_t)W_ * H_ * 12}}, diff --git a/clean/k/runtime/runtime_k.h b/clean/k/runtime/runtime_k.h index ee9eb5e8..1b5a26e9 100644 --- a/clean/k/runtime/runtime_k.h +++ b/clean/k/runtime/runtime_k.h @@ -14,10 +14,11 @@ // next to model.wgcm (runtime option model_k_map= overrides). // // Per frame (clean/k/FORMATS.md "Runtime frame"): [raw inputs: input_convert] -> exposure (spec 07 auto-exposure when -// asked, exposure_auto_diet) -> k_expo_scalars (kappa, rho, lambda on the GPU from the exposure pair) -> input stage -// (k_in_color, k_in_base) -> eleven stage kernels -> output stage (k_out_color, k_out_pixel, k_out_hidden) writing -// the game output in the caller's format. Histories Hc, L, Hh double-buffered and zero at create; reset on the game's -// flag, the first frame and a render-size change; frame index f = 0 on a reset frame, then + 1. +// asked, exposure_auto_diet) -> k_expo_scalars (E_cur, E_prev, E_aux of spec 42 draft 3 section 4, then kappa, rho, +// lambda on the GPU) -> input stage (exact by default: k_io_exact.co) -> eleven stage kernels -> output stage +// (k_out_color, k_out_pixel, k_out_hidden) writing the game output in the caller's format. Histories Hc, L, Hh +// double-buffered and zero at create; reset on the game's flag and the first frame only (not on a render-size +// change); frame index f = 0 on a reset frame, then + 1. #pragma once #include @@ -103,6 +104,9 @@ class KRuntime { // (k_io.hip FAST forms: hardware reciprocals instead of correctly rounded divisions, the luma history cubics as // packed FP16, the filter weight sum instead of the largest exponent; clean/k/tests/test_k_fast.py) bool fast_ = false; + // input-stage kernels from k_io_exact.co (tag "kx"): the default; off with k_fast_in=1 (or k_in_exact=0, k_fast=1) + bool in_exact_ = true; + bool expo_v051_ = false; // runtime option k_expo_rule=v051 (A/B only): the 0.5.1 exposure rule int offsets_table_[8][2] = {}; std::map> blobs_; std::map wdev_, net_; // stage blobs; network maps (X0, S0..S4, A, B, OUT) @@ -110,6 +114,7 @@ class KRuntime { DevPtr Hc_[2] = {0, 0}, L_[2] = {0, 0}, Hh_[2] = {0, 0}; DevPtr color_ = 0, depth_ = 0, motion_ = 0, out_ = 0, cpre_ = 0, cpre2_ = 0, ctap_ = 0, ctapT_ = 0; DevPtr exp2tab_ = 0, tables_ = 0, expo_ = 0, expo_partials_ = 0, expo_computed_ = 0, expo_sync_ = 0, xs_ = 0; + DevPtr kexpo_ = 0; // k_expo_scalars state: E_cur, E_prev, E_aux of the last frame, E_cur kept for the next // frame state int frames_ = 0, parity_ = 0, fidx_ = 0, last_rw_ = 0, last_rh_ = 0, level_last_ = -1, omode_last_ = -1; float jprev_[2] = {0, 0}; diff --git a/clean/k/tests/test_k_exposure_rule.py b/clean/k/tests/test_k_exposure_rule.py index 3a2da4d0..7321a502 100644 --- a/clean/k/tests/test_k_exposure_rule.py +++ b/clean/k/tests/test_k_exposure_rule.py @@ -1,15 +1,24 @@ # SPDX-License-Identifier: MIT -"""Model K's exposure rule, CPU only (no device, no emulator, no weights): clean/k/reference/exposure.py, which exposure -value is which. HDR: E_cur is the game's exposure or 1, E_aux the auto-exposure value when it runs (else E_cur), the -measured value never takes E_cur's place; LDR: every value and factor 1; e1 = e2 is the game's positive finite scale, -else 1. +"""Model K host rules in front of the stages, CPU only (no device, no emulator, no network weights): + +- clean/k/reference/exposure.py: which exposure value is which (E_cur the game's exposure or 1, E_aux the auto-exposure + value when it runs, LDR everything 1, e1 = e2 the game's positive finite scale else 1); +- clean/k/runner/k_frame.py reference_frame: the rule is applied, E_prev is the previous frame's E_cur (E_cur on a + reset), and a reset is the game's flag or the first frame only: a render-size change keeps the history and the + frame index. + +The stages themselves are replaced by recorders; their own tests are the other modules of this folder. Run: python -m unittest clean.k.tests.test_k_exposure_rule """ import math import unittest +from unittest import mock + +import numpy as np from clean.k.reference import exposure +from clean.k.runner import k_frame class TestExposureRule(unittest.TestCase): @@ -42,10 +51,62 @@ def test_apply_rule(self): self.assertEqual((d["E_cur"], d["E_aux"], d["e1"], d["e2"]), (2.0, 0.25, 4.0, 4.0)) self.assertEqual(exposure.apply_rule({}, True)["E_cur"], 1.0) self.assertEqual(exposure.apply_rule({"game_exposure": 9.0}, False)["E_cur"], 1.0) - # the input is not modified - x = {"game_exposure": 2.0} - exposure.apply_rule(x, True) - self.assertEqual(x, {"game_exposure": 2.0}) + + +class TestReferenceFrameRules(unittest.TestCase): + """reference_frame with recorder stages.""" + + W, H, BASE = 40, 24, (32, 16) + + def run_sequence(self, hdr, frames): + state = k_frame.new_reference_state((self.W, self.H), self.BASE) + seen = [] + + def fake_in(d): + seen.append(dict(d)) + return np.zeros((self.BASE[1], self.BASE[0], 16), np.float16) + + def fake_net(x0, params, f): + seen[-1]["f"] = f + return {"d0": {"head": np.zeros((self.BASE[1], self.BASE[0], 40), np.float16)}} + + def fake_out(d): + return {"out": np.zeros((self.H, self.W, 4), np.float32), "L": d["L"], "Hc": d["Hc"], + "Hh": np.zeros((self.BASE[1], self.BASE[0], 4), np.float16)} + + results = [] + with mock.patch.object(k_frame.instage, "input_stage", fake_in), \ + mock.patch.object(k_frame.network, "run_network", fake_net), \ + mock.patch.object(k_frame.outstage, "output_stage", fake_out): + for x in frames: + results.append(k_frame.reference_frame(state, x, None, {"hdr": hdr})) + return results, seen + + def frame(self, size, reset=False, game=None, measured=None): + return {"render_size": size, "reset": reset, "game_exposure": game, "measured": measured} + + def test_reset_is_the_flag_or_the_first_frame_only(self): + frames = [self.frame((27, 16)), self.frame((27, 16)), self.frame((30, 20)), self.frame((30, 20), reset=True), + self.frame((30, 20))] + results, _ = self.run_sequence(True, frames) + self.assertEqual([r["reset"] for r in results], [True, False, False, True, False]) + # the size change of frame 2 keeps the frame index running + self.assertEqual([r["f"] for r in results], [0, 1, 2, 0, 1]) + + def test_e_prev_is_the_previous_e_cur(self): + frames = [self.frame((27, 16), game=0.75), self.frame((27, 16), game=1.25, measured=4.0), + self.frame((30, 20), game=None), self.frame((30, 20), game=2.0, reset=True)] + _, seen = self.run_sequence(True, frames) + self.assertEqual([s["E_cur"] for s in seen], [0.75, 1.25, 1.0, 2.0]) + self.assertEqual([s["E_aux"] for s in seen], [0.75, 4.0, 1.0, 2.0]) + # first frame and reset frame: E_prev = E_cur; otherwise the previous frame's E_cur (not E_aux) + self.assertEqual([s["E_prev"] for s in seen], [0.75, 0.75, 1.25, 2.0]) + + def test_ldr_values_are_one(self): + frames = [self.frame((27, 16), game=3.0, measured=2.0), self.frame((27, 16), game=5.0)] + _, seen = self.run_sequence(False, frames) + for s in seen: + self.assertEqual((s["E_cur"], s["E_aux"], s["E_prev"], s["e1"], s["e2"]), (1.0,) * 5) if __name__ == "__main__": diff --git a/clean/k/tests/test_k_frame_emu.py b/clean/k/tests/test_k_frame_emu.py index edf3da59..e776c93c 100644 --- a/clean/k/tests/test_k_frame_emu.py +++ b/clean/k/tests/test_k_frame_emu.py @@ -1,6 +1,8 @@ # SPDX-License-Identifier: MIT """Model K whole frame (clean/k/runner/k_frame.py) in the host emulator: input stage -> network -> output stage with -histories and the frame index, over a short sequence with a render-size change (reset). +histories and the frame index, over a short sequence with a render-size change (no reset for K: the history is +kept) and then a game reset, with the exposure rule of spec 42 draft 3 section 4 (clean/k/reference/exposure.py: +the game's exposure or 1 as E_cur, an auto-exposure value as E_aux). - Wiring, exactly: each frame's input and output stages equal the NumPy references bit for bit when those are given the emulator's own histories and network output. @@ -21,17 +23,19 @@ from clean.k.tests.test_k_kernels_emu import device W, H, BASE = 40, 24, (32, 16) -SIZES = [(27, 16), (27, 16), (27, 16), (30, 20)] +# (render size, game reset, the game's exposure (None: none), the auto-exposure value (None: off)) +SIZES = [(27, 16, False, None, 0.75), (27, 16, False, None, 1.5), (27, 16, False, 1.25, 1.5), (30, 20, False, 2.0, None), + (30, 20, True, 2.0, 0.5)] def frames(rng, flags): - for rw, rh in SIZES: + for rw, rh, reset, game, measured in SIZES: mshape = (rh, rw, 2) if flags["mv_lowres"] else (H, W, 2) yield dict(color=(rng.random((rh, rw, 3)) * (3 if flags["hdr"] else 1)).astype(np.float32), motion=(rng.standard_normal(mshape) * 0.7).astype(np.float32), depth=rng.random((rh, rw)).astype(np.float32), render_size=(rw, rh), jitter=tuple(float(v) for v in rng.uniform(-0.5, 0.5, 2)), mv_scale=(1.0, 1.0), pre_exposure=1.0, - reset=False, E_cur=1.0, E_aux=1.0, e1=1.0, e2=1.0) + reset=reset, game_exposure=game, measured=measured) class TestFrameEmu(unittest.TestCase): @@ -49,8 +53,18 @@ def test_sequence(self): hist = kf.histories() g = kf.frame(x, debug=True) d = g["frame"] - self.assertEqual(g["reset"], i in (0, 3)) - self.assertEqual(g["f"], {0: 0, 1: 1, 2: 2, 3: 0}[i]) + self.assertEqual(g["reset"], i in (0, 4)) + self.assertEqual(g["f"], {0: 0, 1: 1, 2: 2, 3: 3, 4: 0}[i]) + # the exposure rule (spec 42 draft 3 section 4) + rw, rh, _, game, measured = SIZES[i] + if flags["hdr"]: + want_cur = 1.0 if game is None else game + self.assertEqual(d["E_cur"], want_cur) + self.assertEqual(d["E_aux"], want_cur if measured is None else measured) + prev = SIZES[i - 1][3] if i else None + self.assertEqual(d["E_prev"], want_cur if i in (0, 4) else (1.0 if prev is None else prev)) + else: + self.assertEqual((d["E_cur"], d["E_aux"], d["E_prev"], d["e1"], d["e2"]), (1.0,) * 5) # wiring, bit for bit x0 = instage.input_stage(dict(d, Hc=hist["Hc"], Hh=hist["Hh"])) self.assertTrue(same_bits(g["x0"].astype(np.float16), x0).all(), f"frame {i}: input stage") diff --git a/clean/k/tests/test_k_runtime_gpu.py b/clean/k/tests/test_k_runtime_gpu.py index 0ce62f6a..120b1b27 100644 --- a/clean/k/tests/test_k_runtime_gpu.py +++ b/clean/k/tests/test_k_runtime_gpu.py @@ -6,10 +6,14 @@ - A synthetic model K (random FP16 parameters, clean/k/reference/network.py random_network_params) written as the converter's model.wgcm (clean/converter/OUTPUT-FORMAT.md) with a matching map, loaded and checked by the runtime. -- Sequences with a reset, steady frames, a render-size change and a game reset; HDR and LDR, depth inverted or not, - the three exposure sources (value, the game's texel on the GPU, auto-exposure on the GPU), pre-exposure != 1, both +- Sequences with a reset, steady frames, a render-size change (model K keeps the history) and a game reset; HDR and + LDR, depth inverted or not, the three exposure sources (value, the game's texel on the GPU, auto-exposure on the GPU) + and a texel with auto-exposure on, with the exposure rule of spec 42 draft 3 section 4 (clean/k/reference/exposure.py: + E_cur = the game's exposure or 1, E_aux = the auto-exposure value; KFrame is given the rule's inputs and the + runtime's measured value, and the runtime's [E_cur, E_prev, E_aux] are checked), pre-exposure != 1, both output layouts of the interop (dense RGB float32; RGBA16F rows with alpha 1) and the input-copy path (raw RGBA16F colour, R32F depth, FP16 motion converted on the GPU). +- The input stage: exact (k_io_exact.co) by default, the fast one (k_io.co) with k_fast_in=1. - 1440p Quality (2560 x 1440 from 1706 x 960): the runtime's timing marks (as timing_log=1 in game) per part. GPU only: WG_K_GPU_BUILD=, ROCM_PATH=; @@ -288,13 +292,17 @@ def setUpClass(cls): cls.wgcm, cls.map = write_k_model(cls.mdir, cls.params) def run_sequence(self, flags, frames, out_size=(200, 120), max_render=(150, 90), mode=0, half=False, raw=False, - seed=0, network="fused", options=""): + seed=0, network="fused", options="", exact_input=True, auto=None): """frames: [(rw, rh, reset, E, pre_exposure)]; every frame's runtime output against KFrame's (network: the - runtime option network= and KFrame's variant).""" + runtime option network= and KFrame's variant). E: the game's exposure (value, mode 0, or texel, mode 1; not + used with mode 2). auto: the create flag "auto-exposure" (default: mode == 2).""" rng = np.random.default_rng(seed) + auto = mode == 2 if auto is None else auto rt = KRuntimeExt(self.build, self.mdir, self.map, out_size, max_render, flags["hdr"], flags["depth_inverted"], - mode == 2, ";".join(o for o in ("" if network == "fused" else f"network={network}", options) if o)) - kf = KFrame(self.dev, self.fn, out_size, max_render, self.params, dict(flags, mv_lowres=True), network) + auto, ";".join(o for o in ("" if network == "fused" else f"network={network}", options) if o)) + E_last = None + kf = KFrame(self.dev, self.fn, out_size, max_render, self.params, dict(flags, mv_lowres=True), network, + exact_input=exact_input) try: for i, (rw, rh, reset, E, pi) in enumerate(frames): x = inputs(rng, rw, rh, flags["hdr"], reset) @@ -316,10 +324,20 @@ def run_sequence(self, flags, frames, out_size=(200, 120), max_render=(150, 90), mrow[:, :mch * rw] = mraw.reshape(rh, mch * rw) rawio = (crow, mrow, ms, mo, mch) (got, alpha), _ = rt.frame(x, mode, half=half, raw=rawio) - if mode == 2: # auto-exposure: the runtime's value (spec 07 on the GPU) is KFrame's E_cur - x["E_cur"] = float(rt.buffer("exposure.pair", np.float32)[0]) - x["E_aux"] = x["E_cur"] - want = kf.frame(x) + # spec 42 draft 3 section 4: KFrame gets the game's exposure (none with mode 2) and, when auto-exposure + # runs (HDR, the create flag or mode 2), the runtime's measured value (spec 07 on the GPU) as E_aux + runs = flags["hdr"] and (auto or mode == 2) + measured = float(rt.buffer("exposure.measured", np.float32)[0]) if runs else None + kx = {k: v for k, v in x.items() if k not in ("E_cur", "E_aux", "e1", "e2")} + kx.update(game_exposure=None if mode == 2 else x["E_cur"], measured=measured) + want = kf.frame(kx) + # the runtime's exposure state [E_cur, E_prev, E_aux] (raw values) follows the rule + ec = (1.0 if mode == 2 else x["E_cur"]) if flags["hdr"] else 1.0 + ea = measured if runs else ec + ep = ec if (i == 0 or reset) else E_last + E_last = ec + pair = rt.buffer("exposure.pair", np.float32)[:3] + np.testing.assert_array_equal(pair, np.array([ec, ep, ea], np.float32), err_msg=f"exposure frame {i}") ref = want["out"][..., :3] info = (i, rw, rh, flags, mode, half, raw) if half: @@ -337,7 +355,8 @@ def run_sequence(self, flags, frames, out_size=(200, 120), max_render=(150, 90), info_k = dict(kv.split("=", 1) for kv in rt.lib.wg_rt_kernels(rt.rt).decode().split()) self.assertEqual(info_k["model"], "K") self.assertEqual(info_k["network"], network) - self.assertEqual(want["reset"], i == 0 or reset or frames[i - 1][:2] != (rw, rh)) + self.assertEqual(want["reset"], bool(i == 0 or reset)) # a render-size change keeps the history (K) + self.assertEqual(info_k["input"], "k_io_exact.co" if exact_input else "k_io.co") finally: kf.release() del rt @@ -357,6 +376,45 @@ def test_hdr_auto_exposure(self): seq = [(w, h, r, E, 2.0) for (w, h, r, E, _) in self.SEQ] self.run_sequence({"hdr": True, "depth_inverted": True}, seq, mode=2, half=True, seed=2) + def test_hdr_texel_with_auto_exposure(self): + """The game's texel and auto-exposure on (create flag): E_cur = the texel, E_aux = the measured value.""" + seq = [(w, h, r, E, 1.0 + 0.25 * i) for i, (w, h, r, E, _) in enumerate(self.SEQ)] + self.run_sequence({"hdr": True, "depth_inverted": True}, seq, mode=1, half=True, seed=7, auto=True) + + def test_ldr_auto_flag(self): + """LDR: every exposure value 1, even with the auto-exposure flag (it does not run).""" + self.run_sequence({"hdr": False, "depth_inverted": True}, self.SEQ[:4], mode=2, seed=8) + + def test_fast_input(self): + """k_fast_in=1 (opt-in): the fast input stage (k_io.co), bit for bit KFrame's default plan; the older name + k_in_exact=0 is the same; at a small size and at 1440p Ultra Performance (tiled input, tiled colour).""" + for opt in ("k_fast_in=1", "k_in_exact=0"): + self.run_sequence({"hdr": True, "depth_inverted": True}, self.SEQ[:5], mode=2, half=True, options=opt, + exact_input=False, seed=9) + seq = [(853, 480, False, 1.0, 1.0), (853, 480, False, 1.0, 1.0), (853, 480, False, 1.0, 1.0)] + self.run_sequence({"hdr": True, "depth_inverted": True}, seq, out_size=(2560, 1440), max_render=(853, 480), + mode=2, half=True, options="k_fast_in=1", exact_input=False, seed=10) + + def test_expo_rule_v051(self): + """k_expo_rule=v051 (A/B only): the 0.5.1 rule, E_cur = E_aux = the auto-exposure value.""" + rng = np.random.default_rng(12) + rt = KRuntimeExt(self.build, self.mdir, self.map, (200, 120), (150, 90), True, True, True, "k_expo_rule=v051") + kf = KFrame(self.dev, self.fn, (200, 120), (150, 90), self.params, + {"hdr": True, "mv_lowres": True, "depth_inverted": True}, exact_input=True) + try: + for i in range(3): + x = inputs(rng, 150, 90, True, i == 0) + x["pre_exposure"] = 1.5 + (got, _), _ = rt.frame(x, 2, half=True) + m = float(rt.buffer("exposure.measured", np.float32)[0]) + self.assertEqual(float(rt.buffer("exposure.pair", np.float32)[0]), m) + want = kf.frame(dict(x, E_cur=m, E_aux=m)) + np.testing.assert_array_equal(got.astype(np.float16).view(np.uint16), + want["out"][..., :3].astype(np.float16).view(np.uint16)) + finally: + kf.release() + del rt + def test_ldr(self): self.run_sequence({"hdr": False, "depth_inverted": False}, self.SEQ, mode=0, seed=3) @@ -447,6 +505,27 @@ def test_diagnostics_same_output(self): self.run_sequence({"hdr": True, "depth_inverted": True}, self.SEQ[:4], mode=2, half=True, options="k_diag=input2+copyin") + def test_exact_input(self): + """The default input stage (k_in_exact=1 names it): k_io_exact.co, bit for bit KFrame(exact_input=True) (spec 42 + exactly), at a small size (legacy input path: render > 0.68 output) and at 1440p Performance / Ultra + Performance render sizes (tiled input with host uniforms, separate output colour taps), with a render-size + change inside the 1440p sequence (history kept).""" + self.run_sequence({"hdr": True, "depth_inverted": True}, self.SEQ, mode=0, options="k_in_exact=1", + exact_input=True) + for rw, rh in ((1280, 720), (853, 480)): + seq = [(rw, rh, False, 1.0, 1.0), (rw, rh, False, 1.25, 1.0), (rw, rh, False, 1.25, 1.0), + (rw - 64, rh - 36, False, 1.25, 1.0)] + self.run_sequence({"hdr": True, "depth_inverted": True}, seq, out_size=(2560, 1440), max_render=(rw, rh), + mode=2, half=True, exact_input=True, seed=rw) + + def test_exact_input_with_fast_refused(self): + for opts, msg in (("k_in_exact=1;k_fast=1", "k_in_exact=1 and k_fast=1"), + ("k_in_exact=1;k_fast_in=1", "k_in_exact=1 and k_fast_in=1"), + ("k_fast_in=2", "k_fast_in must be 0 or 1"), ("k_expo_rule=x", "unknown k_expo_rule")): + with self.assertRaises(RuntimeError) as cm: + KRuntimeExt(self.build, self.mdir, self.map, (200, 120), (150, 90), True, True, False, opts) + self.assertIn(msg, str(cm.exception)) + def test_odd_output_size(self): seq = [(97, 61, False, 1.0, 1.0), (97, 61, False, 1.25, 1.0), (101, 64, False, 1.25, 1.0)] self.run_sequence({"hdr": True, "depth_inverted": True}, seq, out_size=(203, 117), max_render=(101, 64), @@ -463,14 +542,15 @@ def test_1440p_quality_timing(self): rt = KRuntimeExt(self.build, self.mdir, self.map, (W, H), (rw, rh), True, True, True, "" if network == "fused" else f"network={network}") kf = KFrame(self.dev, self.fn, (W, H), (rw, rh), self.params, {"hdr": True, "mv_lowres": True, - "depth_inverted": True}, network) + "depth_inverted": True}, network, + exact_input=True) try: x = inputs(rng, rw, rh, True, True) for i in range(2): (got, _), _ = rt.frame(x, 2, half=True) - x["E_cur"] = float(rt.buffer("exposure.pair", np.float32)[0]) - x["E_aux"] = x["E_cur"] - want = kf.frame(x) + kx = {k: v for k, v in x.items() if k not in ("E_cur", "E_aux", "e1", "e2")} + want = kf.frame(dict(kx, game_exposure=None, + measured=float(rt.buffer("exposure.measured", np.float32)[0]))) np.testing.assert_array_equal(got.astype(np.float16).view(np.uint16), want["out"][..., :3].astype(np.float16).view(np.uint16)) x["reset"] = False diff --git a/clean/runner/build.py b/clean/runner/build.py index 90edd82f..272829f8 100644 --- a/clean/runner/build.py +++ b/clean/runner/build.py @@ -52,13 +52,13 @@ # motion vectors at output resolution (runtime option motion_outres=original; MOTION_OUTRES below) "encoder_mo.co", "post_mo_w64.co", "post_mo_w64__q5.co", "post_mo_w64__orig.co", # model K (runtime option model=K, clean/k/runtime/runtime_k.cpp): stage kernels, I/O stages, its map - "k_block.co", "k_block_wide.co", "k_io.co", os.path.basename(K_MAP_FILE)] + "k_block.co", "k_block_wide.co", "k_io.co", "k_io_exact.co", os.path.basename(K_MAP_FILE)] # Model K in the runtime library. --no-model-k (set_model_k(False)) builds a library without it: no clean/k/runtime # source, none of K_EMBED_FILES built in, -DWG_NO_MODEL_K (clean/runtime/no_model_k.cpp: model=K fails at create with # "Model K is not part of this build"). Model M is the same code and the same code objects either way. MODEL_K = True K_RUNTIME_SOURCES = [s for s in RUNTIME_SOURCES if s.startswith(os.path.join("..", "k", ""))] -K_EMBED_FILES = ["k_block.co", "k_block_wide.co", "k_io.co", os.path.basename(K_MAP_FILE)] +K_EMBED_FILES = ["k_block.co", "k_block_wide.co", "k_io.co", "k_io_exact.co", os.path.basename(K_MAP_FILE)] NO_MODEL_K_SOURCE = "no_model_k.cpp" # Optional models of the runtime library (#89). Model M is always built. Each model below is in by default and can be