Skip to content
21 changes: 20 additions & 1 deletion clean/l/FORMATS.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,26 @@ the frame parity, as Model M's.
3. `l_coef` y2 -> A, B, C, new latent. Model L's latent recurrence runs only through the next frame's input
encoder, which samples this latent.

The reconstruction (post) stage is still open.
5. Reconstruction (post) and output filter. The float64 oracle is `clean/l/reference/recon.py`, driven by the
table's `recon` entry. It is Model M's reconstruction with:
- a 4 x 4 colour gather;
- history candidates combined in linear colour;
- the plain linear colour as the fallback of an invalid or reset sample;
- the post stage's learned scale.

Behaviour pinned in the oracle:
- The tiling variant (3/2 or 3/1) follows `table.recon_variant`, a D / render ratio. Both variants compute the same
function.
- A zero total gather weight follows the normal formulas with NaN: history h(E(NaN)) = -1.4404296875 in HDR, NaN
in LDR; display and reduced code 0.
- LDR writes 11/11/10-bit display codes of the plain colour. The output stage unpacks them (code RN32(1/N), FP16)
and filters them with no decode and no exposure (`recon.output_filter`).
- The output filter is Model M's (the project's own taps).

The reconstruction kernel is **planned**. Its arguments and buffers will be Model M's `post` kernels' (Model M's
`clean/FORMATS.md`). Its summation order (taps, candidates) is not fixed yet: the oracle sums the gather columns
first, while Model M's reference sums rows first. The kernel's order decides, and the kernel-order model follows
it.

## Checks (GPU, synthetic data)

Expand Down
7 changes: 5 additions & 2 deletions clean/l/reference/oracle.py
Original file line number Diff line number Diff line change
Expand Up @@ -361,8 +361,11 @@ def decode(x, params, parity, t=_table.TABLE, keep=False):
return out


def reconstruct(*args, **kwargs):
raise NotImplementedError("Model L output reconstruction (table entry post_variant): pending")
def reconstruct(x, params, t=_table.TABLE, ldr=False):
"""Model L's output reconstruction (clean/l/reference/recon.py) with the post stage's learned scale.
x: Model M's reconstruction input dict (clean/reference/post.py reconstruct)."""
from . import recon as _recon
return _recon.reconstruct(x, params["post"]["s"], t, ldr)


# ---- whole network --------------------------------------------------------------------------------------------------
Expand Down
219 changes: 219 additions & 0 deletions clean/l/reference/recon.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,219 @@
# SPDX-License-Identifier: MIT
"""Model L's output reconstruction (stage "post"), NumPy float64 oracle (issue #89).

Model L's reconstruction is Model M's (clean/reference/post.py reconstruct, spec 06) with three changes, selected by
the structure table's "recon" entry (clean/l/reference/table.py):
1. the colour gather uses a window x window taps around the target (Model L: 4 x 4, offsets -1 ... 2; Model M: 3 x 3);
2. each history candidate is decoded and has the previous exposure removed before the two are combined
("history_combine": "linear"): L_i = D(H_i) / e_prev, H_lin = (w1 L1 + w2 L2) e (L2 only when w2 > 0);
3. a sample that is invalid (outside D, or reset) falls back to the render pixel's linear colour as it is
("fallback": "linear_colour"), with no exposure factor and no encode / decode.
The learned scale s is the post stage's tensor "s". Everything else (positions, the score and candidate rules, the
12-tap Catmull-Rom, the blend, the NaN guard, the stored outputs) is Model M's, by import.

Arithmetic is float64 with only the stored outputs rounded, as Model M's oracle: the reconstruction's half-precision
and fused-rounding forms (e.g. the gather exponent) are a matter of the kernel-order model, not of this oracle.
ldr=True gives the LDR form: identity transfer, no exposure (e = e_prev = 1), codes from the values directly; the
display is the 11 / 11 / 10-bit codes of the plain colour (ldr_display_codes), which the output stage unpacks with
code RN32(1/N) and filters without a decode or an exposure (output_filter). Both confirmed by the spec side.

Zero total gather weight (every tap underflows) has no special path, as confirmed by the spec side for Model L and
Model M: the normalized features are 0 / 0 = NaN, both scores NaN (candidate 1 kept, w1 = 1, w2 = 0), C = D(NaN) =
NaN (the decode passes NaN through: dec), alpha = 0, and the NaN guard substitutes C, which is NaN. The stored history
is then h(E(NaN)) = -1.4404296875 (HDR) or NaN (LDR), the display and the reduced code 0.

The reconstruction kernel's summation order (taps, candidates) is not fixed yet; this oracle sums in float64 and is
the tolerance reference, the kernel's order decides the kernel-order model.

The reconstruction variant (3/2 or 3/1) only decides how a kernel tiles the work; both forms compute this function
(table.recon_variant).
"""
import numpy as np

from clean.reference import motion_hi as _motion_hi
from clean.reference import post as _post

from . import table as _table


def dec(q):
"""The decode D(.) with a NaN passed through. The clamp to [1e-4, 3] applies to numbers only (confirmed by the spec
side for Model L and Model M: a zero gather weight gives C = D(NaN) = NaN). clean/reference/post.py dec maps NaN to
D(1e-4) instead."""
q = np.asarray(q, np.float64)
with np.errstate(invalid="ignore", over="ignore"):
return np.where(np.isnan(q), np.nan, _post.dec(q))


def _transfer(ldr):
"""(encode, decode, input encode) for HDR or LDR."""
if ldr:
ident = lambda v: np.asarray(v, np.float64) # noqa: E731
return ident, ident, ident
return _post.enc, dec, _post.enc_in


# LDR display codes (11 / 11 / 10 bits) and the output stage's unpack constants RN32(1/2047), RN32(1/1023).
LDR_CODE_MAX = np.array([2047.0, 2047.0, 1023.0])
LDR_UNPACK = np.array([np.float32(1.0 / 2047.0), np.float32(1.0 / 2047.0), np.float32(1.0 / 1023.0)], np.float64)


def ldr_display_codes(O):
"""LDR packed display codes of the plain display colour O^ [.., 3]: floor(clamp(O^ N + 1/2, 0, N)), N = 2047, 2047,
1023; NaN gives 0."""
with np.errstate(invalid="ignore", over="ignore"):
return np.floor(_post.clamp(np.asarray(O, np.float64) * LDR_CODE_MAX + 0.5, 0.0, LDR_CODE_MAX))


def ldr_unpack(codes):
"""The output stage's LDR unpack: code RN32(1/N), one single-precision product, rounded to FP16 (no decode)."""
v = (np.asarray(codes, np.float64) * LDR_UNPACK).astype(np.float32)
return v.astype(np.float16).astype(np.float32)


def output_filter(display, exposure, output_size, ldr=False):
"""Stage "filter" for Model L: Model M's output filter (clean/reference/post.py, the project's own taps) on the
reconstruction's "display". HDR: E-encoded display, decode and divide by the current exposure, as Model M. LDR:
display holds the unpacked codes (ldr_unpack), filtered as they are: no decode, no exposure read."""
if not ldr:
return _post.output_filter(display, exposure, output_size)
ow, oh = (int(v) for v in output_size)
img = np.asarray(display, np.float64)
dh, dw = img.shape[:2]
if (dw, dh) != _table.recon_size(ow, oh):
raise ValueError(f"display is {dw}x{dh}, expected D = {_table.recon_size(ow, oh)} for O = {ow}x{oh}")
h = _post.filter_pass(img.transpose(1, 0, 2), ow).transpose(1, 0, 2)
return _post.filter_pass(h, oh).astype(np.float32)


def gather(feat, C, px, py, rw, rh, window):
"""Step 2 with a window x window tap window: columns ax - 1 ... ax + window - 2, rows b - 1 ... b + window - 2
(b from the even row of each row pair, shared by both rows). Tap order: i outer, j inner. Returns W [dh, dw]
and the normalized features [dh, dw, 9]."""
ax, b = _post.kernel_window(px, py)
C = np.asarray(C, np.float64)
W = np.zeros((py.size, px.size))
acc = np.zeros((py.size, px.size, feat.shape[2]))
offs = range(-1, window - 1)
with np.errstate(invalid="ignore", over="ignore"):
for i in offs:
sx = np.clip(ax + i, 0, rw - 1)
u = (ax + i + 0.5 - px)[None, :]
for j in offs:
sy = np.clip(b + j, 0, rh - 1)
v = (b + j + 0.5 - py)[:, None]
c = C[sy][:, sx]
w = np.exp(-0.5 * (c[..., 0] * u * u + c[..., 1] * v * v + 2.0 * c[..., 2] * u * v))
W += w
acc += w[..., None] * feat[sy][:, sx]
fbar = acc / W[..., None]
return W, fbar


def reconstruct(x, s, t=_table.TABLE, ldr=False):
"""Stage "post" for Model L. x: Model M's reconstruction input dict (clean/reference/post.py reconstruct); s: the
post stage's learned scale. Returns Model M's output dict ("history", "reduced", "display", "_lin")."""
rc = t["recon"]
window = int(rc["window"])
linear = rc["history_combine"] == "linear"
lin_fallback = rc["fallback"] == "linear_colour"
enc, dec, enc_in = _transfer(ldr)

color = np.asarray(x["color"], np.float64)
rh, rw = color.shape[:2]
dw, dh = (int(v) for v in x["recon_size"])
jx, jy = (float(v) for v in np.asarray(x["jitter"], np.float64).reshape(2))
e, e_prev = (1.0, 1.0) if ldr else (_post.exposure_or_one(x["exposure"]), _post.exposure_or_one(x["exposure_prev"]))
reset = bool(x["reset"])
hist = np.asarray(x["history"], np.float64)
if not reset and hist.shape[:2] != (dh, dw):
raise ValueError(f"history is {hist.shape[1]}x{hist.shape[0]}, expected D = {dw}x{dh} (a size change needs "
"reset)")
s = float(np.asarray(s, np.float64).reshape(-1)[0])

px, py = _post.target_positions(rw, rh, dw, dh, jx, jy)
with np.errstate(invalid="ignore", over="ignore"):
ec = enc_in(e * color)
feat = np.concatenate([ec, np.asarray(x["B"], np.float64), np.asarray(x["A"], np.float64)], axis=2)
W, fbar = gather(feat, x["C"], px, py, rw, rh, window)
cbar, sig, gbar, bh, bo = fbar[..., 0:3], fbar[..., 3:6], fbar[..., 6], fbar[..., 7], fbar[..., 8]

# 3. motion candidates (Model M's rules)
fy, fx = _post.point_index(px, py, rw, rh)
hi = bool(x.get("motion_output_res", False))
nan_valid = bool(x.get("nan_position_valid", hi))
first, second = _post.select_two(sig[..., 0], sig[..., 1], sig[..., 2])
sel = lambda arr, k: np.take_along_axis(arr, k[..., None], -1)[..., 0] # noqa: E731
with np.errstate(invalid="ignore"):
q = _post.clamp(_post.SCORE_SLOPE * sig - _post.SCORE_BIAS, 0.0, 1.0)
w1, w2 = _post.candidate_weights(sel(q, first), sel(q, second))

# 4. history along each candidate
X = np.arange(dw)[None, :] + 0.5
Y = np.arange(dh)[:, None] + 0.5
with np.errstate(invalid="ignore", over="ignore"):
if lin_fallback:
fallback = color[fy, fx] # linear colour, no exposure
else:
fallback = enc(e_prev * color[fy, fx]) # Model M: encoded domain

def along(qx, qy):
"""(candidate [dh, dw, 3] in the combination's domain, mask of NaN positions that count as valid)."""
with np.errstate(invalid="ignore"):
valid = (~np.bool_(reset)) & (qx >= 0) & (qx <= dw) & (qy >= 0) & (qy <= dh)
nanq = (np.isnan(qx) | np.isnan(qy)) & np.bool_(nan_valid and not reset)
hs = _post.history_sample(hist, np.where(valid, qx, 0.5), np.where(valid, qy, 0.5)) if not reset else 0.0
if linear:
with np.errstate(invalid="ignore", over="ignore"):
hs = dec(hs) / e_prev
return np.where(valid[..., None], hs, fallback), nanq

if hi:
qx, qy = _motion_hi.recon_positions(x["motion"], x["motion_size"], (dw, dh), x["mv_scale"],
x.get("mv_bias", (0.0, 0.0)), x.get("motion_origin", (0, 0)),
x.get("motion_texels"))
h1, nan_h = along(qx.astype(np.float64), qy.astype(np.float64))
h2 = h1
else:
m = [np.asarray(x["m_near"], np.float64)[fy, fx], _post.bilinear(x["m_center"], px, py),
np.asarray(x["m_far"], np.float64)[fy, fx]]
with np.errstate(invalid="ignore"):
al = [along(X + mk[..., 0], Y + mk[..., 1]) for mk in m]
Hk = np.stack([a[0] for a in al], axis=-1)
nk = np.stack([a[1] for a in al], axis=-1)
h1 = np.take_along_axis(Hk, first[..., None, None], -1)[..., 0]
h2 = np.take_along_axis(Hk, second[..., None, None], -1)[..., 0]
nan_h = sel(nk, first) | ((w2 != 0) & sel(nk, second))
with np.errstate(invalid="ignore", over="ignore"):
H = np.where((w2 == 0)[..., None], w1[..., None] * h1, w1[..., None] * h1 + w2[..., None] * h2)

# 5. blend
C_lin = dec(cbar)
H_lin = H * e if linear else dec(H) / e_prev * e
alpha = _post.clamp(s * W * gbar, 0.0, 1.0)
bh2 = (alpha * (1.0 - bh) + bh)[..., None]
bo2 = (alpha * (1.0 - bo) + bo)[..., None]
Hh = H_lin + (C_lin - H_lin) * bh2
Oh = H_lin + (C_lin - H_lin) * bo2
Hh = np.where((np.isnan(Hh.sum(-1)) | nan_h)[..., None], C_lin, Hh)
Oh = np.where((np.isnan(Oh.sum(-1)) | nan_h)[..., None], C_lin, Oh)

# 6. outputs (Model M's)
hw, hh = dw // 2, dh // 2
M = Hh[:2 * hh, :2 * hw].reshape(hh, 2, hw, 2, 3).sum(axis=(1, 3)) / 4.0
# Reduced codes: HDR k = E(.) / 2.45, LDR k = the value. "display": HDR the E value clamped to [0, 2.45] (Model
# M's convention); LDR the unpacked 11/11/10-bit codes the output stage reads ("display_codes": the codes).
if ldr:
code = np.rint(_post.clamp(M, 0.0, 1.0) * 1023.0)
else:
code = np.rint(_post.clamp(enc(M) / _post.REDUCED_SCALE, 0.0, 1.0) * 1023.0)
if ldr:
codes = ldr_display_codes(Oh)
disp = ldr_unpack(codes)
else:
codes = None
disp = _post.clamp(enc(Oh), 0.0, _post.DISPLAY_MAX)
return {"history": _post.f16(enc(Hh)), "reduced": (code / 1023.0).astype(np.float32),
"display": _post.f16(disp),
"_lin": {"H_hat": Hh, "O_hat": Oh, "W": W, "first": first, "second": second, "w1": w1, "w2": w2},
**({"display_codes": codes.astype(np.uint16)} if codes is not None else {})}
45 changes: 37 additions & 8 deletions clean/l/reference/table.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
"""
import copy

import numpy as np

from clean.reference import swin as _swin

# ---- stages shared with Model M -------------------------------------------------------------------------------------
Expand Down Expand Up @@ -75,10 +77,18 @@
{"op": "pick", "src": "y2", "values": L_DEC_VALUES},
),
"dec_pixel": "l",
"post_variant": "unfolded", # I8
# Reconstruction variants and the rule that picks one per frame from the render and output sizes ("rule" names
# the comparison, "ratio" its factor). Only the 3/2 form (D = floor((3 O + 1) / 2)) exists in WrongGPU so far.
"recon": {"rule": "both_axes_at_least", "ratio": 3, "variants": ("3/2", "3/1")}, # I7
# The reconstruction (clean/l/reference/recon.py): Model M's (spec 06) with a "window" x "window" colour gather,
# history candidates combined in linear colour ("history_combine"), and the plain linear colour as the fallback
# of an invalid or reset sample ("fallback"). Model M: window 3, "encoded", "encoded_colour". The output stage
# (D -> output filter) is Model M's unchanged. "post_variant" names Model L's kernel set (one function, both
# tilings).
"post_variant": "unfolded",
# Which tiling (both compute the same function) a frame takes: with D = ((3 ow + 1) div 2, (3 oh + 1) div 2) and
# the render size R (not padded), "3/1" when float32(Dw) / float32(rw) >= ratio AND float32(Dh) / float32(rh) >=
# ratio (single-precision division and comparison), else "3/2" (recon_variant). The ratio is D / render, not
# output / render: 3.0 here is about output >= 2 x render.
"recon": {"rule": "d_over_render_at_least", "ratio": 3.0, "variants": ("3/2", "3/1"), "window": 4,
"history_combine": "linear", "fallback": "linear_colour"},
# Window offsets (ox, oy) of an L block for frame parity 0 and 1, in pixels; the blocks run on the padded grid F
# and keep every grid position's output.
"window_offsets": {"first": ((2, 2), (0, 2)), "second": ((0, 0), (2, 0))},
Expand All @@ -88,10 +98,29 @@

DEC_PIXEL_VALUES = {"l": len(L_DEC_VALUES), "m": len(M_DEC0_VALUES)}

PROVISIONAL = {
"I7": "recon: the 3/1 form and the exact rule",
"I8": "post_variant",
}
PROVISIONAL = {}

# Model M's reconstruction in the same terms (clean/reference/post.py reconstruct): for reductions and tests.
M_RECON = {"rule": "d_over_render_at_least", "ratio": 3.0, "variants": ("3/2", "3/1"), "window": 3,
"history_combine": "encoded", "fallback": "encoded_colour"}


def recon_size(output_w, output_h):
"""D = ((3 ow + 1) div 2, (3 oh + 1) div 2) (spec 01)."""
return (3 * int(output_w) + 1) // 2, (3 * int(output_h) + 1) // 2


def recon_variant(t, render_w, render_h, output_w, output_h):
"""The tiling a frame takes ("3/1" or "3/2"): float32(D) / float32(render) >= ratio on both axes, render size not
padded, single-precision division and comparison."""
rc = t["recon"]
if rc["rule"] != "d_over_render_at_least":
raise ValueError(f"unknown recon rule {rc['rule']!r}")
dw, dh = recon_size(output_w, output_h)
r = np.float32(rc["ratio"])
rx = np.float32(dw) / np.float32(render_w)
ry = np.float32(dh) / np.float32(render_h)
return rc["variants"][1] if (rx >= r and ry >= r) else rc["variants"][0]

UNCONFIRMED = {
"U1": "encoder halo: a mirrored render position below 0 (only when fw + halo > 2 rw - 2, i.e. render sizes near "
Expand Down
22 changes: 11 additions & 11 deletions clean/l/tests/test_l_map.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

The map is the source of truth. The table (clean/l/reference/table.py) and the coefficient kernel
(clean/l/kernels/l_ops.hip) hold copies of its value positions, and this test fails when they drift apart. The first
map (v1) still carried Model M's value positions, so the check needs v1.1 or later.
map revision (v1) still carried Model M's value positions, so the check needs revision 1.1 or later.
"""
import json
import os
Expand All @@ -19,15 +19,15 @@


def newest_l_map():
"""Path of the newest preset-L map at version 1.1 or later, else None."""
best = None
for f in os.listdir(DATA):
m = re.fullmatch(r"preset-L\.v(\d+)(?:\.(\d+))?\.map\.json", f)
if m:
v = (int(m.group(1)), int(m.group(2) or 0))
if v >= (1, 1) and (best is None or v > best[0]):
best = (v, os.path.join(DATA, f))
return best[1] if best else None
"""Path of the preset-L map revision that carries Model L's own value positions (revision 1.1 on, which also has
the "preset": "L" key; the file keeps its v1 name), else None."""
for f in sorted(os.listdir(DATA), reverse=True):
if re.fullmatch(r"preset-L\.v\d+(?:\.\d+)?\.map\.json", f):
path = os.path.join(DATA, f)
with open(path, encoding="utf-8") as fh:
if json.load(fh).get("preset") == "L":
return path
return None


def kernel_values():
Expand All @@ -43,7 +43,7 @@ def test_kernel_values_are_the_table(self):
def test_table_values_are_the_map(self):
path = newest_l_map()
if path is None:
self.skipTest("no preset-L map v1.1 or later in data/")
self.skipTest("no preset-L map revision 1.1 or later in data/")
with open(path, encoding="utf-8") as f:
m = json.load(f)
self.assertEqual(tuple(m["perms"]["dec0_values"]), T.L_DEC_VALUES)
Expand Down
Loading
Loading