Skip to content
110 changes: 110 additions & 0 deletions clean/l/FORMATS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Model L: kernels, maps, blobs and launch order (issue #89)

This is the interface the Model L runtime frame builds against.
- A row marked **exists** has a merged HIP kernel.
- A row marked **planned** is the interface the kernel is being written to. Its names, arguments, grids and maps are
fixed here, and a change goes through this file first.
- Each D3D12 port declares the same code object, kernel name and arguments as its HIP kernel (`clean/dx12/kernels/l_*.hlsl`).

The structure follows `clean/l/reference/table.py`. The launch list comes from `clean/l/runner/l_pack.py`
(`encoder_launches`, `decoder_launches`, with each map's `in_format` / `out_format`). The ten shared Swin stages
(enc1 ... dec1) use Model M's blobs, kernels and launch plan unchanged, with Model L's weights.

## Maps

Every map is `[rows][cols][C]`, channel fastest, as Model M's feature maps (`clean/FORMATS.md`, "Feature maps").
- FP8 = E4M3, 1 byte per channel. FP16 = 2 bytes per channel, little-endian.
- F = fw x fh is the padded grid (multiples of 32). R = rw x rh is the render size.
- **Halo maps** have 2 extra pixels on each side: `[fh + 4][fw + 4][C]`. Map pixel (i, j) is grid position
(x, y) = (j - 2, i - 2). The halo pixels hold the per-pixel chain evaluated at those positions: the 20 values at
the mirrored render position, the dither of the unmirrored position. They are what the encoder block reads for
window members outside the grid.

| Map | Shape | Element | Written by | Read by |
|---|---|---|---|---|
| `q` | `[fh + 4][fw + 4][32]` halo | FP8: values 0..19, bytes 20..31 zero | `lev/input_encoder` | `l_linear_32_32_relu` |
| `x1` | `[fh + 4][fw + 4][32]` halo | FP8 | `l_linear_32_32_relu` | `l_linear_32_32_f16` |
| `x2` | `[fh + 4][fw + 4][32]` halo | **FP16** (64 bytes per pixel) | `l_linear_32_32_f16` | `l_swin_block_c32_enc` |
| `x3` | `[fh][fw][32]` | FP8 | `l_swin_block_c32_enc` | `l_project_gather` |
| `enc0.features` | `[fh/2][fw/2][64]` | FP8 | `l_project_gather` | enc1 (Model M's stage) |
| m_near, m_far, m_center | as Model M | as Model M | `lev/input_encoder` | as Model M |
| `y` | `[fh][fw][32]` | FP8 | `l_dec_project` | `l_swin_block_c32_dec` |
| `y2` | `[fh][fw][32]` | **FP16** (64 bytes per pixel) | `l_swin_block_c32_dec` | `l_coef` |
| A, B, C, new latent | as Model M's `coef_decoder` outputs | as Model M | `l_coef` | as Model M |

Extra bytes per frame:
- 32 (fh + 4)(fw + 4) for `q`;
- 32 (fh + 4)(fw + 4) for `x1`;
- 64 (fh + 4)(fw + 4) for `x2`;
- 32 fh fw for `x3`;
- 32 fh fw for `y`;
- 64 fh fw for `y2`.

No kernel works in place.
- `x2` may reuse `q`'s memory (`q` is dead after `l_linear_32_32_relu`), if it is sized for `x2`.
- `y` and `y2` may reuse the encoder maps, which are dead after `l_project_gather`.

## Blobs (`l_pack.pack_model_l`, names = the converter map's stage / tensor names)

| Blob | Format |
|---|---|
| `enc0.values` | Model M's enc0 blob (`clean/runner/enc0.py` pack) with W = 0 and b = 0: dither table T, sigma1, sigma2 (the map's stored values) |
| `enc0.Win1`, `enc0.Win2` | merge / expand section format: bias tile vector (rows R_L(32)), then fragments [mt < 2][kc < 2], rows R_L(32), K order K_L(32). `Win1` has zero weights for inputs 20..31 |
| `enc0_block`, `dec0_block` | Model M's Swin block section (`packing.pack_block`) at C = 32, 2 heads, MLP 128, padded to 256 |
| `enc0.project` | bias tile vector (rows R_L(64)), then fragments [mt][kb], rows R_L(64). K order: block 2i + s of group i holds channel 32 i + 16 h + 8 s + j (dec0's order, `clean/runner/dec0.py` `kord_decoder`, over 4 groups) |
| `dec0.project` | bias tile vector (rows R_E(32)), then fragments [mt < 8][kb < 4], rows R_E(32) (tile 2 s + g: sub-pixel s, channels 16 g ...). K order as `enc0.project` (two groups of 32) |
| `dec0.coef` | Model M's dec0 blob (`clean/runner/dec0.py` pack) with W = 0 and b = 0: the scalars S and T |

## Code objects

| File | Source | Kernels |
|---|---|---|
| `l_enc_values.co` (loaded under the tag `lev`) | `clean/kernels/encoder.hip` with `-DWG_ENC_VALUES_OUT=1` | `input_encoder` |
| `l_ops.co` | `clean/l/kernels/l_ops.hip` | `l_linear_32_32_relu`, `l_linear_32_32_f16`, `l_project_gather`, `l_dec_project`, `l_coef` |
| `l_swin.co` | `clean/l/kernels/l_swin.hip` | `l_swin_block_c32[_nc]` (exists, #102: FP8 in / FP8 out, mirror border; block checks only), `l_swin_block_c32_enc[_nc]`, `l_swin_block_c32_dec[_nc]` |

All are built with `clean/runner/build.py`'s `device_command`. The build's `-cuid` comes from the output file name,
so compare code objects only between outputs of the same name. `_nc` is used when bit 0 of the block blob's first
word is set, as for Model M's blocks. D3D12 ports exist for the `_nc` forms only.

## Kernels

Workgroups are 256 lanes (8 waves of 32) throughout. "Per-pixel" kernels handle 128 pixels (or tokens) per
workgroup, in raster order of their map.

| Kernel | Status | Arguments | Grid | Does |
|---|---|---|---|---|
| `lev/input_encoder` | planned (rework of the old values-only variant) | as Model M's `input_encoder`, with `features` replaced by `q` | ceil(((fh + 4) / 2) ((fw + 4) / 2) / 128) (2 x 2 pixel groups of the halo map) | the 20 values and their dithered FP8 for every halo-map pixel, into `q`. The motion surfaces as Model M, only for pixels inside R. w = `enc0.values` |
| `l_linear_32_32_relu` | planned | `x, y, w, npix` | ceil(npix / 128), npix = (fh + 4)(fw + 4) | y = q8(max(0, RN16(b + W x))), FP8 -> FP8. w = `enc0.Win1` |
| `l_linear_32_32_f16` | planned | `x, y, w, npix` | as above | y = RN16(b + W x), FP8 -> **FP16**. w = `enc0.Win2` |
| `l_swin_block_c32_enc[_nc]` | planned | `x, xadd (null), out, w, H, W, ox, oy` | ceil(nwin / 8), nwin = ceil((W + ox) / 4) ceil((H + oy) / 4) | the encoder block. x = `x2` (FP16 halo map; member at grid (gx, gy) read at map pixel (gy + 2, gx + 2)). out = `x3` (FP8, grid only). H = fh, W = fw. (ox, oy) = `window_offsets["first"]` for the frame parity. w = `enc0_block` |
| `l_project_gather` | planned | `x, features, w, tw, th` | ceil(tw th / 128), tw = fw / 2, th = fh / 2 | token (r, c) gathers pixels (2r + p / 2, 2c + p % 2), p = 0..3, as channels 32 p + i. Running FP16 sum per group of 32 from the bias, clamp +-6.28125, FP8 -> `enc0.features`. w = `enc0.project` |
| `l_dec_project` | planned | `z, y, w, tw, th` | ceil(tw th / 128) | per token of dec1's output z [th][tw][64]: 128 outputs (two RN16 groups of 32 inputs, then FP8). Output 32 s + k goes to pixel (2r + s / 2, 2c + s % 2), channel k, of `y`. w = `dec0.project` |
| `l_swin_block_c32_dec[_nc]` | planned | `x, xadd (null), out, w, H, W, ox, oy` | as the encoder block | the decoder block. x = `y` (FP8). A member outside the grid takes token-grid mirrored pixels with the sub-pixel kept. out = `y2` (**FP16**). (ox, oy) = `window_offsets["second"]`. w = `dec0_block` |
| `l_coef` | planned (new) | `y2, A, B, C, latent, w, params` | ceil((fh / 2)(fw / 2) / 128), lanes as Model M's `coef_decoder` | Model L's per-pixel functions on the 13 channels `table.L_DEC_VALUES` of `y2`: A, B, C in Model M's forms, latent = tanh(candidate). Writes the same buffers and formats as `coef_decoder`, pixels inside R only. **Reads no motion, previous latent, jitter or reset.** params = Model M's `WgDecParams` (only rw, rh, fw, fh are read). w = `dec0.coef` |

## Launch order of one frame

Model M's frame (`clean/FORMATS.md`, "Stage order per frame") with steps 2 and 4 replaced. Window offsets follow
the frame parity, as Model M's.

2. Encoder:
1. `lev/input_encoder` -> `q` (halo), m_near, m_far, m_center
2. `l_linear_32_32_relu` q -> `x1` (halo)
3. `l_linear_32_32_f16` x1 -> `x2` (halo, FP16)
4. `l_swin_block_c32_enc[_nc]` x2 -> `x3`
5. `l_project_gather` x3 -> `enc0.features` (what enc1 reads)
3. enc1 ... dec1: Model M's stages with Model L's weights, unchanged.
4. Decoder:
1. `l_dec_project` dec1 output -> `y`
2. `l_swin_block_c32_dec[_nc]` y -> `y2` (FP16)
3. `l_coef` y2 -> A, B, C, new latent. Model L's latent recurrence runs only through the next frame's input
encoder, which samples this latent.

The reconstruction (post) stage is still open.

## Checks (GPU, synthetic data)

- `clean/l/runner/l_c32_check.py`: the C = 32 block against the oracle.
- `clean/l/runner/l_d3d12_check.py` (with its D3D12 port): each D3D12 port against its HIP kernel, byte for byte.
- One check per new kernel follows with that kernel.
116 changes: 99 additions & 17 deletions clean/l/reference/oracle.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,20 +82,42 @@ def window_grid(H, W, offset, win):
return rows, cols


def swin_block(x, p, offset, heads, consts, max_tokens=1 << 16):
"""One block on a whole map x [H, W, C] with mirrored borders (as clean/reference/swin.py swin_block); returns the
block output in float64, not yet rounded."""
def border_index(border, n, halo=0):
"""Grid position -> index into the block's input along one axis of n grid positions.
"mirror": about the edge pixel, edge not repeated (clean/reference/swin.py); "input": the input holds halo extra
positions on each side (position -halo is index 0); "coarse": the mirror of the 2 x 2 token grid (n / 2 tokens),
the sub-pixel kept."""
if border == "mirror":
return lambda q: _swin.mirror(q, n)
if border == "input":
def f(q):
i = np.asarray(q) + halo
if (i < 0).any() or (i >= n + 2 * halo).any():
raise AssertionError("window reaches beyond the input halo")
return i
return f
if border == "coarse":
return lambda q: 2 * _swin.mirror(np.asarray(q) // 2, n // 2) + np.asarray(q) % 2
raise ValueError(f"unknown border {border!r}")


def swin_block(x, p, offset, heads, consts, border="mirror", halo=0, max_tokens=1 << 16):
"""One block on a whole map x [H, W, C] (border "input": [H + 2 halo, W + 2 halo, C], the grid in the middle);
window members outside the grid per border_index. Returns the block output on the grid [H, W, C] in float64, not
yet rounded."""
x = np.asarray(x, dtype=np.float64)
H, W, C = x.shape
h = halo if border == "input" else 0
H, W, C = x.shape[0] - 2 * h, x.shape[1] - 2 * h, x.shape[2]
win = int(consts["window"])
p = {k: np.asarray(v, dtype=np.float64) for k, v in p.items()}
rows, cols = window_grid(H, W, offset, win)
out = np.full_like(x, np.nan)
out = np.full((H, W, C), np.nan)
fr, fc = border_index(border, H, h), border_index(border, W, h)
rows_per_chunk = max(1, max_tokens // (win * win * cols.shape[0]))
for l0 in range(0, rows.shape[0], rows_per_chunk):
r = rows[l0:l0 + rows_per_chunk]
rr = _swin.mirror(r, H)[:, None, :, None]
cc = _swin.mirror(cols, W)[None, :, None, :]
rr = fr(r)[:, None, :, None]
cc = fc(cols)[None, :, None, :]
xw = x[rr, cc]
nl, nk = xw.shape[:2]
x2 = _block_tokens(xw.reshape(nl, nk, win * win, C), p, heads, consts).reshape(nl, nk, win, win, C)
Expand Down Expand Up @@ -168,9 +190,9 @@ def _run_ops(ops, maps, params, parity, t, default_stage, frame=None):
st = params[op.get("params", default_stage)]
if k == "quant":
v = maps[op["src"]]
fw, fh = frame["feature_size"]
if op["mode"] == "dither":
d = _enc.dither_table(st["T"], frame["frame"], fw, fh)
h = maps.get("_halo", 0)
d = dither_table(st["T"], frame["frame"], v.shape[1], v.shape[0], h)
out = _enc.dither_fp8(v, d[..., None])
elif op["mode"] == "q8":
out = q8(v)
Expand All @@ -187,7 +209,10 @@ def _run_ops(ops, maps, params, parity, t, default_stage, frame=None):
elif k == "swin":
consts = _table.block_consts(t, op["params"])
off = _table.block_offset(t, op["offsets"], parity)
y = swin_block(maps[op["src"]], st, off, op["heads"], consts)
border = op.get("border", "mirror")
y = swin_block(maps[op["src"]], st, off, op["heads"], consts, border, maps.get("_halo", 0))
if border == "input":
maps["_halo"] = 0
out = _round(y, op.get("out_round", "q8"))
elif k == "concat":
parts = [maps[s] for s in op["srcs"]]
Expand All @@ -209,18 +234,45 @@ def _run_ops(ops, maps, params, parity, t, default_stage, frame=None):
return last


def dither_table(T, frame, w, h, halo=0):
"""t = T[n][|y| mod 16][|x| mod 16] at grid positions x = -halo ... w - halo - 1 (y likewise), [h, w]."""
T = np.asarray(T)
Y, X = np.mgrid[0:h, 0:w] - halo
return T[int(frame)][np.abs(Y) % 16, np.abs(X) % 16]


def encoder_halo(t):
"""Extra positions per side the encoder's per-pixel ops run on (a block with border "input"), else 0."""
hs = [_table.block_halo(t, op["offsets"]) for op in t["enc_ops"]
if op["op"] == "swin" and op.get("border") == "input"]
return max(hs, default=0)


def encode(x, params, parity=None, t=_table.TABLE, keep=False):
"""Model L's input encoder. x: Model M's enc0 input dict; params: {stage: tensors} with the stages of
table.param_shapes(t) ("enc0", "enc0_block"). Returns Model M's enc0 outputs ("features" [fh/2, fw/2, 64]
FP8-exact, motion surfaces) and, with keep=True, every named map ("maps")."""
_table.encoder_plan(t)
if parity is None:
parity = int(x["frame"]) & 1
_enc.check_inputs(x)
rw, rh = _enc.check_inputs(x)[:2]
enc = params["enc0"]
pv_params = {"sigma1": enc.get("sigma1", 0.0), "sigma2": enc.get("sigma2", 0.0)}
maps = {"values": _enc.pixel_values(x, pv_params)}
v = _enc.pixel_values(x, pv_params)
h = encoder_halo(t)
if h:
# The 20 values depend only on the mirrored render position (clean/reference/encoder.py reflect), which is
# a grid position, so the halo copies grid values; the dither is applied later at the unmirrored position.
# Only for render sizes near the padding (fw + h > 2 rw - 2) would the mirrored position be negative; reflect
# clamps it to 0 there (table.UNCONFIRMED "U1").
fh, fw = v.shape[:2]
ry = _enc.reflect(np.arange(-h, fh + h), rh)
rx = _enc.reflect(np.arange(-h, fw + h), rw)
v = v[ry][:, rx]
maps = {"values": v, "_halo": h}
feats = _run_ops(t["enc_ops"], maps, params, parity, t, "enc0", frame=x)
if maps.pop("_halo"):
raise ValueError("encoder: the per-pixel halo was not consumed by a block")
m_near, m_far, m_center = _enc.motion_surfaces(x)
res = {"features": feats.astype(np.float32),
"m_near": f16(m_near).astype(np.float32),
Expand Down Expand Up @@ -262,17 +314,47 @@ def coefficients(x, a, S, T):
"codes_A": cA, "codes_B": cB, "codes_L": cL, "p": p}


def tanh32(v):
"""tanh in float64, then one rounding to FP32."""
return np.tanh(np.asarray(v, dtype=np.float64)).astype(np.float32)


def coefficients_l(a, S, T):
"""Model L's per-pixel functions on a [rh, rw, 13] (table.L_DEC_VALUES order; FP16 block outputs held in FP32):
A, B and C with Model M's forms (C's gains from inputs clamped to [-20, 20]), latent = tanh(candidate). No gate,
no previous latent. Same outputs and codes as coefficients()."""
a = np.asarray(a, dtype=np.float32)
A = _dec.sigmoid16(a[..., 0:3])
B = _dec.softmax3(a[..., 3:6])
g = _dec.sigmoid16(np.clip(a[..., 6:8], np.float32(-20), np.float32(20)))
sx, sy = np.float32(S) * g[..., 0], np.float32(S) * g[..., 1]
rho = _dec.rho32(a[..., 8], float(np.asarray(T)))
C = np.stack([_dec.rn16(sx * sx), _dec.rn16(sy * sy), _dec.rn16((sx * sy) * rho)], axis=-1)
lat = tanh32(a[..., 9:13])
cA, cB, cL = _dec.unorm8_code(A), _dec.unorm8_code(B), _dec.snorm8_code(lat)
return {"A": (cA / 255.0).astype(np.float32), "B": (cB / 255.0).astype(np.float32),
"C": C.astype(np.float32), "latent": (cL / 127.0).astype(np.float32),
"codes_A": cA, "codes_B": cB, "codes_L": cL}


def decode(x, params, parity, t=_table.TABLE, keep=False):
"""Model L's coefficient decoder. x: Model M's dec0 input dict (features = dec1's output [fh/2, fw/2, 64]);
params: {stage: tensors} ("dec0", "dec0_block")."""
if t["dec_pixel"] != "m":
raise NotImplementedError("decoder per-pixel variant other than Model M's: pending")
"""Model L's coefficient decoder. x: Model M's dec0 input dict (features = dec1's output [fh/2, fw/2, 64]; with
dec_pixel "l" the motion / latent_prev / jitter / reset entries are not read); params: {stage: tensors} ("dec0",
"dec0_block")."""
kind = t["dec_pixel"]
if kind not in _table.DEC_PIXEL_VALUES:
raise NotImplementedError(f"decoder per-pixel variant {kind!r}: not known")
_table.decoder_plan(t)
if len(t["dec_ops"][-1]["values"]) != _table.DEC_PIXEL_VALUES[kind]:
raise ValueError(f"dec_pixel {kind!r} needs {_table.DEC_PIXEL_VALUES[kind]} picked values")
rw, rh = _dec.check_inputs(x)[:2]
maps = {"features": np.asarray(x["features"], dtype=np.float64)}
a = _run_ops(t["dec_ops"], maps, params, parity, t, "dec0")
dec = params["dec0"]
out = coefficients(x, a[:rh, :rw], dec["S"], dec["T"])
if kind == "l":
out = coefficients_l(a[:rh, :rw], dec["S"], dec["T"])
else:
out = coefficients(x, a[:rh, :rw], dec["S"], dec["T"])
out["a"] = a[:rh, :rw].astype(np.float32)
if keep:
out["maps"] = maps
Expand Down
Loading
Loading