Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions clean/dx12/kernel_sets.json
Original file line number Diff line number Diff line change
Expand Up @@ -336,6 +336,34 @@
"k_stage_e2",
"k_block.co"
],
[
"k_stagew2_d3",
"k_block_wide.co"
],
[
"k_stagew2_d4",
"k_block_wide.co"
],
[
"k_stagew2_e3",
"k_block_wide.co"
],
[
"k_stagew2_e4",
"k_block_wide.co"
],
[
"k_stagew4_d4",
"k_block_wide.co"
],
[
"k_stagew4_e4",
"k_block_wide.co"
],
[
"k_stagew4_m",
"k_block_wide.co"
],
[
"k_warm",
"k_block_wide.co"
Expand Down
4 changes: 3 additions & 1 deletion clean/k/dx12/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,9 @@ Families: `narrow` (the narrow network stages of `k_block.co`: `k_stage_e2`, `k_
`k_stage_d1`, `k_stage2_e0`, `k_stage2_e1`, `k_stage2_d1`, `k_stage2_d0_hp`, `k_stage2_d0_hidden`) on random stage weights
and inputs, the stage runner's launch plans run on HIP and replayed on D3D12 (`k_plan_bridge.py`); the inputs of every
third case carry special values (infinity, +-65504, zero, -0, subnormals) and of every third case NaN as well; the
half-grid control launches a stage with half its workgroups. `expo` (`k_expo_scalars`) against `k_io.co`; `warm`
half-grid control launches a stage with half its workgroups. `wide` (the wide stages of `k_block_wide.co`:
`k_stagew2_e3`, `k_stagew4_e4`, `k_stagew2_e4`, `k_stagew4_m`, `k_stagew4_d4`, `k_stagew2_d4`, `k_stagew2_d3`: one window
per 128 NG lanes) is checked the same way. `expo` (`k_expo_scalars`) against `k_io.co`; `warm`
(`k_warm`, a cache warm-up with no HIP kernel: checked against its definition, through the struct argument path).

NaN: IEEE 754 does not define the sign or payload of a NaN, and where a NaN is generated or propagated the two devices
Expand Down
198 changes: 198 additions & 0 deletions clean/k/dx12/hlsl/k_wide.hlsli
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
// SPDX-License-Identifier: MIT
// D3D12 port of model K's wide block (clean/k/kernels/k_block_wide.hip, template k_block_wide<C, HEADS, KMIX_SOFTMAX,
// IN, CIN, CNEXT, NG>, FP32 accumulation; HIP build: k_block_wide.co). WrongGPU's own code. Spec 41 block (spec 40 stage
// table), exactly the HIP kernel's operations and orders:
// - every matrix product is v_wmma_f32_16x16x16_f16 (driver channel, wg_wmma_f16) with the same A / B operands, and
// every FP32 accumulator sums in the HIP order (bias first; q / k / v over kt; scores bias then ct; z over st; Wo
// over the z tiles of heads 0, 1, ..; W1 over kt; W2 over hidden tiles 0, 1, ..; merge / expand over kt), rounded
// once to FP16 to nearest even by the driver's matrix conversion (wg_to_f16_ags, op 0x32: RNE);
// - every FP16 elementwise step is one IEEE FP16 operation (packed half2 here, v_pk_* as HIP). The driver's compiler
// fuses an FP16 multiply into a following add / sub (v_fma_mix / v_pk_fma) and regroups multiplication chains even
// under -Gis (wg_math.hlsli "driver exceptions"), so every multiply whose result feeds an add, a sub or another
// multiply goes through kw_opq (an integer xor with a zero the compiler cannot see): the activation (b |t|, t p),
// the softmax polynomial (u^2, kappa u^2), the normalization (f16(x^2), r g1);
// - normalization: s = f16(sum f16(x_c^2)) (FP32 sum in channel order, then the lane ^ 16 half), r = 0 for s = 0 else
// RN16(v_rsq_f32(s)) (wg_f2h: RNE; the plain f32tof16 rounds toward zero on this driver);
// - softmax weights: w = v_exp_f32(g), FP32 sum in key order plus the lane ^ 16 half, inv = v_rcp_f32(sum), and
// A = f16(w inv) rounded ONCE from the exact product, as HIP's compiled form (v_fma_mixlo/hi_f16 w * inv + 0, seen in
// the k_block_wide.co ISA): the FP32 product rounded to odd with its exact residual (wg_round_odd), then the RNE
// conversion (FP32 has 13 more bits than FP16: no double rounding);
// - mirrored window members, ownership of the output tiles (a group without a last tile computes tile KT - 1 too and
// does not store it), merge / expand index maps: as HIP.
// Deviations (no effect on any result): weights and LDS operands of each product group are loaded before its multiplies
// (driver channel calls are UAV atomics the compiler never moves loads across): every global weight fragment comes from
// one ring of KW_RD tiles that runs through the whole kernel in product order, across phase borders (the weight stream,
// k_block_wide.hlsl), each loaded through a per-lane base address plus a constant (the load's immediate offset);
// independent accumulators are interleaved (the scores' four key tiles, z's two channel tiles, W1's MM hidden tiles,
// the expand's tiles), each still summing in its own order; the LDS indices are masked to the array's power of two
// (kw_lr / kw_lw); the residual x of the owned tiles is reloaded at step 3 for map inputs; the softmax round-to-odd uses
// the positive-product form (kw_round_odd_pos, same bits); NG = 4 uses another LDS layout that fits D3D12's 32 KB (HIP: 56 KB): K / VT per group unpadded with an xor
// swizzle of the 16-byte chunks, and the round's Z, the MLP's A and its H buffers all start at 0, with extra barriers
// where the phases now overlap (K / VT reads done before Z is written, Z reads done before A / the next round's K / VT,
// A reads done before H, H reads done before the merge's x2). The arithmetic is the same in both layouts.
//
// Compiled once per stage by clean/dx12/build_dx12.py with the defines of the @wg-kernel line (k_block_wide.hlsl):
// KW_C channels, KW_HEADS heads, KW_IN 0 = map input / 1 = expand input, KW_CIN expand input channels, KW_CNEXT merge
// output channels (0: no merge), KW_NG wave groups (block = 128 KW_NG lanes).
// Arguments: KBlkParams (184 bytes, struct token s184:p19.b32: 19 pointers as arena offsets, then 8 ints).
#pragma once

static const int KT = KW_C / 16; // channel tiles of a token
static const int DT = KW_HEADS * 2; // channel tiles of z
static const int HT = 4 * KW_C / 16; // hidden tiles of the MLP
static const int NG = KW_NG;
static const int NW = 4 * NG; // waves of the workgroup
static const int NOWN = (KT + NG - 1) / NG; // output tiles a group owns (at most)
static const int MM = 2; // hidden tiles per group and chunk
static const int CH = NG * MM; // hidden tiles per chunk
static const int KTI = KW_CIN / 16; // expand input tiles
static const int TOT = 4 * KW_C / 16; // expand output tiles
static const int KTM = 4 * KW_C / 16; // merge input tiles
static const int NTM = KW_CNEXT / 16; // merge output tiles
#ifndef KW_TIGHT
#define KW_TIGHT (KW_NG == 4)
#endif
static const bool TIGHT = KW_TIGHT; // the overlapping layout (see above; required for NG = 4)

// LDS layout in halfs (HIP's for NG = 2; the overlapping one for NG = 4)
static const int KVH = TIGHT ? 4096 : 64 * 40 + 32 * 72; // one group's K [64][32|40] + VT [32][64|72]
static const int ZS = 32 * NG + 8;
static const int OFF_Z = TIGHT ? 0 : NG * KVH;
static const int ATT = TIGHT ? (NG * KVH > 64 * ZS ? NG * KVH : 64 * ZS) : NG * KVH + 64 * ZS;
static const int OFF_A = TIGHT ? 0 : (64 * KW_C <= NG * KVH ? 0 : ATT);
static const int OFF_H = TIGHT ? 0 : OFF_A + 64 * KW_C;
static const int END_H = OFF_H + 2 * CH * 64 * 16;
static const int LDS_HALFS0 = ATT > END_H ? ATT : END_H;
static const int LDS_HALFS = LDS_HALFS0 > 64 * KW_C ? LDS_HALFS0 : 64 * KW_C;
groupshared uint4 kw_lds[LDS_HALFS / 8]; // 16-byte chunks (every access is one chunk)

// ring depth (tiles) of the merge when a wave has more than one merge output tile (4K NG = 2 e4); everything else uses
// the weight stream's ring (KW_RD, k_block_wide.hlsl)
#ifndef KW_PD
#define KW_PD 4
#endif

// ---- helpers --------------------------------------------------------------------------------------------------------
// 0 that the compiler cannot see, with no known bits (grids are below 2^31; 0 or 0xffffffff for the compiler: a 0 / 1
// value would leave the high FP16 half of a packed pair visibly unchanged, and the driver then fused it anyway)
uint kw_zero() { return 0u - (wg_grid_ >> 31); }
uint kw_opq(uint v) { return v ^ kw_zero(); }
uint4 kw_opq4(uint4 v) { return v ^ kw_zero(); }
half2 kw_h(uint v) { return wg_as_h2(v); }
uint kw_u(half2 h) { return wg_h2_bits(h); }

uint4 kw_add8(uint4 a, uint4 b)
{
uint4 r;
[unroll] for (uint i = 0; i < 4; ++i) r[i] = kw_u(kw_h(a[i]) + kw_h(b[i]));
return r;
}
uint4 kw_mul8(uint4 a, uint4 b)
{
uint4 r;
[unroll] for (uint i = 0; i < 4; ++i) r[i] = kw_u(kw_h(a[i]) * kw_h(b[i]));
return r;
}
uint4 kw_splat8(uint h) { const uint d = (h & 0xffffu) | (h << 16); return uint4(d, d, d, d); }

// spec 40 activation, FP16 per operation: t = min(max(x, -2), 2); p = f16(a - f16(b |t|)); q = f16(0.5 + f16(t p));
// act = f16(x q) (k_common.h wgk_act2; maxNum / minNum clamps)
uint4 kw_act8(uint4 x)
{
const half2 lo = half2(-2.0h, -2.0h), hi = half2(2.0h, 2.0h);
const half2 A = half2(0.412109375h, 0.412109375h), B = half2(0.0810546875h, 0.0810546875h);
const half2 half_ = half2(0.5h, 0.5h);
uint4 r;
[unroll] for (uint i = 0; i < 4; ++i) {
const half2 xv = kw_h(x[i]);
const half2 tv = min(max(xv, lo), hi);
const half2 bt = kw_h(kw_opq(kw_u(B * abs(tv))));
const half2 p = A - bt;
const half2 tp = kw_h(kw_opq(kw_u(tv * p)));
const half2 q = half_ + tp;
r[i] = kw_u(xv * q);
}
return r;
}

// softmax polynomial of one score fragment: u = clamp(s, -c, c), g = f16(u f16(1 - f16(kappa f16(u u))))
uint4 kw_softpoly8(uint4 s)
{
const half2 c = half2(21.640625h, 21.640625h), nc = half2(-21.640625h, -21.640625h);
const half2 kap = (half2)asfloat16((uint16_t2)0x11d5u), one = half2(1.0h, 1.0h); // 0.0007119178771972656
uint4 r;
[unroll] for (uint i = 0; i < 4; ++i) {
const half2 u = min(max(kw_h(s[i]), nc), c);
const half2 u2 = kw_h(kw_opq(kw_u(u * u)));
const half2 k2 = kw_h(kw_opq(kw_u(kap * u2)));
const half2 o = one - k2;
r[i] = kw_u(u * o);
}
return r;
}

// wg_round_odd (wg_math.hlsli) for a product p = RN32(w inv) that is positive, finite and normal, with its exact residual
// e: the softmax weights (w = v_exp_f32 of a clamped FP16 polynomial value in about [-14.4, 14.4], inv = v_rcp_f32 of a
// sum of 64 such w, so p >= ~3e-11). There wg_round_odd's finite / nonzero test always holds and "one ulp toward x" is
// +1 for e > 0 and -1 for e < 0 (p's sign bit is 0): the same bits in fewer instructions.
uint kw_round_odd_pos(float p, float e)
{
const uint ur = asuint(p);
return ur + ((e != 0.0f && (ur & 1u) == 0u) ? (uint)((asint(e) >> 31) | 1) : 0u);
}

v8f kw_widen8(uint4 a)
{
v8f r;
[unroll] for (uint i = 0; i < 4; ++i) {
r.v[2 * i] = f16tof32(a[i] & 0xffffu);
r.v[2 * i + 1] = f16tof32(a[i] >> 16);
}
return r;
}
uint4 kw_out(v8f c) { return wg_to_f16_ags(c); }
v8f kw_mma(uint4 a, uint4 b, v8f c) { return wg_wmma_f16(a, b, c); }

// The value of v computed in the current (wave-uniform) control flow: a lane read at lane ^ z with z the opaque zero is
// a convergent operation the compiler cannot simplify, so it cannot sink the producer of v (a WMMA chain) into a
// following divergent branch. Without it the merge's whole product chain moved under its store's `if (inside)`, and
// the WMMAs then ran with a partial EXEC mask (wrong results; the driver channel needs uniform control flow).
uint4 kw_pin(uint4 v)
{
const uint l = WaveGetLaneIndex() ^ (kw_zero() & 31u);
return uint4(WaveReadLaneAt(v.x, l), WaveReadLaneAt(v.y, l), WaveReadLaneAt(v.z, l), WaveReadLaneAt(v.w, l));
}

// global fragments: pointer (arena byte offset) + element (half) offset
uint4 kw_ldg(uint p, uint e) { return wg_ld128(p, e * 2u); }
void kw_stg(uint p, uint e, uint4 v) { wg_st128(p, e * 2u, v); }

// Mirror about the edge token, edge not repeated (spec 41; k_common.h wgk_mirror), branch free.
int kw_mirror(int p, int n)
{
const int period = max(2 * (n - 1), 1);
int r = p % period;
r = r < 0 ? r + period : r;
r = r < n ? r : period - r;
r = (p >= 0 && p < n) ? p : r;
return n == 1 ? 0 : r;
}

// LDS chunk indices. Every index goes through kw_lr / kw_lw, masked to the power of two at or above the array size
// (no effect: every index is inside the array). The known bound lets the driver prove the byte offsets do not wrap, so
// it keeps 16-byte fragments as ds_load_b128 / ds_load_2addr_b64 instead of splitting them into 2addr_b32 pairs.
static const uint KW_LN = LDS_HALFS / 8;
static const uint KW_LM = (KW_LN > 1024u ? 2048u : KW_LN > 512u ? 1024u : KW_LN > 256u ? 512u : 256u) - 1u;
uint4 kw_lr(uint i) { return kw_lds[i & KW_LM]; }
void kw_lw(uint i, uint4 v) { kw_lds[i & KW_LM] = v; }

uint kw_kidx(int g, int row, int c) // K[row = token][chunk c of 4] of group g
{
return TIGHT ? (uint)(g * 512 + row * 4 + (c ^ (row & 3))) : (uint)(g * (KVH / 8) + row * 5 + c);
}
uint kw_vidx(int g, int row, int c) // VT[row = channel][chunk c of 8 = tokens 8 c ..] of group g
{
return TIGHT ? (uint)(g * 512 + 256 + row * 8 + (c ^ (row & 7))) : (uint)(g * (KVH / 8) + 320 + row * 9 + c);
}
uint kw_zidx(int t, int c) { return (uint)(OFF_Z / 8 + t * (ZS / 8) + c); }
uint kw_aidx(int t, int c) { return (uint)(OFF_A / 8 + t * (KW_C / 8) + c); }
43 changes: 30 additions & 13 deletions clean/k/dx12/k_port_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
the half-grid control (the port again with half its workgroups) is used by the families whose grid has more than one
workgroup;
- Model M's scale_rgb (frame.co, its port already proven) runs through the same replay as a control.
The families: narrow = the narrow network stages (k_block.co: k_stage_e2 .. k_stage2_d0_hidden; the stage runner's
The families: wide = the wide network stages (k_block_wide.co: k_stagew<NG>_e3 .. _d3), as narrow; narrow = the narrow network stages (k_block.co: k_stage_e2 .. k_stage2_d0_hidden; the stage runner's
launch plans, clean/k/dx12/k_plan_bridge.py), on random stage weights and inputs with special values (NaN, infinity,
+-65504, zero, -0, subnormal) in every other case; expo = k_expo_scalars against k_io.co (HIP); warm = k_warm (D3D12 only, no HIP kernel: it is checked
against its own definition, the dummy word is written exactly when the key equals a lane's loaded value).
Expand Down Expand Up @@ -49,7 +49,9 @@
NARROW_KERNELS = ["k_stage_e2", "k_stage_e0", "k_stage_e1", "k_stage_d2", "k_stage_d1", "k_stage2_e0", "k_stage2_e1",
"k_stage2_d1", "k_stage2_d0_hp", "k_stage2_d0_hidden"]
FAMILY_KERNELS = {"expo": (["k_io.co"], ["k_expo_scalars"]), "warm": ([], ["k_warm"]),
"narrow": (["k_block.co"], NARROW_KERNELS)}
"narrow": (["k_block.co"], NARROW_KERNELS),
"wide": (["k_block_wide.co"], ["k_stagew2_e3", "k_stagew4_e4", "k_stagew2_e4", "k_stagew4_m",
"k_stagew4_d4", "k_stagew2_d4", "k_stagew2_d3"])}
CONTROL_CO = "frame.co"


Expand Down Expand Up @@ -321,11 +323,21 @@ def narrow_inputs(stage, size, rng, specials):
return x, skip


# the wide stage kernels (k_block_wide.co: k_stagew<NG>_<stage>, 128 NG lanes per window) the runtime launches:
# kernel -> (stage, stage-runner suffix, None)
WIDE = {"k_stagew2_e3": ("e3", "@wide2", None), "k_stagew4_e4": ("e4", "@wide4", None),
"k_stagew2_e4": ("e4", "@wide2", None), "k_stagew4_m": ("m", "@wide4", None),
"k_stagew4_d4": ("d4", "@wide4", None), "k_stagew2_d4": ("d4", "@wide2", None),
"k_stagew2_d3": ("d3", "@wide2", None)}
# block-level grids (W, H) of the wide stages: fewer windows than a workgroup, ragged edges, more windows than the GPU runs at once
WIDE_SIZES = [(8, 8), (16, 24), (13, 9), (24, 16), (33, 17), (12, 20)]


def itemsize(name):
return 4 if name == "out" else 2


def run_narrow(build, rocm, seeds, cdir, hip, only=None):
def run_stages(table, co_name, tag, build, rocm, seeds, cdir, hip, grid_list=None, only=None):
from clean.k.reference.layout import STAGE
from clean.k.reference.network import random_stage_params
from clean.k.runner import k_launch as KL
Expand All @@ -335,14 +347,15 @@ def run_narrow(build, rocm, seeds, cdir, hip, only=None):
items, lines, neg = [], [], None
for si, seed in enumerate(seeds):
rng = np.random.default_rng(seed)
for kernel, (stage, suffix, d0) in NARROW.items():
for kernel, (stage, suffix, d0) in table.items():
if only and kernel not in only:
continue
st = STAGE[stage]
for ci in range(len(NARROW_SIZES)):
size, offset = NARROW_SIZES[ci], NARROW_OFFSETS[(ci + si) % len(NARROW_OFFSETS)]
grids = grid_list or NARROW_SIZES
for ci in range(len(grids)):
size, offset = grids[ci], NARROW_OFFSETS[(ci + si) % len(NARROW_OFFSETS)]
special = ci % 3 # 0 none, 1 specials without NaN, 2 with NaN
name = f"nar_s{si}_{kernel}_{ci}"
name = f"{tag}_s{si}_{kernel}_{ci}"
blob = k_stages.pack(stage, random_stage_params(st, rng))
x, skip = narrow_inputs(stage, size, rng, special)
pdev = PB.PlanDev(hip.h)
Expand All @@ -368,12 +381,12 @@ def run_narrow(build, rocm, seeds, cdir, hip, only=None):
sizes = {n: pdev.sizes[plan["bufs"][n]] for n in outputs}
got = {n: bytes(hip.h.download(plan["bufs"][n], sizes[n] + SLACK)) for n in outputs}
defined = {n: min(int(np.prod(sh)) * itemsize(n), sizes[n]) for n, sh in plan["outputs"]}
lines += PB.chain_lines(name, plan, pdev, lambda k: tokens[k.split(":")[-1]], cdir, ["k_block.co"],
lines += PB.chain_lines(name, plan, pdev, lambda k: tokens[k.split(":")[-1]], cdir, [co_name],
outputs)
if neg is None and plan["launches"][0][1] >= 4: # the dropped windows hold output pixels
neg = name
lines += PB.chain_lines(name, plan, pdev, lambda k: tokens[k.split(":")[-1]], cdir,
["k_block.co"], outputs, prefix="negative_", grid_div=2)
[co_name], outputs, prefix="negative_", grid_div=2)
items.append({"case": name, "kernel": kernel, "stage": stage, "size": list(size),
"offset": list(offset), "specials": special, "outputs": outputs, "hip": got,
"defined": defined})
Expand Down Expand Up @@ -450,8 +463,11 @@ def run(build, rocm, families, seeds, json_out):
state["warm"] = run_warm(build, rocm, seeds, cdir, hip)
lines += state["warm"][1]
if "narrow" in families:
state["narrow"] = run_narrow(build, rocm, seeds, cdir, hip)
state["narrow"] = run_stages(NARROW, "k_block.co", "nar", build, rocm, seeds, cdir, hip)
lines += state["narrow"][1]
if "wide" in families:
state["wide"] = run_stages(WIDE, "k_block_wide.co", "wid", build, rocm, seeds, cdir, hip, WIDE_SIZES)
lines += state["wide"][1]
# the control: Model M's scale_rgb, same replay, against HIP
rng = np.random.default_rng(seeds[0])
cn = 1000
Expand Down Expand Up @@ -504,9 +520,10 @@ def read(n):
fails.append(it["case"])
res["results"]["warm"] = {"cases": len(state["warm"][0]), "as_defined": good, "failures": fails[:5]}
ok = ok and good == len(state["warm"][0]) and good > 0
if "narrow" in state:
res["results"]["narrow"], good = check_narrow(state["narrow"], read)
ok = ok and good
for fam in ("narrow", "wide"):
if fam in state:
res["results"][fam], good = check_narrow(state[fam], read)
ok = ok and good
ctl_got = read("ctl.out")
ctl = compare(ctl_got, h[0], 1000 * 4, D3D12_SENTINEL, HIP_SENTINEL)
res["control"] = {"kernel": CONTROL, "identical": bool(ctl["identical"]), "differ": ctl["differ"],
Expand Down
Loading
Loading