Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion clean/dx12/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ as its HIP kernel (`clean/dx12/compare.py` proves it per kernel on synthetic fra
| `dxil_asm.cpp` | tool used by the build (assembles the rewritten DXIL listing) |
| `compare.py` | HIP vs hybrid D3D12, every stage output / state buffer byte for byte |
| `kernels/exposure.hlsl` | the reference port (`exposure_auto_diet`): copy its structure |
| `kernels/fg_*.hlsl` | frame generation's ports (#96, `co=fg_*.co`): groundwork, built only when `--only` names them, never by a full build (`is_frame_gen`), so the Model M / Model L sets stay as they are; checks: `clean/fg/runner/fg_d3d12_check.py` (each kernel), `clean/fg/runner/fg_net1_gpu.py` (net1's 32 steps as a `kernel_replay` chain) |
| `kernels/fg_*.hlsl` | frame generation's ports (#96, `co=fg_*.co`): groundwork, built only when `--only` names them, never by a full build (`is_frame_gen`), so the Model M / Model L sets stay as they are; checks: `clean/fg/runner/fg_d3d12_check.py` (each kernel), `clean/fg/runner/fg_net1_gpu.py` (net1's 32 steps as a `kernel_replay` chain), `clean/fg/runner/fg_net1_mma.py` (net1's fast path on the matrix instructions, fused / unfused / exact) |
| `tests/kernel_replay.cpp` | replays launches from a case file on the native D3D12 device; `chain` blocks run several launches on buffers that persist between them (a whole network, optionally timed) |

## Arithmetic mode (why build_dx12.py and nothing else)
Expand Down Expand Up @@ -71,6 +71,16 @@ The rules above hold per operation. Across operations the gfx1201 D3D12 shader c
(`n + (wg_grid_ >> 31)`; `swin_common.hlsli` `sw_zero()`), and fold LDS offsets with `sw_zero_bits()`.
* HLSL 2021 `?:`, `&&`, `||` short-circuit and become branches (one load in flight per branch): use `select()`,
`and()`, `or()` in hot code.
* **lanes past the end of the grid corrupted other lanes' matrix results** (frame generation's `fg_mpair`, 2026-11-14):
a per-lane `bool valid[]` with `pc = select(valid, p, 0)` for the pixel, then WMMAs, then `if (valid) store`. With part
of the last 16-pixel group past the end, the valid lanes got NaN in exactly the output rows whose A-fragment row sits in
an invalid lane; HIP was right. Remedy: no validity array; every lane computes a clamped pixel (`min(p, n - 1)`) and
only the store tests `p < n`. Check odd grid sizes in every matrix port.
* **`wg_f2h` is software** (~20 instructions per value; a `(half)` cast rounds toward zero here). In a hot loop round
a whole fragment with the driver conversion, `wg_to_f16_ags` (v_cvt_f16_f32, HIP's `(_Float16)` bits), and widen back
with `wg_h2f`; for 8 values that are not a matrix result (an element-wise stage), XOR each with an opaque zero first
(`fg_f2h8` in `fg_mpair.hlsl`) so the producing multiply is not folded into the conversion (see above: a folded
product turns -0 into +0). Frame generation's net1 fast path went from 3.5x to 1.5x HIP's GPU time with this.

The native helper regression additionally constructed tiny exact values that round to the smallest normal FP32
number: `0x00800000 * 0x3f7fffff` and `0x00ffffff / 0x40000000`. The multiply flushes before rounding on this driver;
Expand Down
28 changes: 28 additions & 0 deletions clean/dx12/kernels/fg_leaky_fast.hlsl
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
// SPDX-License-Identifier: MIT
// D3D12 port of fg_leaky_fast_f16 of fg_net1_mma.co (clean/fg/kernels/fg_net1_mma.hip): fg_leaky_fast16 on n float16
// values, the exhaustive check of the fast path's leaky ReLU (runner/fg_net1_mma.py). Built only when --only names it.
//
// @wg-kernel fg_leaky_fast_f16 co=fg_net1_mma.co block=256 wave=32 args=p,p,u
#include "wg_platform.hlsli"

#define FG_FN
#define FG_I64 int64_t
#define FG_FBH32(v) firstbithigh((uint)(v))
#include "../../fg/kernels/fg_exact16.h"
#define FG_F2H(f) wg_f2h(f)
#define FG_H2F(h) wg_h2f(h)
#define FG_SEL(c, a, b) select(c, a, b)
#include "../../fg/kernels/fg_fast16.h"

WG_ARGS_BEGIN
uint x, y; // pointers (arena byte offsets)
int n;
WG_ARGS_END

WG_KERNEL(fg_leaky_fast_f16)
{
WG_SETUP();
const int idx = (int)(wg_block_id() * 256u + wg_lane_id());
if (idx >= n) return;
wg_st16(y, (uint)idx * 2u, fg_leaky_fast16(wg_ld16(x, (uint)idx * 2u)));
}
52 changes: 38 additions & 14 deletions clean/dx12/kernels/fg_mconv.hlsl
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
// D3D12 port of frame generation's matrix convolution (issue #96; tier B of the net1 fast path): fg_mconv<MT> of
// fg_net1_mma.co (clean/fg/kernels/fg_net1_mma.hip), byte-identical to HIP: the same fragments and chunk order,
// v_wmma_f32_16x16x16_f16 through the driver channel (wg_wmma_f16) with the float16 accumulator as C, its result rounded
// to float16 after every chunk (wg_f2h = v_cvt_f16_f32, nearest even), the exact integer leaky ReLU and pool
// (fg_exact16.h). Driver rules (clean/dx12/README.md): every load of a chunk before its first matrix call; branchless
// to float16 after every chunk (wg_f2h = v_cvt_f16_f32, nearest even), the leaky ReLU of fg_fast16.h (fg_leaky16's
// bits) and the exact pool (fg_exact16.h). Driver rules (clean/dx12/README.md): every load of a chunk before its first matrix call; branchless
// selects; the chunk loop bound through an opaque zero (no forced unrolling); store pointers through wg_uniform().
// Built only when --only names the kernels (fg_ kernels are in no model's set).
//
Expand All @@ -16,6 +16,10 @@
#define FG_I64 int64_t
#define FG_FBH32(v) firstbithigh((uint)(v))
#include "../../fg/kernels/fg_exact16.h"
#define FG_F2H(f) wg_f2h(f)
#define FG_H2F(h) wg_h2f(h)
#define FG_SEL(c, a, b) select(c, a, b)
#include "../../fg/kernels/fg_fast16.h"

#ifndef FG_MT
#define FG_MT 4
Expand All @@ -29,6 +33,27 @@ WG_ARGS_END
uint fg_opq0() { return wg_grid_ >> 31; } // an opaque zero (the grid size is below 2^31)

uint4 fg_pack8(uint r[8]) { return uint4(r[0] | (r[1] << 16), r[2] | (r[3] << 16), r[4] | (r[5] << 16), r[6] | (r[7] << 16)); }
uint fg_half(uint4 u, uint j)
{
const uint d = select(j < 2u, u.x, select(j < 4u, u.y, select(j < 6u, u.z, u.w)));
return (d >> ((j & 1u) * 16u)) & 0xffffu;
}
// 8 packed float16 values -> the float32 accumulator fragment (exact)
v8f fg_widen(uint4 hp)
{
v8f r;
[unroll] for (uint v = 0; v < 8; ++v) r.v[v] = wg_h2f(fg_half(hp, v));
return r;
}
// One chunk (tier B): the matrix instruction, then the whole fragment rounded to float16 by the driver channel's
// conversion (v_cvt_f16_f32, nearest even: HIP's (_Float16) of each element; the software wg_f2h costs ~20 instructions
// per element on this driver, where a (half) cast rounds toward zero). hp keeps the packed float16 accumulator.
void fg_chunk_b(uint4 av, uint4 bv, inout v8f c, inout uint4 hp)
{
const v8f d = wg_wmma_f16(av, bv, c);
hp = wg_to_f16_ags(d);
c = fg_widen(hp);
}

WG_KERNEL(fg_mconv)
{
Expand All @@ -44,13 +69,15 @@ WG_KERNEL(fg_mconv)
const uint py = (uint)wg_uniform((int)y), pp = (uint)wg_uniform((int)p);

v8f a32[FG_MT][2];
uint4 h16[FG_MT][2]; // the same accumulator as packed float16
[unroll] for (int mt = 0; mt < FG_MT; ++mt) {
const int o0 = (mg * FG_MT + mt) * 16 + 8 * hh;
[unroll] for (uint v = 0; v < 8; ++v) {
const float bv = wg_h2f(wg_ld16(b, (uint)(o0 + (int)v) * 2u));
a32[mt][0].v[v] = bv;
a32[mt][1].v[v] = bv;
}
uint bb[8];
[unroll] for (uint v = 0; v < 8; ++v) bb[v] = wg_ld16(b, (uint)(o0 + (int)v) * 2u);
h16[mt][0] = fg_pack8(bb);
h16[mt][1] = h16[mt][0];
a32[mt][0] = fg_widen(h16[mt][0]);
a32[mt][1] = a32[mt][0];
}
const int nq_loop = nq + (int)fg_opq0();
for (int q = 0; q < nq_loop; ++q) {
Expand All @@ -72,18 +99,15 @@ WG_KERNEL(fg_mconv)
[unroll] for (int mt2 = 0; mt2 < FG_MT; ++mt2)
fa[mt2] = wg_ld128(wf, (uint)((((mg * FG_MT + mt2) * nq + q) * 32 + (int)lane) * 16));
[unroll] for (int mt3 = 0; mt3 < FG_MT; ++mt3)
[unroll] for (int nt2 = 0; nt2 < 2; ++nt2) {
const v8f d = wg_wmma_f16(fa[mt3], fb[nt2], a32[mt3][nt2]);
[unroll] for (uint v3 = 0; v3 < 8; ++v3) a32[mt3][nt2].v[v3] = wg_h2f(wg_f2h(d.v[v3]));
}
[unroll] for (int nt2 = 0; nt2 < 2; ++nt2) fg_chunk_b(fa[mt3], fb[nt2], a32[mt3][nt2], h16[mt3][nt2]);
}
[unroll] for (int mt4 = 0; mt4 < FG_MT; ++mt4) {
const int o0 = (mg * FG_MT + mt4) * 16 + 8 * hh;
uint r0[8], r1[8];
[unroll] for (uint v4 = 0; v4 < 8; ++v4) {
const uint h0 = wg_f2h(a32[mt4][0].v[v4]), h1 = wg_f2h(a32[mt4][1].v[v4]);
r0[v4] = select(act == 1, fg_leaky16(h0), h0);
r1[v4] = select(act == 1, fg_leaky16(h1), h1);
const uint h0 = fg_half(h16[mt4][0], v4), h1 = fg_half(h16[mt4][1], v4);
r0[v4] = select(act == 1, fg_leaky_fast16(h0), h0);
r1[v4] = select(act == 1, fg_leaky_fast16(h1), h1);
}
if (px < W && py0 < H) wg_st128(py, (uint)((py0 * W + px) * cout + o0) * 2u, fg_pack8(r0));
if (px < W && py0 + 1 < H) wg_st128(py, (uint)(((py0 + 1) * W + px) * cout + o0) * 2u, fg_pack8(r1));
Expand Down
Loading
Loading