Skip to content

Commit 05bf233

Browse files
Merge remote-tracking branch 'origin/main' into k/d3d12-wide
2 parents 54f8ef8 + 0eb6eac commit 05bf233

29 files changed

Lines changed: 7761 additions & 191 deletions

‎clean/FORMATS.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -492,12 +492,12 @@ the runtime is a shared D3D12 object except the three fences.
492492

493493
| Buffer | Size | Content |
494494
|---|---|---|
495-
| color | max R × 16 | `[rh][rw]` 4 × f32 (alpha 0), the game's color in the active region |
495+
| color | max R × 16 | `[rh][rw]` 4 × f32, the game's color in the active region; the fourth value +0 (model M), or for model K the texel's alpha, unmodified (1 for a format without an alpha channel): model K's output alpha follows it |
496496
| depth | max R × 4 | `[rh][rw]` f32, the game's depth as given |
497497
| motion | max R × 8 | `[rh][rw]` 2 × f32, m = motion_scale · raw + motion_offset (render pixels); with `WG_CREATE_MOTION_OUTPUT_RES` the texel ⌊(x + ½)·mw/rw⌋, ⌊(y + ½)·mh/rh⌋ (our point resampling: what the network reads with `motion_outres=point`; with `original` it reads `motion_hi`) |
498498
| motion_hi | O × 4 + 8, or none | only with `WG_CREATE_MOTION_OUTPUT_RES` (model M; with `motion_outres=point` only while a frame capture is armed): `[oh][ow]` 2 × FP16, the raw motion texels of the output-size region ("Motion vectors at output resolution"). It follows `out` in each slot; without it every offset is what it was |
499499
| expo | 256 bytes | f32 at offset 0: texel (0, 0) channel R of the exposure texture (exposure mode texture) |
500-
| out | O × 12 | `[oh][ow]` 3 × f32, the network's result (the runtime's `filter.out` format) |
500+
| out | O × 12 (model K: O × 16) | `[oh][ow]` 3 × f32, the network's result (the runtime's `filter.out` format); model K: `[oh][ow]` 4 × f32, the fourth value its output alpha (`FrameIo.out_rgba32f`), which the output pass stores |
501501

502502
Fences (shared, `hipImportExternalSemaphore`, D3D12 fence): `ready` (the split request's done fence: the front end's
503503
queue signals v after the first part of the game's list, which ends with the input conversion), `done` (HIP signals v
@@ -524,7 +524,7 @@ whole texture; otherwise the frame uses the input pass (logged). The network's f
524524
0.309 → 0.251 ms, output 0.856 → 0.838 ms against the FP32 slot).
525525
- Model K's compact input pass (default for `model=K` when no copy is made; `k_input=f32`: the FP32 input pass):
526526
where the game's colour is RGBA16F or R11G11B10 and its motion RG16F or RGBA16F (FP16-representable values), the
527-
pass `inputs16` writes dense RGBA16F colour (rw·8-byte rows, alpha 0), R32F depth and the raw RG16F motion texel
527+
pass `inputs16` writes dense RGBA16F colour (rw·8-byte rows, the texel's alpha kept; 1 for R11G11B10), R32F depth and the raw RG16F motion texel
528528
(rw·4-byte rows; at output size the texel under the render pixel's centre, as the FP32 pass samples it) into the
529529
slot's buffers, and the network reads them as it reads the copies (motion_scale / motion_offset applied on load).
530530
Same output bits as the FP32 pass (`test_interop_d3d12 hipsame`, render- and output-size motion); logged as
@@ -568,7 +568,8 @@ pre-exposure and exposure, then signals `done = v`. Values only grow.
568568
3. Network enqueued (above). If it cannot be enqueued, the runtime's own compute queue waits for the previous `done`
569569
value and signals `done = v` (and `pipe = v`), so the game's queue cannot wait forever; the dispatch returns an error.
570570
4. Output pass into the re-opened list: `out` of slot v (inline) or of the previous network frame (pipelined; the
571-
first frame: bilinear upscale of the color) → the output texture (alpha 1; sRGB formats encoded), output
571+
first frame: bilinear upscale of the color) → the output texture (alpha 1; model K: the result's fourth value, in
572+
the upscale the bilinear of the color's alpha; sRGB formats encoded), output
572573
transitioned to UNORDERED_ACCESS if needed and back (else a UAV barrier). Shaders: HLSL compiled at create with
573574
`d3dcompiler_47.dll` (cs_5_0); the result buffer is read through an R32_FLOAT typed UAV.
574575

‎clean/dx12/README.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ as its HIP kernel (`clean/dx12/compare.py` proves it per kernel on synthetic fra
2020
| `dxil_asm.cpp` | tool used by the build (assembles the rewritten DXIL listing) |
2121
| `compare.py` | HIP vs hybrid D3D12, every stage output / state buffer byte for byte |
2222
| `kernels/exposure.hlsl` | the reference port (`exposure_auto_diet`): copy its structure |
23-
| `kernels/fg_*.hlsl` | frame generation's ports (#96, `co=fg_*.co`): groundwork, built only when `--only` names them, never by a full build (`is_frame_gen`), so the Model M / Model L sets stay as they are; checks: `clean/fg/runner/fg_d3d12_check.py` (each kernel), `clean/fg/runner/fg_net1_gpu.py` (net1's 32 steps as a `kernel_replay` chain) |
23+
| `kernels/fg_*.hlsl` | frame generation's ports (#96, `co=fg_*.co`): groundwork, built only when `--only` names them, never by a full build (`is_frame_gen`), so the Model M / Model L sets stay as they are; checks: `clean/fg/runner/fg_d3d12_check.py` (each kernel), `clean/fg/runner/fg_net1_gpu.py` (net1's 32 steps as a `kernel_replay` chain), `clean/fg/runner/fg_net1_mma.py` (net1's fast path on the matrix instructions, fused / unfused / exact) |
2424
| `tests/kernel_replay.cpp` | replays launches from a case file on the native D3D12 device; `chain` blocks run several launches on buffers that persist between them (a whole network, optionally timed) |
2525

2626
## Arithmetic mode (why build_dx12.py and nothing else)
@@ -71,6 +71,16 @@ The rules above hold per operation. Across operations the gfx1201 D3D12 shader c
7171
(`n + (wg_grid_ >> 31)`; `swin_common.hlsli` `sw_zero()`), and fold LDS offsets with `sw_zero_bits()`.
7272
* HLSL 2021 `?:`, `&&`, `||` short-circuit and become branches (one load in flight per branch): use `select()`,
7373
`and()`, `or()` in hot code.
74+
* **lanes past the end of the grid corrupted other lanes' matrix results** (frame generation's `fg_mpair`, 2026-11-14):
75+
a per-lane `bool valid[]` with `pc = select(valid, p, 0)` for the pixel, then WMMAs, then `if (valid) store`. With part
76+
of the last 16-pixel group past the end, the valid lanes got NaN in exactly the output rows whose A-fragment row sits in
77+
an invalid lane; HIP was right. Remedy: no validity array; every lane computes a clamped pixel (`min(p, n - 1)`) and
78+
only the store tests `p < n`. Check odd grid sizes in every matrix port.
79+
* **`wg_f2h` is software** (~20 instructions per value; a `(half)` cast rounds toward zero here). In a hot loop round
80+
a whole fragment with the driver conversion, `wg_to_f16_ags` (v_cvt_f16_f32, HIP's `(_Float16)` bits), and widen back
81+
with `wg_h2f`; for 8 values that are not a matrix result (an element-wise stage), XOR each with an opaque zero first
82+
(`fg_f2h8` in `fg_mpair.hlsl`) so the producing multiply is not folded into the conversion (see above: a folded
83+
product turns -0 into +0). Frame generation's net1 fast path went from 3.5x to 1.5x HIP's GPU time with this.
7484

7585
The native helper regression additionally constructed tiny exact values that round to the smallest normal FP32
7686
number: `0x00800000 * 0x3f7fffff` and `0x00ffffff / 0x40000000`. The multiply flushes before rounding on this driver;
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
// SPDX-License-Identifier: MIT
2+
// D3D12 port of fg_leaky_fast_f16 of fg_net1_mma.co (clean/fg/kernels/fg_net1_mma.hip): fg_leaky_fast16 on n float16
3+
// values, the exhaustive check of the fast path's leaky ReLU (runner/fg_net1_mma.py). Built only when --only names it.
4+
//
5+
// @wg-kernel fg_leaky_fast_f16 co=fg_net1_mma.co block=256 wave=32 args=p,p,u
6+
#include "wg_platform.hlsli"
7+
8+
#define FG_FN
9+
#define FG_I64 int64_t
10+
#define FG_FBH32(v) firstbithigh((uint)(v))
11+
#include "../../fg/kernels/fg_exact16.h"
12+
#define FG_F2H(f) wg_f2h(f)
13+
#define FG_H2F(h) wg_h2f(h)
14+
#define FG_SEL(c, a, b) select(c, a, b)
15+
#include "../../fg/kernels/fg_fast16.h"
16+
17+
WG_ARGS_BEGIN
18+
uint x, y; // pointers (arena byte offsets)
19+
int n;
20+
WG_ARGS_END
21+
22+
WG_KERNEL(fg_leaky_fast_f16)
23+
{
24+
WG_SETUP();
25+
const int idx = (int)(wg_block_id() * 256u + wg_lane_id());
26+
if (idx >= n) return;
27+
wg_st16(y, (uint)idx * 2u, fg_leaky_fast16(wg_ld16(x, (uint)idx * 2u)));
28+
}

‎clean/dx12/kernels/fg_mconv.hlsl‎

Lines changed: 38 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22
// D3D12 port of frame generation's matrix convolution (issue #96; tier B of the net1 fast path): fg_mconv<MT> of
33
// fg_net1_mma.co (clean/fg/kernels/fg_net1_mma.hip), byte-identical to HIP: the same fragments and chunk order,
44
// v_wmma_f32_16x16x16_f16 through the driver channel (wg_wmma_f16) with the float16 accumulator as C, its result rounded
5-
// to float16 after every chunk (wg_f2h = v_cvt_f16_f32, nearest even), the exact integer leaky ReLU and pool
6-
// (fg_exact16.h). Driver rules (clean/dx12/README.md): every load of a chunk before its first matrix call; branchless
5+
// to float16 after every chunk (wg_f2h = v_cvt_f16_f32, nearest even), the leaky ReLU of fg_fast16.h (fg_leaky16's
6+
// bits) and the exact pool (fg_exact16.h). Driver rules (clean/dx12/README.md): every load of a chunk before its first matrix call; branchless
77
// selects; the chunk loop bound through an opaque zero (no forced unrolling); store pointers through wg_uniform().
88
// Built only when --only names the kernels (fg_ kernels are in no model's set).
99
//
@@ -16,6 +16,10 @@
1616
#define FG_I64 int64_t
1717
#define FG_FBH32(v) firstbithigh((uint)(v))
1818
#include "../../fg/kernels/fg_exact16.h"
19+
#define FG_F2H(f) wg_f2h(f)
20+
#define FG_H2F(h) wg_h2f(h)
21+
#define FG_SEL(c, a, b) select(c, a, b)
22+
#include "../../fg/kernels/fg_fast16.h"
1923

2024
#ifndef FG_MT
2125
#define FG_MT 4
@@ -29,6 +33,27 @@ WG_ARGS_END
2933
uint fg_opq0() { return wg_grid_ >> 31; } // an opaque zero (the grid size is below 2^31)
3034

3135
uint4 fg_pack8(uint r[8]) { return uint4(r[0] | (r[1] << 16), r[2] | (r[3] << 16), r[4] | (r[5] << 16), r[6] | (r[7] << 16)); }
36+
uint fg_half(uint4 u, uint j)
37+
{
38+
const uint d = select(j < 2u, u.x, select(j < 4u, u.y, select(j < 6u, u.z, u.w)));
39+
return (d >> ((j & 1u) * 16u)) & 0xffffu;
40+
}
41+
// 8 packed float16 values -> the float32 accumulator fragment (exact)
42+
v8f fg_widen(uint4 hp)
43+
{
44+
v8f r;
45+
[unroll] for (uint v = 0; v < 8; ++v) r.v[v] = wg_h2f(fg_half(hp, v));
46+
return r;
47+
}
48+
// One chunk (tier B): the matrix instruction, then the whole fragment rounded to float16 by the driver channel's
49+
// conversion (v_cvt_f16_f32, nearest even: HIP's (_Float16) of each element; the software wg_f2h costs ~20 instructions
50+
// per element on this driver, where a (half) cast rounds toward zero). hp keeps the packed float16 accumulator.
51+
void fg_chunk_b(uint4 av, uint4 bv, inout v8f c, inout uint4 hp)
52+
{
53+
const v8f d = wg_wmma_f16(av, bv, c);
54+
hp = wg_to_f16_ags(d);
55+
c = fg_widen(hp);
56+
}
3257

3358
WG_KERNEL(fg_mconv)
3459
{
@@ -44,13 +69,15 @@ WG_KERNEL(fg_mconv)
4469
const uint py = (uint)wg_uniform((int)y), pp = (uint)wg_uniform((int)p);
4570

4671
v8f a32[FG_MT][2];
72+
uint4 h16[FG_MT][2]; // the same accumulator as packed float16
4773
[unroll] for (int mt = 0; mt < FG_MT; ++mt) {
4874
const int o0 = (mg * FG_MT + mt) * 16 + 8 * hh;
49-
[unroll] for (uint v = 0; v < 8; ++v) {
50-
const float bv = wg_h2f(wg_ld16(b, (uint)(o0 + (int)v) * 2u));
51-
a32[mt][0].v[v] = bv;
52-
a32[mt][1].v[v] = bv;
53-
}
75+
uint bb[8];
76+
[unroll] for (uint v = 0; v < 8; ++v) bb[v] = wg_ld16(b, (uint)(o0 + (int)v) * 2u);
77+
h16[mt][0] = fg_pack8(bb);
78+
h16[mt][1] = h16[mt][0];
79+
a32[mt][0] = fg_widen(h16[mt][0]);
80+
a32[mt][1] = a32[mt][0];
5481
}
5582
const int nq_loop = nq + (int)fg_opq0();
5683
for (int q = 0; q < nq_loop; ++q) {
@@ -72,18 +99,15 @@ WG_KERNEL(fg_mconv)
7299
[unroll] for (int mt2 = 0; mt2 < FG_MT; ++mt2)
73100
fa[mt2] = wg_ld128(wf, (uint)((((mg * FG_MT + mt2) * nq + q) * 32 + (int)lane) * 16));
74101
[unroll] for (int mt3 = 0; mt3 < FG_MT; ++mt3)
75-
[unroll] for (int nt2 = 0; nt2 < 2; ++nt2) {
76-
const v8f d = wg_wmma_f16(fa[mt3], fb[nt2], a32[mt3][nt2]);
77-
[unroll] for (uint v3 = 0; v3 < 8; ++v3) a32[mt3][nt2].v[v3] = wg_h2f(wg_f2h(d.v[v3]));
78-
}
102+
[unroll] for (int nt2 = 0; nt2 < 2; ++nt2) fg_chunk_b(fa[mt3], fb[nt2], a32[mt3][nt2], h16[mt3][nt2]);
79103
}
80104
[unroll] for (int mt4 = 0; mt4 < FG_MT; ++mt4) {
81105
const int o0 = (mg * FG_MT + mt4) * 16 + 8 * hh;
82106
uint r0[8], r1[8];
83107
[unroll] for (uint v4 = 0; v4 < 8; ++v4) {
84-
const uint h0 = wg_f2h(a32[mt4][0].v[v4]), h1 = wg_f2h(a32[mt4][1].v[v4]);
85-
r0[v4] = select(act == 1, fg_leaky16(h0), h0);
86-
r1[v4] = select(act == 1, fg_leaky16(h1), h1);
108+
const uint h0 = fg_half(h16[mt4][0], v4), h1 = fg_half(h16[mt4][1], v4);
109+
r0[v4] = select(act == 1, fg_leaky_fast16(h0), h0);
110+
r1[v4] = select(act == 1, fg_leaky_fast16(h1), h1);
87111
}
88112
if (px < W && py0 < H) wg_st128(py, (uint)((py0 * W + px) * cout + o0) * 2u, fg_pack8(r0));
89113
if (px < W && py0 + 1 < H) wg_st128(py, (uint)(((py0 + 1) * W + px) * cout + o0) * 2u, fg_pack8(r1));

0 commit comments

Comments
 (0)