Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
9e4a18b
Emulator: swin.hip builds on the host again (wg_uniform, asm guard)
j0nsh1n Nov 15, 2026
b033c53
Runner: a gfx1100 target, with an exact-name family map
j0nsh1n Nov 15, 2026
b28436e
Tests: Model M stages without matrix instructions run on a GPU (WG_GP…
j0nsh1n Nov 15, 2026
9354939
Platform: a gfx11 base in wg_platform.h (family macro, half-wave exch…
j0nsh1n Nov 15, 2026
d4ba9ad
Platform: a faster exact FP32 -> E4M3 conversion for gfx11, and its F…
j0nsh1n Nov 15, 2026
d05290f
Platform: gfx11 matrix wrappers (E4M3 widened to F16) and a gfx11 mod…
j0nsh1n Nov 15, 2026
4483ee5
Selftest: gfx11 probes for the matrix wrappers, the instruction's lan…
j0nsh1n Nov 15, 2026
9773983
Runner: Model M stage enc1 on a gfx11 GPU through the FP16 route (swi…
j0nsh1n Nov 15, 2026
d14270e
Swin: weights widened to F16 once per workgroup on gfx11 (variant f16w)
j0nsh1n Nov 15, 2026
b8b4e82
Swin: activation operands made once on gfx11 (variant f16wa)
j0nsh1n Nov 15, 2026
e8927b4
Swin: gfx11 operands made straight from the FP32 values (variant f16wq)
j0nsh1n Nov 15, 2026
b4873c6
Swin: evidence sheet for Model M on gfx11, FP16 route (enc1 and b0)
j0nsh1n Nov 15, 2026
7cb0f79
Swin: the other block shapes on the gfx11 card, and resource gates fo…
j0nsh1n Nov 15, 2026
e372b2c
Swin: the C = 128 block kernel at 7 waves per SIMD on gfx11 (variant …
j0nsh1n Nov 15, 2026
87b4fe3
Swin: gfx11 F16 operands for the C = 128 kernel (variant f16wk)
j0nsh1n Nov 15, 2026
e8882b2
Swin: gfx11 F16 operands for the MLP of the C = 96 kernels (variant f…
j0nsh1n Nov 15, 2026
e72dad7
Swin: patch merge and expand run the matrix chain in every lane on gf…
j0nsh1n Nov 15, 2026
e018d7b
Swin: variant mgx, the patch merge widens each operand to F16 once on…
j0nsh1n Nov 15, 2026
00e5433
Swin: the persistent kernels swin_bneck6 and swin_mid on the gfx11 ca…
j0nsh1n Nov 15, 2026
dd30ce2
Swin: no matrix instruction under a lane mask in the C = 64 persisten…
j0nsh1n Nov 15, 2026
2f4f424
Encoder: input_encoder on the gfx11 card, with no matrix instruction …
j0nsh1n Nov 15, 2026
c41b4f2
Decoder: coef_decoder on the gfx11 card (E4)
j0nsh1n Nov 15, 2026
bb0cb95
Fused dec1 + dec0: swin_block_c64_add_dec0 on the gfx11 card (E4)
j0nsh1n Nov 15, 2026
3e2523d
Scan: find llvm-objdump when ROCM_PATH is a link folder (the build te…
j0nsh1n Nov 15, 2026
3de364e
Post: the reduced-history exchange under a lane mask is harmless; the…
j0nsh1n Nov 15, 2026
0e0abf6
Swin: the fused merge and expand under a loop-entry mask are harmless…
j0nsh1n Nov 15, 2026
bb6f14b
Whole frame on the gfx11 card (E5): card test, skipped until the runt…
j0nsh1n Nov 15, 2026
19fd7ec
Runtime: the HIP device gate takes gfx1100 and gfx1101 by the exact n…
j0nsh1n Nov 15, 2026
e3ff995
Runtime: the weights are packed with the accumulator rows of the devi…
j0nsh1n Nov 15, 2026
90c8540
Build: a gfx11 target also builds the runtime library, and writes <ar…
j0nsh1n Nov 15, 2026
bba5b4a
Whole frame on the gfx11 card (E5): the card test runs on the runtime…
j0nsh1n Nov 15, 2026
9a41598
D3D12 platform: WG_GFX 11 helpers for RDNA 3 and build_dx12.py --gfx …
j0nsh1n Nov 16, 2026
fb0edba
D3D12 tests: headless Wine runner, Linux/mingw build of the replay ho…
j0nsh1n Nov 16, 2026
fdbe607
D3D12 probe: an F16-only driver-channel probe for RDNA 3, and its lay…
j0nsh1n Nov 16, 2026
37de8d3
D3D12: a developer-only opening of RDNA 3 (WG_DEV_GFX11), off in ever…
j0nsh1n Nov 16, 2026
af27c53
D3D12: a developer-only shortcut inside WG_DEV_GFX11: the unchanged g…
j0nsh1n Nov 16, 2026
1517442
D3D12 tests: a whole Model M frame on the native executor under headl…
j0nsh1n Nov 16, 2026
e9738b5
Kernels: the gfx12 source of every HIP kernel is again exactly main's…
j0nsh1n Nov 17, 2026
e8ab57a
D3D12 platform: the gfx11 helpers move to wg_gfx11.hlsli; wg_platform…
j0nsh1n Nov 16, 2026
11c7de2
test: gfx12 has no wg_xchg16_u; the exchange test checks that instead…
j0nsh1n Nov 17, 2026
0971e02
test: Windows CI: no relpath across drives in the kernel digest; the …
j0nsh1n Nov 17, 2026
5f3d524
feat: native RDNA 3 K = 16 operands for the D3D12 ports (S1) and a CP…
j0nsh1n Nov 16, 2026
30030c8
feat: native RDNA 3 form of Model K's narrow network stages (K2), gfx…
j0nsh1n Nov 16, 2026
782c954
test: k_native_check.py, native K stages against the shortcut on the …
j0nsh1n Nov 16, 2026
2ec668e
test: card tests of the K = 16 opcode and the native operand forms (s…
j0nsh1n Nov 16, 2026
1da8461
test: the native K check's unit test covers the merge store's row bou…
j0nsh1n Nov 16, 2026
eac6c4e
test hook: k_smx_test=mix_unchecked (developer diagnostics builds only)
j0nsh1n Nov 16, 2026
3bf789c
runtime: WG_DEV_GFX11_NATIVE_K launches the native K kernels (<name>_…
j0nsh1n Nov 16, 2026
c78561e
runtime: WG_DEV_GFX11_NATIVE_K takes a list of ports, so native can b…
j0nsh1n Nov 16, 2026
7917c54
test: the K = 16 opcode is translated only with the shortcut's config…
j0nsh1n Nov 16, 2026
898836c
docs: the native K test's docstring names the list form of WG_DEV_GFX…
j0nsh1n Nov 16, 2026
90eda06
native K: register knobs for the two-window encoders (all off by defa…
j0nsh1n Nov 17, 2026
fb5e4ee
native K: KN_A16, 16-byte arena accesses as one 128-bit load or store…
j0nsh1n Nov 17, 2026
3e86787
native K: each stage kernel declares its measured register and load k…
j0nsh1n Nov 17, 2026
54b5a10
runtime: developer builds refuse Model K stage pointers that are not …
j0nsh1n Nov 17, 2026
5a28bd3
native K: KN_MERGE_STEP is 1, 2 or 4 (a preprocessor error otherwise)
j0nsh1n Nov 17, 2026
d877391
runtime: the native K list is split without find(, so rdna4.h keeps #…
j0nsh1n Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions clean/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,34 @@ objects and the converter map into `build`; the FidelityFX front end loads it (c
other than `gfx12`, set `WG_D_ROW` to the reported table (for example `set WG_D_ROW=[[0,2,4,...],[1,3,...]]`)
before running.

## Linux (gfx1100 and gfx1201)

`--rocm` names an SDK in the layout of AMD's installer: `bin/clang`, the device library in `amdgcn/bitcode` and
`libamdhip64.so` in `lib`. A distribution's ROCm that keeps these elsewhere is used through a folder of links in that
layout.

```
python clean/runner/build.py --rocm <ROCm> --out build --no-runtime --no-model-k # gfx1201, all code objects
python clean/runner/build.py --arch gfx1100 --rocm <ROCm> --out build-gfx1100 # the gfx11 default sources
python clean/runner/build.py --arch gfx1100 --rocm <ROCm> --out build-gfx1100 --sources post.hip frame.hip
```

Targets are matched by exact name (`runner/family.py`): gfx1200 and gfx1201 are gfx12, gfx1100 and gfx1101 are gfx11,
anything else is refused. A gfx11 build compiles the Model M kernels of `FAMILY_SOURCES` in `build.py` (with `--variants`, the
swin variants of the family) and no Model K or L kernels, post_w64 or motion variants or resource gate; `--sources` builds a
chosen list for either family. Both write `target.json` next to the code objects, and `Runner` takes its default accumulator
row table from it (`WG_D_ROW` and the `d_row` argument still win). Unless `--no-runtime` or `--sources` is given, a gfx11 build
also makes `libwg_runtime.so` (nothing built in: it reads its code objects from the folder as `<arch>.<name>`, copies of the
files that the build writes next to them, and the runtime takes the row table of its device's target; `--runtime-only` does
this step alone). The runtime's HIP gate takes the same four targets (`runtime/target_family.h`).

On a gfx11 card (headless, one GPU job at a time): `python clean/runner/selftest.py --build <build> --rocm <ROCm>` (the
half-wave exchange and the software FP8 conversions; the matrix and hardware-conversion probes are gfx12 only),
`WG_GPU_TEST=1 WG_BUILD=<build> WG_ROCM=<ROCm> python -m unittest clean.tests.test_gpu_stages clean.tests.test_gpu_q8` (the
stages without matrix instructions, and the conversions on a set of input chunks) and
`python clean/runner/q8_exhaustive.py --build <build> --rocm <ROCm>` (the conversions over all 2^32 FP32 patterns, about two
minutes).

## Spec team: stages post and filter (spec 06)

```
Expand Down
6 changes: 6 additions & 0 deletions clean/dx12/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,12 @@ WG_KERNEL(exposure_auto_diet) { WG_SETUP(); ... } // entry name = HIP kernel n
builds (one manifest entry per name and `co`, for example `w64/post_tf_f32_p12_diet_vsit` of `post_w64__orig.co`
and of `post_w64.co`): the device runs the port whose code object is loaded. Every port of a name but the first
sets `file=<stem>`, its DXIL file name (`<stem>.dxil`).
* `gfx=<11|12|11,12>` (optional, default 12) = the GPU families the declaration is built for (`build_dx12.py --gfx`,
default 12): `--gfx 11` (RDNA 3, `WG_GFX=11` in every kernel) builds only the declarations that list 11 into
`<build>/dx12-gfx11/` with a `"gfx": 11` key in its manifest (a gfx12 manifest has no `gfx` key), without RGA. The
gfx11 helpers (`wg_wmma_f16_k8` / `_k16`, `wg_d_row`, no FP8 matrix instruction) are in `wg_gfx11.hlsli`, which a gfx11 kernel includes
instead of `wg_platform.hlsli` (it includes that itself); the platform layer and `wg_ags.hlsli` hold no gfx11 code, because
model K's softmax self-check digest covers them.
* `args` = the HIP kernel's arguments in order: `p` pointer (DWORD: arena byte offset), `u` any 32-bit scalar,
`b<N>` struct by value (N bytes = N/4 DWORDs, copied verbatim). At most 57 DWORDs. The `WG_ARGS_BEGIN` block must
declare exactly these DWORDs in order, **32-bit scalars only** (no vectors / arrays / structs: cbuffer packing).
Expand Down
50 changes: 38 additions & 12 deletions clean/dx12/build_dx12.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,11 @@
and rcp() (fdiv 1.0, x) is the hardware v_rcp_f32; it is reassembled (dxil_asm.cpp, DXC's IDxcAssembler) and signed by
dxv. Any other float division ('/', which would be an IEEE division) or float remainder is refused: use wg_div.

GPU family (--gfx 12, the default, or 11 for RDNA 3): a gfx11 build defines WG_GFX=11 for every kernel, writes <out>/dx12-gfx11/
and a "gfx": 11 key in its manifest, and builds only the kernels whose declaration says gfx=11 (a comma list, default
gfx=12: every existing declaration is a gfx12 kernel, so a build without --gfx is the same set as before). RGA is not used
for gfx11 (no VGPR check, no --isa).

VGPR check: every Swin / bottleneck kernel built (a name with a swin_ or l_swin_ part) is run through RGA (gfx1201) and its
USED_VGPRs compared with its bound in clean/dx12/vgpr_limits.json (the count that keeps its present workgroups per CU);
over the bound, or no bound listed, fails the build. WG_DX12_VGPR_CHECK=0 skips it.
Expand Down Expand Up @@ -210,6 +215,9 @@ def kernel_decls(path):
raise SystemExit(f"{path}: {name}: gscale {d['gscale']}")
# probe=1: a measurement probe, built only when --only names it (never part of a model's kernel set)
d["probe"] = kv.get("probe", "0") == "1"
d["gfx"] = {int(x) for x in kv.get("gfx", "12").split(",")}
if not d["gfx"] <= {11, 12}:
raise SystemExit(f"{path}: {name}: gfx={kv.get('gfx')} (11, 12 or 11,12)")
d["file"] = kv.get("file", "") # DXIL file stem of a second port of the same name (module doc)
if d["file"] and not re.fullmatch(r"[A-Za-z0-9_]+", d["file"]):
raise SystemExit(f"{path}: {name}: file={d['file']} (letters, digits and _ only)")
Expand Down Expand Up @@ -258,14 +266,15 @@ def source_files(model_k=False, only_k=False):
return out


def kernel_set(model_l=False, model_k=False):
def kernel_set(model_l=False, model_k=False, gfx=12):
"""(name, code object) of every kernel a full build compiles (no --only), sorted: Model M's set, plus Model L's
with model_l, plus Model K's with model_k. From the sources' declarations alone (nothing is compiled). Frame
generation's kernels and measurement probes (probe=1) are never in it (is_frame_gen)."""
generation's kernels and measurement probes (probe=1) are never in it (is_frame_gen). gfx: the GPU family whose
kernels are listed."""
keys = set()
for path, k_source in source_files(model_k):
for d in kernel_decls(path):
if is_frame_gen(d) or d["probe"]:
if is_frame_gen(d) or d["probe"] or gfx not in d["gfx"]:
continue
if k_source or is_model_k(d):
if model_k:
Expand Down Expand Up @@ -431,7 +440,10 @@ def text(path):
def label(path):
if gen_dir and os.path.normcase(os.path.abspath(path)).startswith(os.path.normcase(os.path.abspath(gen_dir))):
return "gen/" + os.path.relpath(path, gen_dir).replace(os.sep, "/")
return os.path.relpath(path, root).replace(os.sep, "/")
try:
return os.path.relpath(path, root).replace(os.sep, "/")
except ValueError: # another drive (Windows): no relative path exists; a source outside the repo is named by its file
return "ext/" + os.path.basename(path)

seen, order, missing = set(), [], set()

Expand All @@ -452,7 +464,8 @@ def walk(path):

walk(src)
h = hashlib.sha256()
h.update(b"decl\n" + json.dumps(decl, sort_keys=True).encode() + b"\n")
# the gfx set (a set, not JSON) is left out: a build's family is in its defines, and a gfx12 digest stays what it was
h.update(b"decl\n" + json.dumps({k: v for k, v in decl.items() if k != "gfx"}, sort_keys=True).encode() + b"\n")
for path in sorted(order, key=label):
h.update(b"file " + label(path).encode() + b"\n" + text(path) + b"\n")
h.update(b"missing " + ",".join(sorted(missing)).encode() + b"\n")
Expand Down Expand Up @@ -575,6 +588,9 @@ def main(argv=None):
ap.add_argument("--model-k", action="store_true",
help="also build Model K's kernels (clean/k/dx12/kernels; off by default: Model M's and Model L's "
"kernel sets unchanged)")
ap.add_argument("--gfx", type=int, choices=(11, 12), default=12,
help="GPU family of the build: 12 = RDNA 4 (default, <out>/dx12), 11 = RDNA 3 (WG_GFX=11, "
"<out>/dx12-gfx11, only kernels declared gfx=11, no RGA)")
ap.add_argument("--define", action="append", default=[], metavar="NAME=VALUE",
help="an extra preprocessor define for every kernel built (test variants: negative-control mutants, "
"test counters; never for a shipped build)")
Expand All @@ -586,7 +602,7 @@ def main(argv=None):
"the cheap check before a build (kernel_sets.json, tests/test_kernel_sets.py)")
a = ap.parse_args(argv)
if a.list:
print(json.dumps([list(k) for k in kernel_set(a.model_l, a.model_k)], indent=1))
print(json.dumps([list(k) for k in kernel_set(a.model_l, a.model_k, a.gfx)], indent=1))
return 0
if not a.out:
ap.error("--out is required")
Expand All @@ -595,7 +611,9 @@ def main(argv=None):
if not re.fullmatch(r"[A-Z][A-Z0-9_]*=-?[0-9A-Za-z_]+", kv):
ap.error(f"--define {kv!r} (NAME=VALUE)")
extra_defs[kv.split("=", 1)[0]] = kv.split("=", 1)[1]
dst = os.path.join(a.out, "dx12")
if a.gfx == 11 and a.isa:
ap.error("--isa needs RGA, which a gfx11 build does not use")
dst = os.path.join(a.out, "dx12" if a.gfx == 12 else "dx12-gfx11")
os.makedirs(dst, exist_ok=True)
asm = build_asm_tool(a.out)
only = set(a.only.split(",")) if a.only else None
Expand Down Expand Up @@ -623,6 +641,8 @@ def main(argv=None):
jobs = [] # (source, declaration, output path) in declaration order (sorted file name, then line)
for src, k_source in source_files(a.model_k or bool(only)):
for d in kernel_decls(src):
if a.gfx not in d["gfx"]:
continue
if only and d["name"] not in only:
continue
if not only and (k_source or is_model_k(d)) and not a.model_k:
Expand Down Expand Up @@ -656,7 +676,8 @@ def main(argv=None):

def build_one(job):
src, d, out, k_source = job
defs = {"WG_BLOCK": d["group"], "WG_WAVE": d["wave"], **d["defines"], **extra_defs}
defs = {"WG_BLOCK": d["group"], "WG_WAVE": d["wave"], **({"WG_GFX": a.gfx} if a.gfx != 12 else {}),
**d["defines"], **extra_defs}
if d["group"] != d["block"]:
defs["WG_LAUNCH_BLOCK"] = d["block"]
info = {}
Expand Down Expand Up @@ -707,14 +728,19 @@ def build_one(job):
if only and not (only <= names):
raise SystemExit(f"not declared in clean/dx12/kernels or clean/k/dx12/kernels: {sorted(only - names)}")
try:
vgpr_check(built, {d["name"] for _, d, _, _ in jobs if hyb_scanned(d["name"]) and not d["probe"]})
isa_check(built)
if a.gfx == 12:
vgpr_check(built, {d["name"] for _, d, _, _ in jobs if hyb_scanned(d["name"]) and not d["probe"]})
isa_check(built)
finally:
if scan_dir:
shutil.rmtree(scan_dir, ignore_errors=True)
manifest = {"format": FORMAT, "dxc": os.path.basename(os.path.dirname(os.path.dirname(DXC_DIR)))}
if a.gfx != 12:
manifest["gfx"] = a.gfx # absent = 12
manifest["test_only"] = test_only
manifest["kernels"] = [kernels[k] for k in sorted(kernels)]
with open(man_path, "w") as f:
json.dump({"format": FORMAT, "dxc": os.path.basename(os.path.dirname(os.path.dirname(DXC_DIR))),
"test_only": test_only, "kernels": [kernels[k] for k in sorted(kernels)]}, f, indent=1)
json.dump(manifest, f, indent=1)
print(f"{len(kernels)} kernel(s) in {man_path}" + (" (TEST-ONLY: never embedded or packaged)" if test_only else ""))
return 0

Expand Down
Loading
Loading