Skip to content

RDNA 3 (RX 7900 XT): Model M in HIP on gfx1100 and a developer-only D3D12 route under Proton (draft) - #259

Draft
j0nsh1n wants to merge 41 commits into
phantoslayer21:mainfrom
j0nsh1n:rdna3/proton-route
Draft

j0nsh1n wants to merge 41 commits into
phantoslayer21:mainfrom
j0nsh1n:rdna3/proton-route

Conversation

@j0nsh1n

@j0nsh1n j0nsh1n commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Draft, for visibility. Model M on RDNA 3 (RX 7900 XT, gfx1100): the HIP kernels on Linux, and a developer-only D3D12 route that runs in a game under Proton. Nothing here changes what a player gets: every RDNA 4 code object and DXIL file builds byte-identical, and the RDNA 3 openings are off in every normal build. Several open decisions are yours (listed at the end); I am not asking for a merge.

What this changes and why

RDNA 3 has no FP8 matrix instruction. These commits run Model M's arithmetic through the F16 one (v_wmma_f32_16x16x16_f16): E4M3 operands are widened to F16 exactly, the FP32 -> E4M3 roundings stay exact in software, so each stage stays close to the RDNA 4 result (not bit-identical: the F16 instruction's FP32 sums round differently, measured in #28's analysis).

In order (37 commits, each signed off):

  • 12ef8dc A stand-alone fix: swin.hip builds for the host emulator again (the raw readfirstlane and an +s asm constraint broke it on main).
  • Build and platform: a gfx1100 target with an exact-name family map (clean/runner/family.py, and clean/runtime/target_family.h kept equal to it by a test); a gfx11 base in wg_platform.h (half-wave exchange, a faster exact FP32 -> E4M3 conversion, the F16 matrix wrappers) and a gfx11 mode of the lane emulator; card probes.
  • Kernels, stage by stage (enc1, the other block shapes, patch merge / expand, the persistent bottleneck, encoder, decoder, fused dec1+dec0, post), each with a card test against the emulator and float64, and FP16 speed variants (f16w*, mgx, c128w7).
  • A gfx11 hazard and a scan for it: the compiler moved a v_wmma chain into the branch of a per-lane store (if (token < ntok)), so on partly valid tiles the output had NaN or wrong values (the emulator cannot show it). Fixed by unguarded stores and a wave-uniform loop bound; clean/runner/exec_scan.py scans the disassembly for matrix ops and half-wave exchanges under a lane mask (tests red on the old code).
  • Runtime: the HIP device gate takes gfx1100/gfx1101 by exact name, weights are packed with the device family's accumulator rows, and a gfx11 build also builds the runtime. After R1: no in-game route to HIP; GPU identity from the driver; other cards use the front end's upscaler #225 a release build never routes a game to HIP, so this adds no player route.
  • D3D12 (Proton): WG_GFX 11 helpers and build_dx12.py --gfx 11; a headless Wine runner and Linux/mingw builds of the replay hosts; an F16-only driver probe; and, behind a compile-time macro WG_DEV_GFX11 (off by default; a test pins that no build or release file defines it), an opening of the RX 7900 XT for the D3D12 device. Inside it, a "shortcut" mode (WG_DEV_GFX11_SHORTCUT=1, read only in such a build) runs the unchanged gfx12 kernel set on RDNA 3, with a start-up check that the driver rounds F32 -> F16 -> F32 as the encoder needs.

What it needs outside this repo (not in this PR)

The shortcut runs because of a local dxil-spirv change on top of the raw-fragment support of dxil-spirv #362: under a new DXIL_SPIRV_CONFIG value it emulates the FP8 raw-fragment matrix ops on RDNA 3 (F16 ops, the lane layout RDNA 3 needs, exact FP8 rounding), and vkd3d-proton built with enable_extended_emulation. Whether and how to offer that upstream is open; it overlaps #362 in dxil_ags.cpp.

How it was tested

  • RDNA 4 guard on every commit: the 42 gfx1201 HIP code objects (Model M, K, L, FG) and the 46 gfx12 DXIL files build byte-identical to main's (Linux, ROCm 7.1.1 packages, native DXC 1.10). Please repeat it on your toolchain.
  • On the RX 7900 XT: the HIP card tests (114 tests on the latest full run), CPU tests, the lane-emulator suites, the scan, and a whole Model M frame (17.9 ms at 1706x960 -> 2560x1440 in HIP). Headless under Wine: the D3D12 helper and probe tests, and a whole frame on the native executor in shortcut mode, compared buffer by buffer with the HIP frame on the same inputs (one-FP8-code differences per stage, which a random synthetic network then amplifies).
  • In a game (Cyberpunk 2077, GE-Proton 11-11, 1706x960 -> 2560x1440, ray tracing and frame generation off): the shortcut creates its device on the 7900 XT, passes the probe and runs Model M. The posterized picture of the first run came from OptiScaler's FSR 3.1.6 frame generation, not from this path. Built-in benchmark, same settings: 52.6 fps with Model M against 134.8 fps with FSR 3.1.5 (0.45 ms), so Model M costs about 12 ms per frame on this card today.
  • Speed (dxil-spirv side, not in this PR): the shortcut was bound by per-byte FP8 decode. Shift-form widening (NaN codes kept, K order unchanged) plus one full-size F16 op per step is bit-identical to the first version on every probe and on all 93 frame buffers, and takes a whole frame from 31.6 to 12.4 ms (median of 60, quiet GPU, synthetic model). More work on that side is in progress.
  • Not run: Windows, an RDNA 4 card with these commits, RR on RDNA 3.

Open decisions (yours; none chosen here)

  1. Whether RDNA 3 gets Model M at all, and how it is gated (compile-time macro as here, an environment or ini switch, PCI ids, or kept out of main).
  2. The output standard for RDNA 3: same as the HIP gfx11 path, or a tolerance against RDNA 4.
  3. gfx11 code in shared files behind WG_GFX / family macros (as here) or separate *_gfx11 files.
  4. One family map in Python and C++ kept equal by a test, or one generated from the other.
  5. Whether a patched dxil-spirv / vkd3d-proton is acceptable for players.

Checklist

@phantoslayer21

Copy link
Copy Markdown
Owner

Thanks for this, and for the RX 7900 XT work behind it.

It will get a proper review after 0.6.5 ships; until then the release work comes first, so expect a delay. One thing to know up front: the route the game uses is D3D12 only, and that stays so. A HIP path for gfx1100 can land as a developer or test route, not as the in-game one, so please keep it behind the developer and test entry points.

I've put it on the 0.6.6 milestone.

@phantoslayer21 phantoslayer21 added this to the 0.6.6 milestone Nov 16, 2026
@j0nsh1n

j0nsh1n commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

RDNA 4 source text. This PR leaves the gfx12 program the same, but not its text: the gfx11 operand types, fragment
loads, store forms and wave id helper in swin.hip, encoder.hip and wg_platform.h were defined for every target. A
different compiler can schedule different text differently. On the RDNA 3 line, ROCm 10.1 (clang 24) moved instructions in
swin.co and swin__bmegahg.co for one restructured store even though the program was unchanged. Under 10.1 the code
objects of this PR's head are still identical to main's, so nothing is broken today; the point is that RDNA 4 should not
depend on one compiler's scheduling.

Fix (one commit on top of the head, 4ea9f74). Every gfx11 form now sits under a WG_SWIN_G11 switch (gfx11 or the
emulator), with main's earlier text in the #else. The gfx11-only types and wg_xchg16_u exist only for gfx11 and the
emulator. For gfx12, the preprocessed text of every HIP kernel is again exactly main's. gfx11 and the emulator keep their
behaviour.

Guard. clean/tests/check_gfx12_source_unchanged.py preprocesses every shipped HIP compile for gfx1201 (each tree's own
build.py with its compiler calls recorded, FG's three objects, every other clean/**/*.hip) and requires the text to match
the base tree's token for token. Its unit tests plant an edit (including one only the WG_LZ_PROFILE=1 form reaches) to show
it can fail. Not wired into CI. dx12/tests/gfx11_hip_ref.hip (a host program for the gfx11 helper tests) is named as
unbuildable in the test and as a new form on the command line.

Evidence (ROCm 10.1, clang 24).

  • Guard against the PR's base on main: 160 forms, 0 failed. Same guard with the PR head as it was: 160 of 160 failed.
  • gfx1201 code objects (M, K, L, FG: 42), fixed tree vs the PR's base: byte-identical 42 of 42.
  • gfx1100 code objects (--variants f16wm,mgx: 11) before and after: byte-identical 11 of 11.
  • Guard unit tests: 14 OK. Emulator tests (test_kernels_emu_gfx11, test_e4_gfx11_emu, test_kernels_emu): 36 OK.
  • Only HIP kernel sources and the guard changed; no HLSL, D3D12 or runtime file.

Rebase note. Main has moved since this PR's base (the hybrid form, #256, touches the same three places in
swin.hip). Rebasing needs the same three hand merges; with them resolved, the guard against current main also gives 0
failed (172 forms).

@j0nsh1n

j0nsh1n commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto 0.6.5 (2b31cbc); the branch was force-pushed, so commit ids changed (the gfx12-inert commit from my earlier comment is now 82dfe69).

  • Conflicts were with Model L: the hybrid form as the native D3D12 default (hybrid_full) #256's hybrid form in swin.hip and with 0.6.5's build_dx12.py (probe, the VGPR/RGA check, isa_check, test_only in the manifest): main's behaviour is kept, the gfx11 option is added; the gfx12 manifest keys and order are main's.
  • This PR now also carries the move of the gfx11 helpers into a new wg_gfx11.hlsli (it was at the start of the Model K work before). Without it, Model K: softmax weight rounding in one v_fma_mix*_f16 (exact), with a probe and a logged fallback #254's softmax self-check digest refused the Model K build, because this PR touched wg_platform.hlsli / wg_ags.hlsli; now those two headers are main's again and the release runtime with Model K builds from this PR alone.
  • One test fix: test_gfx11_platform no longer compiles the gfx11-only wg_xchg16_u for gfx1201; it checks that gfx1201 keeps main's single-ds_bpermute wg_xor16 instead.

Checked with ROCm 10.1: the gfx12 source guard (172 compile forms, 0 differ), all 154 gfx12 DXIL files and all 52 gfx1201 code objects byte-identical to main's, the release runtime builds. Not run here: 0.6.5's RGA post-build scan (Windows tool; the DXIL bytes it reads are main's), and nothing on an RDNA 3 card since the rebase.

The block kernel called __builtin_amdgcn_readfirstlane directly and
held a "+s" asm constraint for the RESIDENT kernels. Neither exists in
the host build (-DWG_EMU), so build.py --emu-only and every emulator test
failed with "use of undeclared identifier '__builtin_amdgcn_readfirstlane'"
and "invalid output constraint '+s' in asm".

The wave index now goes through wg_uniform, which is readfirstlane on the
device and the value itself on the host. The asm is left out of the host
build (#ifndef WG_EMU), the same form as upstream PR phantoslayer21#279 for this line.
Device code is unchanged.

Verified: build.py --emu-only fails with those two errors before and
builds after; TestLayoutIndependence passes (27 s); the 27 gfx1201 code
objects have identical SHA-256 before and after (clang 20, ROCm 7.1.1).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
build.py built for gfx12 only.

- build.py takes --arch (gfx1200, gfx1201, gfx1100, gfx1101) and --sources
  (build exactly these kernel sources and write target.json). gfx11 builds
  post, exposure and frame by default (FAMILY_SOURCES; the matrix kernels
  need gfx12 builtins) and no Model K / L kernels, variants, resource gate
  or runtime.
- family.py maps exact target names to gfx12 / gfx11 and refuses everything
  else: gfx1170-1172 are LLVM gfx11 names with the gfx12 matrix design, and
  gfx1102 / gfx1103 / gfx115x are untested. Runner takes its default
  accumulator row table from the build's target.json (WG_D_ROW and the
  d_row argument still win; no file means gfx12, as before).
- clean/README.md gains a short Linux section. --rocm names an SDK in the
  layout of AMD's installer; how the SDK is found is unchanged.

Verified: the new tests fail before and pass after; mapping gfx1170 to
gfx11 turns them red, restored they are green. build.py --arch gfx1100
--sources post.hip exposure.hip frame.hip exits 0. The gfx1201 code objects
(27 Model M, 8 Model K, 4 Model L) keep their SHA-256 (clang 20,
ROCm 7.1.1).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…U_TEST=1)

First run of the HIP host path on a Radeon RX 7900 XT (gfx1100): exposure,
post, filter and frame kernels through the spec 90 runner.

- test_gpu_stages.py (skipped unless WG_GPU_TEST=1; WG_BUILD, WG_ROCM) reuses
  the emulator test classes of exposure, post and filter with the runner
  replaced, so their cases, float64 references and limits are the same, and
  adds color_divide / scale_rgb of frame.hip against FP32 NumPy (bit equal,
  including zeros, denormals, infinities and NaN).
- test_exposure.py: the kernel cases move to KERNEL_CASES / kernel_color;
  test_post_emu.py: TestFilterEmu.KERNELS names the filter kernels. Both are
  behaviour-preserving and let the GPU test pick them up.
- Runner no longer needs swin.co in a HIP build made for a target without the
  Swin kernels (gfx11 so far); a gfx12 build still fails at construction.
- post_filter_quality9 is left out of the GPU filter test: it differs from
  reference/post.py in every value on the host emulator as well (the failure
  already on main); the card agrees with the emulator to 3e-7 relative, so it
  is not a gfx11 defect.

Verified on the card: 8 tests pass (exposure 10 cases, post 14 cases and the
reset test, filter 7 sizes x 6 kernels bit-identical across kernels, frame 50
cases). With the references changed on purpose (frame divisor x1.0001,
exposure alpha x0.8) 57 subtests fail; restored, green. The emulator tests
keep their result (same 7 failures as before, post_filter_quality9). The 27
gfx1201 code objects have the same SHA-256 as before (nothing built
differently).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ange, clear errors)

RDNA 3 (gfx1100, gfx1101) has no FP8 conversion or matrix instruction and
reads the other half of the wave with v_permlanex16 instead of ds_bpermute.
The header knows that now; the gfx12 branches are untouched.

- WG_GFX11 from __gfx1100__ / __gfx1101__ only (exact names; LLVM's gfx1170-
  1172 have the gfx12 matrix design). It forces WG_SOFT_FP8, so every gfx11
  kernel uses the software conversions.
- wg_xchg16_u / wg_xor16 on gfx11 are one v_permlanex16_b32 (selectors
  0x76543210 / 0xfedcba98, fetch-inactive on): lane l gets lane l ^ 16.
  wg_xchg16_u is the 32-bit word form on every target (ds_bpermute on gfx12,
  the lane emulator on the host); wg_mem.h's wg_xor16_u uses it on gfx11.
- wg_wmma_fp8, wg_wmma_f16, wg_dot4_fp8 (WG_NORM_DOT4) and the WG_PROFILE
  counters (wg_realtime, wg_cycles, wg_hwid, wg_drain) are declared with
  __attribute__((error(...))) on gfx11: a call that survives to code
  generation fails with a plain message, while the dead `if (WG_NORM_DOT4 &&
  ...)` branches of swin.hip still compile.
- probe.hip: the matrix kernels and the hardware conversions are not built
  for gfx11; probe_xchg16 (gfx11 and the emulator only, so the gfx12 probe.co
  keeps its hash). build.py's gfx11 default sources gain probe.hip.
- selftest.py: a gfx11 build skips the matrix and hardware-conversion probes,
  and runs the exchange probe (table, word and float forms, NaN payloads)
  plus the software conversion checks on the card.
- pipeline.py takes its default accumulator row table from the build's
  target, as stage_runner.py does (WG_D_ROW and d_row still win).

Verified on a Radeon RX 7900 XT: selftest.py on a gfx1100 build passes, with
the exchange table [16..31, 0..15] (lane l gets lane l ^ 16), 0 word and 0
float mismatches in 512 lanes, and the software conversions bit-exact against
reference/fp8.py (4274 encode, 256 decode, 8544 pack, 0 mismatches). New
tests: the exchange in the emulator (test_kernels_emu), the gfx11 compile
checks (test_gfx11_platform: macro, one v_permlanex16 on gfx11 and
ds_bpermute on gfx12, each error message, dead code still compiles) and
Pipeline's default rows (test_family). With the exchange replaced by the
lane's own value, the card probe reports 512 lanes wrong and the emulator
test fails; restored, both pass. The 27 gfx1201 code objects have the same
SHA-256 as before. The emulator, build, family and stage tests (87) keep
their result: only the 7 failures of post_filter_quality9 that predate this.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…16 form

On gfx11 every FP8 conversion is software. The plain form (rint for the
subnormal range, integer rounding for the rest, selects for the edges) costs
about 19 vector instructions per value. This adds a form with the same
results: z = |x| clamped to [2^-6, 448] gives the magic constant
C = 1.5 * 2^(E + 20); the FP32 sum |x| + C rounds to the E4M3 grid with ties
to even, and the difference sum - C is exact. The code comes from the sum's
bits, or the value goes to F16 bits for the next F16 matrix operand.

- wg_platform.h, gfx11 only (WG_FAST_Q8; -DWG_SLOW_Q8 selects the plain
  form): wg_f32_to_e4m3, wg_pack_q8, wg_pack_q8_inner and wg_pack_q8_nc (for
  |x| <= 464 the bytes of wg_pack_q8) replace the software forms there, and
  wg_q8_to_f16, _inner (NaN -> +-0) and _nc (|x| <= 464) are new. The gfx12
  path and the host emulator keep their forms.
- The sum must not be reassociated: the header refuses -ffast-math and
  -ffinite-math-only, build.py's compiler flags are checked for the others,
  and a compile test shows that the sums disappear with -fassociative-math
  -fno-signed-zeros (test_gfx11_platform.FastE4m3).
- runner/q8_exhaustive.py and probe_q8_sweep (probe.hip, gfx11): all 2^32
  FP32 patterns in 256 chunks of 2^24, converted on the card by the functions
  above and compared with a NumPy reference written from the definition
  (checked against reference/fp8.py, test_gpu_q8.TestReference). About 0.4 s
  per chunk, 1:42 in total, 1.9 GB of host memory, 150 MB on the card.
  test_gpu_q8 (WG_GPU_TEST=1) runs 11 chunks that cover every class of input.

Verified on a Radeon RX 7900 XT (gfx1100, clang 20, ROCm 7.1.1): all 2^32
patterns, 0 mismatches in each of the 7 functions; selftest.py still reports
the software conversions bit-exact against reference/fp8.py. The check can
fail: with the magic constant's mantissa made odd (ties go wrong) 87 patterns
of the 11 chunks differ in every function, and with the F16 saturation at 480
instead of 448 19.9 million differ; restored, 0. The 27 gfx1201 code objects
have the same SHA-256 as before.

Timing, 672 workgroups x 256 lanes, 4096 iterations x 16 values (11.3 G
values) per launch, median of 10 launches, the same loop without a
conversion subtracted (2.1 ms), two runs:
  FP32 -> 8 code bytes (wg_pack_q8): plain 13.6 / 13.7 ms, new 9.0 / 9.1 ms
    (1.5x; 1.21 -> 0.80 ps per value over the device)
  FP32 -> F16 operand: plain pack + widening 19.4 / 18.4 ms, new
    wg_q8_to_f16 6.9 / 6.9 ms (2.7x to 2.8x; 1.69 -> 0.61 ps per value)

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…e of the lane emulator

The Swin kernels call wg_wmma_fp8 and wg_wmma_f16 with the gfx12 operand
shape: a lane holds 8 K values of its half of the wave. On gfx11 these are
now made from the F16 matrix instruction, so the kernels and the weight
blob stay as they are.

- wg_platform.h, gfx11 and the emulator's gfx11 mode (-DWG_EMU_GFX11): the
  lane takes the 8 values of the other half with wg_xchg16_u, puts them
  first or second by its half (K slot (h, j) is index 8h + j, both
  operands alike), widens the 16 codes to F16 and calls wg_wmma16_f16
  (v_wmma_f32_16x16x16_f16). The widening puts the 7 magnitude bits under
  the F16 exponent and multiplies by 256: all 256 codes are exact, and the
  NaN code 0x7f gets bit 14 so that it stays a NaN (a compare and select
  per code made the block kernels spill; this form costs about 3 VGPRs).
  The accumulator rows are 2v + h: d_row "gfx11".
- emu/wg_emu.{h,cpp}: wmma16_f16 models the instruction as measured on an
  RX 7900 XT: output lanes 16h..16h+15 read A and B from the lanes of
  their own half. (The documented layout repeats lanes 0-15 in lanes
  16-31; the wrappers do.) build.py --emu-gfx11 builds wg_emu__gfx11.so;
  Runner(..., backend="emu", arch="gfx11") loads it and takes d_row "gfx11".
- probe.hip: probe_wmma_fp8 / probe_wmma_f16 run on gfx11 through the
  wrappers; probe_wmma16 calls the instruction on raw registers.
- build.py: swin.hip for gfx11 is built with -DWG_SLOW_Q8 (FAMILY_DEFINES).
  With the faster conversion of the previous change the C=128 block kernel
  spills 27 VGPRs and the other block kernels take 30 to 35 more; the
  speed slice budgets that.

Tests: test_kernels_emu_gfx11 runs enc1 (and the other stages of the
emulator tests' extra cases) through the wrappers against the float64
reference with the limits of test_kernels_emu, and the wrappers against
exact arithmetic (random E4M3 and F16 operands bit for bit, every code,
NaN codes). test_gfx11_platform checks the instruction mix of the wrappers
for gfx1100 and gfx1201, and gates the registers of the five block kernels:

  kernel               VGPRs  spills  waves/SIMD
  swin_block_c64        175     0        8
  swin_block_c64_add    175     0        8
  swin_block_c96        206     0        7
  swin_block_c96_add    206     0        7
  swin_block_c128       250     0        5
  (clang 20, ROCm 7.1.1, -DWG_SLOW_Q8; patch_merge_c64 237, 0, 6)

The comparisons can fail: swapping the half select for operand B only gives
91 % of the enc1 skip values more than one step off; using the lane's own
half twice or dropping the multiply by 256 fails the wrapper and enc1 tests;
restored, all pass. Swapping the select for both operands is not caught, by
design: both then take the same K order, and an exact sum does not depend on
it (the hardware rounding does). The 27 gfx1201 code objects have the same
SHA-256 as before; the gfx11 build of swin.hip compiles; no card run yet.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…es and the exchange in divergent code

runner/selftest.py now runs the matrix probes on a gfx11 build (through the
wrappers of the previous change) and has three new probes for gfx11 and the
gfx11 emulator mode (wg_emu__gfx11.so, --emu-library).

- probe_wmma (existing): on the RX 7900 XT it reports d_row
  [[0, 2, .., 14], [1, 3, .., 15]], name "gfx11", for FP8 and F16, with A
  row = lane % 16, B column = lane % 16, K slots paired, 1 x 1 = 1.
- probe_wmma_arithmetic: 32768 random problems per kind (8.4 M outputs)
  against exact float64 sums, whatever the K slot order; error in units of
  2^(E - 24), E the exponent of the largest product. Card, E4M3 (uniform
  random codes): 98.86 % within 4 units, 100 % within 16, largest 17.1
  (analysis: largest 23 on 33.5 M outputs); F16 (normal values): 87.8 %,
  99.96 %, largest 37.3; the same in a second seed. The emulator's exact
  model: 99.9999 % within 4, largest 8. Limits: none beyond 64; E4M3 at
  least 98 % within 4; F16 at least 99.9 % within 16 (the 99 % within 4 of
  the plan is the analysis' figure for the P1b-like data, 99.8 %; uniform
  random codes give 98.9 %).
- probe_wmma16_lanes (probe_wmma16, the instruction on raw registers):
  lanes 16-31 of A and B are not ignored. The lanes 16-31 of the result (odd
  rows 2v + 1) take A and B from lanes 16-31, lanes 0-15 (even rows) from
  lanes 0-15. The wrappers give both halves the same operands, so the sum
  does not depend on this; with other operands in the halves it does.
- probe_xchg16_divergent (probe_xchg16_div, gfx11): wg_xchg16_u in code that
  only some lanes execute (lanes 0-15 only, lanes 16-31 only, odd lanes,
  every fourth lane, a data-dependent half): every lane that executes gets
  the value of lane l ^ 16, also when that lane does not execute (its
  register is read all the same); the other lanes are untouched. 0 wrong
  values in all five modes.

Tests: test_gpu_probes (WG_GPU_TEST=1) asserts the card's report;
test_kernels_emu_gfx11 runs the same probes in the gfx11 emulator mode.
Wave64 post on gfx11 is still not tested.

The probes can fail: built with the multiply by 256 left out of the
widening, the card reports 1 x 1 = 1.5e-05 (2^-16) and the arithmetic probe
is off by up to 1.6e8 units; built without the exchange, 1 x 1 = 2 and the
K pairing fails; an emulator that reads lanes 0-15 for B in all rows fails
the selftest and test_the_instruction_reads_each_half_for_its_own_rows.
Restored, everything passes (card: 5 probe tests; A2-A4 card tests 27).
The 27 gfx1201 code objects have the same SHA-256 as before.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…n.hip in the gfx11 build)

The default build for gfx1100 / gfx1101 now includes swin.hip (built with
-DWG_SLOW_Q8, see the first change), so Runner on a gfx11 build runs the Swin
stages. test_gpu_enc1 (WG_GPU_TEST=1) runs enc1 (swin_block_c64, then
patch_merge_c64) on the card with the emulator tests' four enc1 cases.

Measured on a Radeon RX 7900 XT (gfx1100, clang 20, ROCm 7.1.1, headless):

- Against the float64 reference (the emulator tests' cases, d_row "gfx11"):
  spec 09 holds in all eight outputs: no non-finite value; 0.05 to 0.77 %
  of the values more than one FP8 step off (limit 2 %); mean absolute error
  0.005 to 0.115 % of the reference RMS (limit 1 %).
- The stricter limits of the emulator tests: below 5 % different per case
  holds (0.2 to 3.4 %). Below 1 % different over all cases does not:
  1.0076 % over the four enc1 cases (173,184 values). The gfx11 emulator
  equals the reference on these cases, so the same number is the distance
  card to emulator: 10,076 per million one-code flips, 2,292 per million
  more than one step (per case and output 2,000 to 34,000 per million; the
  analysis expects about 450 per million on realistic data, and random
  networks amplify). Swapping the K order of the odd rows on the card (a
  change of rounding only) flips 1,386 per million, so most of it is not
  the order of the sum. The limit is a product question (the plan's
  question 7); the test is an expected failure, so a change shows.
- Time, Runner.time, median of 50 launches, inputs on the card: 3.17 to
  3.23 ms at the 1706x960 render (480x864 tokens), 6.86 to 6.97 ms at
  2560x1440 (720x1280), over three runs. The tree gives enc1 0.35 ms at the
  first size for an RX 9070 XT (runner/sweep.py GAME_SIZES) and 0.94 ms for
  all of Swin at 1440p Quality (clean/l/SPEED.md): a different card, so about
  9 times only as a rough comparison.

Static ISA of swin_block_c64: 12,260 instructions, 10,519 vector ALU, 44
v_wmma_f32_16x16x16_f16, 124 v_permlanex16_b32, 432 v_pk_mul_f16. Without the
widening the kernel has 3,940 vector ALU instructions: the widening is about
6,600 of them, 150 per matrix instruction.

Checks that can fail: WG_D_ROW=gfx12 on the gfx11 build fails the spec 09
test in all eight outputs; a packer that ignores d_row fails
test_the_row_table_matters (0.0 differ, limit 0.2); restored, both pass.
The 27 gfx1201 code objects have the same SHA-256 as before.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On gfx11 every matrix instruction widened both E4M3 operands to F16 (about 6 600 of the 10 500 vector instructions of
swin_block_c64). The weights, which one wave uses once per window, are now widened when the staged unit is copied to
LDS: once per workgroup, the fragment stored as two planes of 16 rows, read back as 2 x 16 bytes.

Opt-in (WG_F16_W, variants.py "f16w", build.py --arch gfx1100 --variants f16w); the default gfx11 build and every
gfx12 object are unchanged. A unit of the plain block kernels holds half the code bytes, so the LDS stays the same; the
C = 128 kernel keeps codes (with F16 fragments it needs more than 256 registers). The C = 64 kernels take the 8-wave
occupancy floor (without it 256 registers, 27 spilled).

Verified: bit-identical to the default gfx11 build in the gfx11 emulator for the five block kernels and merge / expand
(test_kernels_emu_gfx11.TestVariantsGfx11); identity goes red when one FP32 sum order is changed in the variant and
green when restored. On the RX 7900 XT enc1 meets spec 09 as before; median of 50: 1.90 ms at 1706x960 (3.2 ms before)
and 4.04 ms at 2560x1440 (6.9 ms before).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On top of f16w: each activation operand (the quantized values of a 16-token tile, used by up to 16 matrix instructions)
is exchanged with the other half of the wave and widened to F16 where it is quantized (pack_act), 8 registers per K
group instead of 2, and the softmax probabilities are exchanged once for both value tiles. The register cost makes the
C = 96 kernels spill 30, so the variant applies to the C = 64 kernels only (the others stay as f16w); the C = 64 kernels
take an occupancy floor of 7 waves (8 spills 8).

Opt-in (WG_F16_A, variants.py "f16wa"); the default gfx11 build and every gfx12 object are unchanged.

Verified: bit-identical to the default gfx11 build in the gfx11 emulator (TestVariantsGfx11); the identity goes red when
one FP32 sum order is changed under WG_F16_A and green when restored. enc1 on the RX 7900 XT meets spec 09 as before;
median of 50: 1.63 ms at 1706x960 and 3.48 ms at 2560x1440 (f16w 1.90 and 4.04, default 3.2 and 6.8).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On top of f16wa: the activation operands come from the exact E4M3 rounding done as FP32 arithmetic and v_cvt_pk_rtz_f16_f32
(wg_q8_to_f16), not from code bytes widened again; the code bytes are made only for the stores. That takes the plain
software conversion (about 3 900 vector instructions per window of swin_block_c64) and the widening of the activations
out of the kernel: 6 060 to 3 620 vector instructions in swin_block_c64. The attention output o keeps the code form
(with the F16 form the C = 64 kernels spill 75 registers).

The F16 forms of the rounding (wg_q8m_round, wg_q8_to_f16*) move out of the WG_FAST_Q8 block of wg_platform.h so that they
exist in the default gfx11 build (plain code conversion, -DWG_SLOW_Q8) and in the emulator; the code is the same.
-DWG_SLOW_Q8=0 now selects the faster code conversion.

Opt-in (WG_F16_Q, variants.py "f16wq"); the default gfx11 build and every gfx12 object are unchanged.

Verified: bit-identical to the default gfx11 build in the gfx11 emulator (TestVariantsGfx11); the identity goes red when one
FP32 sum order is changed under WG_F16_Q and green when restored. enc1 on the RX 7900 XT meets spec 09 as before; median
of 50: 1.30 ms at 1706x960 and 2.79 ms at 2560x1440 (f16wa 1.63 and 3.48, default 3.2 and 6.8).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
runner/gfx11_decision.py measures stages enc1 and b0 (C = 128) for the default
gfx11 build and each variant of variants.FAMILY_VARIANTS["gfx11"], and prints
one markdown sheet: spec 09 numbers against the float64 reference, the verdict
word of equiv/compare.py on the same pair (its use on a stage output against our
own reference is marked INFERRED in the sheet), the one-code flip fraction
against the gfx11 emulator, VGPRs, spills and waves per SIMD from the compiler,
the median of 50 launches at 1706x960 and 2560x1440, and a per-frame Swin time
extrapolated from them (INFERRED; the RX 9070 XT figure of l/SPEED.md is a
different-card comparator only). It presents evidence and decides no output
standard. Synthetic weights and inputs only.

tests/test_gfx11_decision.py (CPU only): the verdict mapping on maps with a known
share of one-step differences, the spec 09 limits, the per-frame arithmetic, the
resource-report reading and the structure of the sheet. Checked red on purpose
by reporting DIFFERENT as CLOSE, loosening the 2 % limit and counting b0 five
times per frame; green again after restoring.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…r them (E1)

The FP16 route of the C = 96 (enc2, dec2), C = 128 (b0 to b5) and `_add` (dec1, dec2) block kernels and the weight-resident `_r`
kernels on the RX 7900 XT, each against the float64 reference with the acceptance of test_gpu_enc1 (provisional: the output
standard is the maintainer's decision): spec 09 per case and output, fewer than 5 % of the values different per case, the
"< 1 % over all cases" of the emulator tests. Worst case and output, default gfx11 build, over the cases of each test:

  stage  more than 1 step off   mean abs error (% RMS)   values different   over all cases
  enc2   0.977 %                0.217                    4.54 %             0.638 %
  b0-b5  0.187 %                0.037                    1.12 %             0.631 %
  dec2   0.181 %                0.033                    0.94 %             0.839 %
  dec1   0.225 %                0.027                    0.77 %             0.454 %

No non-finite value, no limit missed, and the overall bound holds for every group (enc1's 1.008 % stays the one expected
failure). Distance to the gfx11 emulator, reported: 4 500 to 6 900 one-code flips per million over all cases.

The `_r` kernels give the plain kernels' bytes for 1, 2 and 5 workgroups (each wave then takes several windows), and the same
reference numbers. Runner.resident = N launches them for enc1 and dec1 (the runtime's swin_resident); at 84 workgroups they
are slower than the plain kernels on this card (enc1 4.0 against 3.2 ms at 1706x960, dec1 4.0 against 3.4 ms).

Finding, pinned as an expected failure and left to slice E2: expand_c96 (107 spilled registers) writes NaN codes for valid
tokens when a wave's second 16-token tile is only partly valid (60, 84, 120 low-resolution tokens; 36, 40, 100, 288 are fine).
The emulator tests' case dec1 12x20 gives 5 120 non-finite values on the card for this reason, so the dec1 cases here use sizes
without that shape; the block kernel `swin_block_c64_add` is not involved (dec1 36x32 and 16x20 are within the limits).

gfx11_gates.py and gfx11_gates.json: compiler-reported VGPRs, spills and waves per SIMD of the 14 block kernels (`_nc` forms
included) in the default gfx11 build and in f16w, f16wa and f16wq. A gate is no spill, at most the recorded registers, at least
the recorded waves; a new table for the gfx11 builds, not clean/dx12/vgpr_limits.json. `python clean/runner/gfx11_gates.py
--rocm <ROCm> --scratch <folder>` checks, `--write` records (clang 20). The merge, expand and fused kernels are not gated: they
spill in these builds (E2).

Verified: the new tests and test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages, test_kernels_emu_gfx11,
test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass. Break on purpose, red then green:
c128 heads and the dec skip add (bottleneck, dec1, dec2 red, enc2 green); c96 heads (enc2 red); the resident loop skipping a
window (bit-identity red); the runner ignoring d_row and the resident option (row-table and launch tests red); one GELU
coefficient changed (the overall bound red, spec 09 still green); the gate check's spill test, and a recorded wave count one
too high (the compile-based gate test red).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…c128w7)

The default gfx11 build leaves swin_block_c128 without an occupancy floor: the allocator takes 250 registers and 5 waves per
SIMD. With a floor of 7 waves (WG_MINW_C128=7) it fits in 206 registers with no spill; a floor of 8 gives 192 and no spill too.
Nothing else changes: no source edit, one entry in variants.py.

On the RX 7900 XT, median of 50 launches, three runs (ms), at 1706x960 and 2560x1440:

  stage  default                         c128w7
  b0     1.089 1.117 1.171 (1.117)       0.911 0.913 0.910 (0.911)        2.139 2.130 2.154 (2.139)    1.887 1.744 1.743 (1.744)
  b1     1.092 1.085 1.094 (1.092)       0.831 0.827 0.830 (0.830)        2.086 2.090 2.073 (2.086)    1.616 1.618 1.615 (1.616)

That is 18 to 24 % less per launch, six launches per frame. A floor of 8 timed the same at 1706x960 and 1.65 ms at 2560x1440.

New on the card: TestVariantsGpu runs every gfx11 variant of the build against the default gfx11 build on seven cases of every
block shape and asks for the same bytes. They are the same for f16w, f16wa, f16wq and c128w7 (the F16 operands are exact and
the K order is the matrix instruction's own). The reference numbers of the stage tests for the c128w7 build equal the default's.
gfx11_gates.json gets the row of c128w7 (206 registers, no spill, 7 waves).

Verified: the new test red with a build whose c128w7 object has one GELU coefficient changed, green again with the real one;
the gate table test red with the c128w7 row deleted; test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages,
test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11 (c128w7 bit-identical to the default in the emulator),
test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
Timing-only builds (output wrong on purpose) show where the C = 128 kernel's 1.1 ms goes on the card: without the W1 and W2
multiplies it takes 0.41 ms, without the QKV multiplies 0.75 ms, with no matrix work at all (gather and store) 0.035 ms. The
matrix instructions themselves are a small part of that (INFERRED: about 770 of them a window); each call also widens its
operands from codes to F16, and f16wq already removed that for the C = 64 kernels, whose activation operands are made once as F16
(C < 96). The wider kernels kept codes because the F16 operands cost registers (C = 96: 30 spilled, then 12 with the attention
output left on codes).

Three knobs name the smallest C they apply to (0 = none, default build unchanged): WG_F16_AWIDE makes the activation operands
F16 operands in the kernels with C >= N, WG_F16_QWIDE makes them straight from the FP32 values there too, WG_SCHED_FRAGS puts a
scheduler barrier after each fragment step of the QKV, Wp, W1 and W2 loops of those kernels. Variant f16wk is f16wq with
AWIDE = QWIDE = SCHED = 128: only swin_block_c128 changes. 238 registers, 6 waves per SIMD, no spill (default: 250 registers,
5 waves). Without the barriers the register count is the same and the kernel is 20 % slower (0.85 against 0.70 ms at
1706x960; the cause was not looked into). F16 weights for
C = 128 (WG_F16_W on this kernel) still spill 94 registers with or without the barriers, so the weights stay codes there.

RX 7900 XT, median of 50 launches, three runs (ms), 1706x960 and 2560x1440:

  stage  default                      f16wk                         default                      f16wk
  b0     1.089 1.117 1.171 (1.117)    0.705 0.715 0.703 (0.705)     2.139 2.130 2.154 (2.139)    1.276 1.290 1.284 (1.284)
  b1     1.092 1.085 1.094 (1.092)    0.628 0.626 0.634 (0.628)     2.086 2.090 2.073 (2.086)    1.193 1.193 1.186 (1.193)

37 to 43 % less per launch; c128w7 (previous commit) gave 18 to 24 %. Same card bytes as the default build on seven cases
(TestVariantsGpu), same reference numbers as the default build (TestBottleneckGpu, TestEnc2Gpu, TestDec2Gpu with the f16wk build).

Verified: bit-identical to the default gfx11 build in the gfx11 emulator for every case (TestVariantsGfx11); the identity goes
red for f16wk, stage b1 only, when the C = 128 operand is scaled by 1.01 under WG_F16_QWIDE, green when restored; gate table
re-recorded with the f16wk row (c64 212 / 7 waves, c96 248 / 5, c128 238 / 6, no spill); test_gpu_enc1, test_gpu_probes,
test_gpu_q8, test_gpu_stages, test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11, test_gfx11_platform,
test_gfx11_decision, test_build_targets, test_family, test_hip_env pass.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…16wm)

The C = 96 block kernels (enc2, dec2) keep code activations under f16wq: F16 operands for the whole block spill 12 registers
(30 with the operands made from FP32). Their MLP is the larger half of the matrix work (W1 and W2 against the norm output and
the hidden values), so WG_F16_M makes only those two operands F16 operands, made once and used by every hidden tile, in the
kernels with C >= 96 that WG_F16_A leaves on codes. Variant f16wm is f16wk with WG_F16_M = 1 and the scheduler barriers of
WG_SCHED_FRAGS in the C = 96 kernels too (WG_SCHED_FRAGS = 96): without them the C = 96 kernels spill 23. The operands are made
from codes, not straight from the FP32 values (that spills 133); the C = 64 kernels are f16wq's and the C = 128 kernel f16wk's.
swin_block_c96[_add]: 252 registers, 5 waves per SIMD, no spill (f16wq: 248, 5 waves).

RX 7900 XT, median of 50 launches, three runs (ms), 1706x960 and 2560x1440:

  stage  f16wq                         f16wm                         f16wq                        f16wm
  enc2   1.523 1.509 1.505 (1.509)     1.328 1.347 1.322 (1.328)     3.002 3.016 2.995 (3.002)    2.648 2.697 2.688 (2.688)
  dec2   1.206 1.199 1.214 (1.206)     1.042 1.036 1.036 (1.036)     2.395 2.384 2.385 (2.385)    2.058 2.055 2.056 (2.056)

12 to 14 % less than f16wq, 44 % (enc2) and 49 % (dec2) less than the default gfx11 build (2.360 and 2.014 ms at 1706x960).
Same card bytes as the default build on seven cases (TestVariantsGpu); the reference numbers of enc2, dec2, dec1 and the
bottleneck with the f16wm build equal the default's (TestEnc2Gpu, TestDec2Gpu, TestDec1Gpu, TestBottleneckGpu).

Verified: bit-identical to the default gfx11 build in the gfx11 emulator for every case (TestVariantsGfx11); the identity goes
red for f16wm, stages enc2 and dec2 only, when the hidden values are scaled by 1.01 under WG_F16_M, green when restored; gate
table re-recorded with the f16wm row (c64 212 / 7 waves, c96 252 / 5, c128 238 / 6, no spill); test_gpu_enc1, test_gpu_probes,
test_gpu_q8, test_gpu_stages, test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11, test_gfx11_platform,
test_gfx11_decision, test_build_targets, test_family, test_hip_env pass.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…x11 (E2)

On gfx11 expand_c96 wrote NaN codes (or wrong values) for valid tokens when a
16-token tile of a wave was only partly valid (60, 84, 120 or 140 low-resolution
tokens). The store was `if (token < ntok)`; the compiler moved the whole
v_wmma chain of the result into that branch (its only user), so the matrix
instruction ran with EXEC partly off. v_wmma needs every lane executing; the
emulator runs all lanes, which is why it never showed. The same move made the
kernels spill (expand_c96 107 registers, patch_merge_c96 101) and patch_merge_c64
and c96 carried the same latent hazard.

Fix at the source, for the gfx11 family only (WG_TILE_STORES_UNGUARDED; the other
targets keep their code, gfx1201 objects are byte-identical): a lane past the last
token repeats the work of the last token and stores the same bytes to the same
place, so there is no branch to move work into. Results are unchanged: the
kernels are bit-identical in the gfx11 emulator.

Spills now 0 in every gfx11 build: patch_merge_c64 185 VGPRs, patch_merge_c96
251, expand_c96 199, expand_c128 248; they join the gate table. On the card
(1706x960): patch_merge_c96 0.639 -> 0.422 ms, expand_c96 0.688 -> 0.418 ms,
patch_merge_c64 and expand_c128 unchanged. The expected-failure test becomes a
passing test and a new test runs each kernel alone with partial tiles against
the gfx11 emulator (red with the branch put back, green with the fix).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… gfx11 (E2)

The default patch_merge widens the weight fragment and the token operand for every
matrix instruction (the half-wave exchange is not merged), so each v_wmma pays for
two widenings. In the variant the loops run K group outermost over 6 (C = 64) or 4
(C = 96) output tiles at a time: each token operand is widened once for all of them
and each weight fragment once for both token groups. Every accumulator adds its K
groups in the same order, so the result is bit-identical (gfx11 emulator, card).
Opt-in (variants.FAMILY_VARIANTS): the default build and the gfx12 objects are
unchanged.

No spills in any kernel of the build (patch_merge_c64 250 VGPRs, patch_merge_c96
143, expand_c96 199, expand_c128 248); the table of resource gates has the mgx build.
On the card (RX 7900 XT, median of 50, 1706x960): patch_merge_c64 0.402 -> 0.354 ms,
patch_merge_c96 0.422 -> 0.311 ms (2560x1440: see the report); expand is unchanged.
Tried and not adopted: 3 token groups per wave (merge c96 0.223 ms, expand_c128 0.342
ms but 9 spilled registers; splitting the expand into passes spills more or gains
nothing), 128-lane workgroups, 1 token group per wave.

Also: gfx11_gates.py --write no longer empties the table when a kernel spills (it
built the table after opening the file); the partial-tile test has a third case
size (36 tokens: a partial third tile with 3 groups per wave).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…rd (E3)

swin_v2.hip with swin.hip's arithmetic (swin__bmegahg.co, the runtime's default for
b0..b5 on gfx12) joins the default gfx11 build: swin_bneck6 (b0 to b5 in one launch) and
swin_mid (enc2 with its merge, b0 to b5, dec2 with its expand in one launch). No kernel
source changes: the file compiles for gfx1100 through the wrappers of wg_platform.h, and
the sync-word protocol is the one of gfx12. build.py builds the object with the family's
defines (variants.FAMILY_MEGA) and writes variants.json for it; the Runner takes it as a
gfx11 variant with the grids of the family's card (FAMILY_MEGA_GROUPS: 126 workgroups for
swin_bneck6, 84 for swin_mid; any grid is correct, these are the resident capacity).

The E2 hazard, checked: a control-flow scan of the compiler's assembly (every basic block
reachable under an s_and_saveexec before its join) finds no v_wmma and no v_permlanex16
under a lane-divergent exec in swin_bneck6, swin_mid or fused_merge. The kernels have no
branch round a store for the compiler to sink a matrix chain into.

Card (RX 7900 XT), against the plain launches of the default build: bit-identical for both
parities, 1 / 3 workgroups and the build's grid, maps with partial windows and one or two
window rows, clamping and clamp-free kernels; sync words zero after every launch. Block by
block against the float64 reference (each block fed the card's own output of the one
before): spec 09 and < 5 % different per block everywhere (worst: b1 12x16 1.27 % different,
0.31 % more than one step off). The six blocks chained against the reference chain compound
the one-code flips of the F16 matrix instruction: 26.5 % more than one step off at 12x16
(plain launches the same bytes). A reference chain given flips of 0.2 % of the values per
block shows the same amplification (34 % at the end), so this is the random synthetic
network, not the kernel.

swin_bneck6: 239 VGPRs, 0 spills, 6 waves per SIMD, 32 KB LDS (gate table, new build
"bmegahg"; the other recorded builds are untouched). swin_mid: 256 VGPRs, 143 VGPRs and 28
SGPRs spilled, 5 waves: correct on the card but NOT gated. Its two C = 96 phases (enc2 with
the merge, dec2 with the expand) spill about 70 registers each, the C = 128 phases none;
plain C = 96 phases spill 79 too, so it is the C = 96 streamed body in the 256-lane queue,
not the fusion. Two knobs tried without effect (WG_V2_PIPE=0: 131 spilled; no scheduling
fences: 141). Open.

Times, median of 50 launches, three runs (ms; 1706x960 | 2560x1440):
  bneck  plain 6.165 | 12.354   megakernel 4.600 | 9.369   plain f16wm 3.449 | 6.808
  mid    plain 10.037 | 20.439  swin_mid  10.127 | 22.272  plain f16wm 5.407 | 11.075
The megakernel is 25 % faster than the default plain blocks and 33 % slower than the f16wm
blocks; swin_mid is level with the plain launches at 1706x960 and 9 % slower at 2560x1440
(the spills), and 1.9 times f16wm. Grid sweep (swin_bneck6, 1706x960): 42 / 84 / 126 / 252
workgroups 12.0 / 6.2 / 4.5 / 4.5 ms. Header in LDS (bmegahg with
WG_V2_MEGA_HDR_GLOBAL=0) 4.40 ms, 128 lanes 8.0 ms: no variant is added.

Verified: TestBmegaGfx11 and TestMidGfx11 (the existing v2 emulator tests with a gfx11
mode) bit-identical to the plain gfx11 kernels; the gfx12 modes of those classes unchanged
and green; clean.tests.test_gpu_swin_mega (new, card) and test_gpu_swin_stages,
test_gpu_enc1 (known expected failure), test_gpu_probes, test_gpu_q8, test_gpu_stages,
test_kernels_emu_gfx11, test_gfx11_platform, test_gfx11_decision, test_gfx11_gates
(compiler check of all ten builds), test_build_targets, test_family, test_hip_env pass.
Break on purpose, red then green: a wrong ping-pong buffer in swin_bneck6 plus a wrong
expand slot in swin_mid (emulator tests: 14 failures; card tests red), a plain build with
the attention switched off (card bit-identity and both reference tests red), the gate
entry removed, the persistent build not built, a gfx12 build accepted as a gfx11 variant.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…t-build blocks on gfx11, and a scan that checks it

The scan of the gfx11 code objects found every v_wmma and v_permlanex16 of
swin_block_c64[_add][_nc] in swin__bmegahg.co under one lane-divergent mask
(s_and_saveexec on vcc): the window loop's `if (win0 + i >= nwin) break;`.
win0 is the wave's first window, a vector value to the compiler, so it could not
see that the test is the same for the whole wave. v_wmma and v_permlanex16 need
every lane executing. At run time the mask is all or nothing (win0 comes from
the wave index), so the kernels passed the card cases; they carried the pattern.

Fix for the gfx11 family only (WG_TILE_STORES_UNGUARDED, the switch of the E2
fix): win0 goes through wg_uniform (readfirstlane), the break becomes a scalar
branch, and the whole loop runs with EXEC full. Other targets keep their code.
Spills of the four kernels fall 48/46/108/90 -> 41/40/11/50 (still above zero:
these kernels are not in the gate table, the runtime uses swin_bneck6).

clean/runner/exec_scan.py: disassembles a code object and runs a forward
data-flow over the control-flow graph of each function: EXEC is partial after
s_and_saveexec, s_and/s_and_not1/s_xor into exec and v_cmpx, restored by
s_or/s_mov exec from a register saved from EXEC (and after a divergent loop),
joins are "partial if any path is". It lists every v_wmma / v_permlanex16 that
may run under a partial mask. test_exec_scan: 15 hand-written functions (each
rule, with a look-alike that must not be flagged) and, with a gfx11 build, a scan
of every kernel the runner launches (0 hits in the fixed build, 4 kernels red in
the build before this commit).

On the card the old code cannot give a wrong result (the mask is uniform), so
TestResidentBlockGpu (partly filled workgroups, 1 to 17 windows, both block
kinds, clamping and clamp-free) is green before and after; it goes red with the
window count off by one (checked). The scan test is the one that is red before.

Not changed, reported by the scan: swin_mid and the fused kernels (_mg, _ex,
fused_merge) still carry the pattern (swin_mid is parked for its owner);
post.co (v_permlanex16 in the reconstruction kernels) and probe_xchg16_div
(deliberate) are flagged too.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…under a lane mask (E4)

kernels/encoder.hip builds for gfx1100 with the gfx11 wrappers unchanged (the
packers and the weight blob are the ones of gfx12: only d_row differs). On a
partly valid 16-token tile the first card build was wrong: the features store
is `if (tok < ntok)`, the compiler moved the whole projection (the half-wave
exchange of the pixel codes, v_permlanex16, and the v_wmma chain) into that
branch, and both need every lane executing. Cases 3 and 13 of the encoder
cases (22x18 padded to 26x20, 4x6: 130 and 12 tokens) gave 8 % and 23 % wrong
features.

Fix, gfx11 family only (WG_ENC_STORE_UNGUARDED, the E2 rule): the features are
stored from every lane; a lane past the last token repeats the work of the last
token (its loads are already clamped to it) and stores the same bytes to the
same place. Other targets keep their code; gfx1201 objects are unchanged.

Resources (gfx1100, clang 20.0.0.rocm): input_encoder 142 VGPRs, 0 spills, 10
waves per SIMD (gate recorded: new "enc0" entry in gfx11_gates.json; nothing
else re-recorded). exec_scan.py: 0 of 113 matrix / exchange instructions under
a lane mask (122 of 126 before the fix); the scan learns the if / else shape
`s_and_not1_saveexec` that the encoder's code contains.

Card (RX 7900 XT), the 14 cases of test_encoder_emu against the float64
reference: 0.000 to 0.003 % more than one step off, mean error at most 0.0015 %
of RMS, 0.000 to 0.095 % of the features different per case (bound 5 %), 0.0575 %
over all cases (467 840 values; the 1 % overall bound of the provisional
acceptance); against the gfx11 emulator 0 to 946 per million one-code flips, the
motion surfaces equal. The emulator test's own limits (0.01 % overall) are an
expected failure on the card. Time, median of 50 launches, three runs:
1706x960 0.637 0.616 0.633 ms, 2560x1440 1.281 1.275 1.289 ms.

Tests: test_gpu_e4.TestEncoderGpu (card; red with the encoder object built before
this fix: cases 3 and 13 and the partly-valid-tile test; red with the store
moved by 16 bytes: every case fails), test_e4_gfx11_emu (the same cases in the gfx11
emulator; red with the same mutation), test_exec_scan (encoder.co joins the
kernels the card runs: red with the old object), test_gfx11_gates.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
kernels/decoder.hip needs no source change for gfx1100: its projection stays in
registers, the pixel stores are scalar-condition stores with no matrix work
behind them, and the scan finds 0 of its 68 matrix / exchange instructions under
a lane mask. The change is the build list (decoder.hip joins the default gfx11
sources), the gate and the tests.

Resources (gfx1100, clang 20.0.0.rocm): coef_decoder 103 VGPRs, 0 spills, 12
waves per SIMD, LDS 10 240 bytes (new "dec0" entry in gfx11_gates.json; nothing
else re-recorded).

Card (RX 7900 XT), the 5 cases of test_decoder_emu, the reset and the border
case, against the float64 reference: A, B and latent in 8-bit codes, C in FP16
steps; at most 0.139 % of a case's values more than one step off, mean error at
most 0.0052 % of RMS, at most 0.139 % different per case and output (bound
5 %), 0.0354 % over all cases (78 988 values; the 1 % overall bound of the
provisional acceptance). Against the gfx11 emulator 0 to 1 389 per million
flips. The emulator test's own limits (99.9 % identical) are an expected failure
on the card. Time, median of 50 launches, three runs: 1706x960 0.431 0.411
0.413 ms, 2560x1440 0.728 0.729 0.728 ms.

Tests: test_gpu_e4.TestDecoderGpu (card), test_e4_gfx11_emu.TestDecoderEmuGfx11
(gfx11 emulator, the emulator test's own limits), test_exec_scan (decoder.co
joins the kernels the card runs), test_gfx11_gates. Red with the pixel-pair store
condition `x0 + 1 < rw` changed to `x0 + 2 < rw`: the card cases and the
emulator cases fail (5 cases, reset and border each); green restored.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
kernels/dec1_dec0.hip builds for gfx1100 unchanged and needs no source change:
its epilogue (dec0's half-wave exchange and projection) runs after the swin
block's work, in every lane, and the scan finds 0 of its 480 matrix / exchange
instructions under a lane mask. The change is the build list (dec1_dec0.hip
joins the default gfx11 sources, with -DWG_SLOW_Q8 like swin.hip, whose block it
includes: 178 VGPRs against 214 with the faster conversion), the gate and the
tests.

Resources (gfx1100, clang 20.0.0.rocm): swin_block_c64_add_dec0 and _nc 178
VGPRs, 0 spills, 8 waves per SIMD, LDS 32 768 bytes (new "dec1dec0" entry in
gfx11_gates.json; nothing else re-recorded).

Card (RX 7900 XT): the fused kernel gives exactly the bytes of dec1 then dec0
(the kernel's contract; the 4 cases of test_dec1_dec0_emu, both clamping
variants). Against the float64 reference, dec0's reference fed the card's own
dec1 output: at most 0.104 % of a case's values more than one step off, mean
error at most 0.003 % of RMS, at most 0.243 % different per case and output
(bound 5 %), 0.0266 % over all cases (71 461 values; the 1 % overall bound of the
provisional acceptance); dec1 itself 0.065 to 0.169 % more than one step off.
Time, median of 50 launches, three runs: 1706x960 fused 3.412 3.446 3.434 ms,
two launches 3.593 3.433 3.446 ms; 2560x1440 fused 7.406 7.404 7.439 ms, two
launches 7.346 7.644 7.372 ms. The fusion is no faster than the two launches on
this card (dec1's block dominates), so no speed variant was added.

Tests: test_gpu_e4.TestDec1Dec0Gpu (card), test_e4_gfx11_emu.TestDec1Dec0EmuGfx11
(fused equals separate in the gfx11 emulator), test_exec_scan (dec1_dec0.co
joins the kernels the card runs), test_gfx11_gates, test_build_targets (red
with the define dropped). Red with dec0's pixel-pair store condition changed:
the three reference cases of the card test fail; green restored.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…sts no longer skip)

The tests of exec_scan ran with ROCM_PATH set to the link folder made for
build.py (it has the compiler, not llvm/bin/llvm-objdump). The tool lookup took
ROCM_PATH only and gave up, so TestExecScanBuild skipped every time in those
runs, the acceptance runs of E3 and E4 included: the "no matrix instruction
under a lane mask" test of the kernels the card runs was never evaluated there.
find_objdump() tries WG_ROCM, ROCM_PATH, /usr/lib64/rocm and /opt/rocm in that
order, then PATH.

Checked by hand on the E4 head build (gfx1100, all kernels the card runs): the
test now runs and passes, 0 findings in them.

Tests: TestFindObjdump (first candidate with the tool wins; a folder without it
is passed over). Red with the existence check dropped (the second case gets the
missing path); green restored.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… post kernels on the gfx11 card

The scan flags all 117 v_permlanex16 of post.co: 39 kernels (post_recon2 ..
post_recon6, post_fused, post_fused3) with three each, the `nb[c] = wg_xor16(hsum[c])`
of the reduced-history step, which the compiler put inside the store's branch
`if (active && has1 && wl < 16 && x + 1 < dw)`. There is no v_wmma in post.co.

Verdict: harmless, not a defect. In that branch only lanes 0 to 15 of a half-wave
pair run, so the lanes the exchange reads (16 to 31) are off, but v_permlanex16
reads the register of a lane whose EXEC bit is off all the same (wg_platform.h
says so; selftest probe_xchg16_div measures it on the card, mode 0 is this exact
shape), and hsum was summed by every lane before the branch. The partner's value
is the partner's real one. No source change, no change of any object: the
gfx1100 post.co of this head and of the E4 head are the same bytes.

The scan cannot prove this (it would need the reaching definitions of the source
register), so it is pinned as a known, explained exception: test_exec_scan
POST_EXCHANGE_KERNELS, exactly these 39 kernels with exactly three
v_permlanex16 and no v_wmma each; a new site, a missing one, or a matrix
instruction in a post kernel fails the test.

Card (RX 7900 XT), new test_gpu_post: the emulator test classes of the post
kernels (recon2 family, recon3 / 5 / 6, recon4, banded, output mode, fused,
fused3) run on the card with their own cases (odd D, partial 32-column tiles, one
D row or column, 3x ratio, NaN / inf inputs, reset): bit-identical to post_recon,
which has no exchange, and the float64 limits where the emulator test has them.
21 tests, 0 failures; 38 of the 39 kernels are launched (a test checks the
list). post_recon2_r_out_diet is not: the runtime's option post_diet=1 is its
only caller and the stage runner has no launch plan for it; its body is
post_recon2_r_out's.

Not used by the Model M path on this card, for the record: the C++ runtime has
no gfx11 build; its defaults would be post_recon2_r_band[_p12] with
post_filter_c1_s4_band (no exchange), post_recon as the fallback.

Red with the exchange removed (wg_xor16 returns its argument on gfx11, post.co
rebuilt): 191 failures in test_gpu_post (test_identical_to_post_recon,
test_identical_to_two_kernel_path, test_exact_variants, test_recon3,
test_recon6); green with the source restored. Red with one kernel dropped from
POST_EXCHANGE_KERNELS, and with a card class left out (the launched-kernels
check); green restored.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…; the _mg / _ex blocks on the gfx11 card

The scan flags swin_block_c64_add_ex, swin_block_c96_add_ex (and their _nc forms)
and the two fused_merge functions the _mg blocks call, in swin.co, swin__mgx.co
and swin__f16wm.co: 216 / 300 / 32 (240 in f16wm for C = 64) matrix and exchange
instructions per function, all under ONE mask. Its origin is the entry test of
the loop over the output tiles, `for (mt = wave; mt < tiles; mt += 8)`: `wave` =
tid >> 5 is a vector value to the compiler, so it tests `tid < 32 * tiles` per
lane (v_cmpx with 0x1ff, 0x2ff, 0xc0, 0x100). The bound is a multiple of 32, a
wave32 holds 32 consecutive thread indices from a multiple of 32, so every lane of
a wave is on one side of it: the mask is full or empty at run time, and an empty
one skips the loop. The stores behind `if (tok[n] >= 0)` are after the matrix
chain, not around it. Verdict: harmless as launched (256 threads), not live.

No source change, on purpose. Making `wave` scalar (wg_uniform, as the persistent
blocks got) does clear the scan, but it adds spilled registers to kernels that
already spill: swin.co _ex_nc c64 / c96 37 -> 47, f16wm 4 -> 29, mgx 37 -> 47
(fused_merge shared with swin_mid kept its code with a template argument; checked
by building it). That is a speed cost for a mask that never happens, so the
pattern is pinned instead: test_exec_scan FUSED_GUARDED lists these functions with
their exact counts, one origin each, a v_cmpx with an immediate that is a multiple
of 32. swin_mid (swin__bmegahg.co, parked) is outside the pin and keeps its result.

Card (RX 7900 XT), new test_gpu_swin_fused (the first card runs of these kernels,
variants base, f16wm, mgx): 23 cases of enc1, enc2, dec1, dec2 with window offset
0 or 2, grids from 4x4 to 36x28 tokens (windows that stick out of the grid, a
workgroup of 8 windows only partly filled), fused kernels against the separate
ones: every output byte equal, no NaN left from the 0x7F fill; against the
float64 reference with the spec 09 limits of the stage tests (5 % different per
case where an output has 2000 values or more). 3 tests per variant, all pass.

Red with fused_merge's store changed to `tok[n] > 0` (swin.co rebuilt):
test_fused_kernels_give_the_bytes_of_the_separate_ones fails in 12 cases and the
reference test in 12; green with the source restored. FUSED_GUARDED red with a
count changed (216 -> 217) and red on the build that has the wg_uniform fix
("not flagged any more"); green restored. Not used by the Model M path on this card:
the C++ runtime has no gfx11 build (it would run the _ex kernels for dec1 and
dec2, WG_SWIN_FUSE_STAGES default); the stage runner's `fuse` geometry runs the
_mg ones.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ime accepts gfx1100

clean/runner/pipeline.py needs libwg_runtime.so, and the repository has none for
a gfx11 target, so this test skips (with the runtime's own message) at this head:
  - build.py's gfx11 branch returns before the runtime step (its docs pass
    --no-runtime); build.runtime_command compiles the unchanged sources fine;
  - device.cpp:398 accepts gfx1200 / gfx1201 only ("no matching HIP device with
    packaged gfx1200/gfx1201 kernels"), and load_module reads <arch>/<name> or
    <arch>.<name> from the build folder.
Not changed here: that is the runtime's device policy, the maintainer's call.

Run in a scratch copy with exactly those three changes (nothing committed; synthetic
weights and frames only), RX 7900 XT, HIP, default kernels (swin.co, swin_bneck6 for
b0..b5, post_recon2_r_band + post_filter_c1_s4_band, separate dec1 / dec0):
 - whole frame, median of 50, three runs: 64x40 -> 96x60 2.993 3.048 2.999 ms;
   45x29 -> 67x43 3.008 2.999 2.995 ms; 1706x960 -> 2560x1440 17.761 18.018
   17.891 ms. Output finite in all cases and frames. Per-stage times are not
   exposed by Pipeline without the interop io (wg_rt_frame_io_timed); the stage
   timings are those of the stage tests.
 - stage by stage against the gfx11 lane emulator on the card's own stage inputs
   (64x40 / 45x29 render): one-code flips 0.000 to 2.368 % per stage (enc1.merged
   the highest), more than one step at most 0.651 %, b0..b5 chain 0.000 %.
   Under the 5 % per case and 2 % bounds.
 - output pixels against the exact (gfx12-arithmetic) emulator frame: 99.97 %
   different (64x40, first frame); per-stage compounding on the random synthetic
   network: enc2.merged 3.2 % of RMS, b5 8.0 %, output 28.3 %. Both the 5 % per
   case and the 1 % overall bound are missed: recorded as expectedFailure.

Tests: test_gpu_pipeline (4 tests: 3 pass, 1 expected failure on the scratch
runtime; 4 skipped at this head). Red with the per-stage flip bound lowered to
0.001 % (9 failures); green restored.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ames of family.py

device.cpp accepted gfx1200 and gfx1201 by two string comparisons, so a gfx11 card was
refused at create. The gate now asks wgrt::target_family (clean/runtime/target_family.h),
the exact-name map of clean/runner/family.py in C++: gfx1200 / gfx1201 -> gfx12,
gfx1100 / gfx1101 -> gfx11, every other name refused (gfx1102, gfx1103, gfx115x, and
gfx1170-1172, which are gfx11 names with the gfx12 matrix design, stay out). The
refusal text names the four targets. Device gets arch(), the exact target whose code
objects the device loads (empty for the emulator), for the runtime to derive the
accumulator row table in the next change.

Code objects are still looked up per exact target as <arch>.<name> or <arch>/<name>
(never the unqualified file), so a build without gfx1100 objects fails closed with
"missing gfx1100 code object". gfx12 devices take the same path as before. The D3D12
gates (rdna4.h, device_d3d12.cpp) are not touched. Whether players ever get a gfx11
build, and how that is gated, is the maintainer's decision; there is no option or
environment variable for it here.

Tests:
 - test_device_targets.cpp (Windows mock; run on Linux through a scratch windows.h
   shim and the mock built as libamdhip64.so, 27 checks): gfx1100 and gfx1101 now
   load their own gfx<arch>.<name> and report arch(); gfx1102, gfx1103, gfx1151,
   gfx1170, gfx1299, gfx11 and "unknown" are refused. The old expectation (gfx1100
   refused) is flipped because this change is the policy change.
 - test_target_family.py (new, Linux and Windows): the C++ map equals family.py for
   every name, plus 16 refused names (decorated, partial, case, other families).
 Red on purpose, then restored: a gfx11 prefix match turned test_device_targets red
 (exit 7: gfx1102 accepted) and test_target_family red; dropping gfx1100 from the map
 turned both red (exit 3 and a map difference).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ce's target family

The matrix instruction of gfx11 returns accumulator row 2 v + h in element v of lane
half h, that of gfx12 row v + 8 h, and the packed weights follow the table (A3:
packing.py D_ROW_LAYOUTS["gfx11"]). clean/runner/pipeline.py sets WgRtConfig.d_row from
the build's target, but clean/runtime/interop never sets it, so a game's runtime always
packed in gfx12 order and the gfx11 kernels would have read the wrong rows.

The runtime now makes its device before it packs, and when the config's d_row is all
zero it takes drow_gfx11() for a device whose arch() maps to gfx11 (target_family.h),
else the gfx12 table as before. A d_row given in the config still wins. No device-less
case changes: the emulator and D3D12 devices have no arch(), so they keep the gfx12
table, and on gfx12 the packed bytes are the ones of before. Creating the device is
the only step that moved up; a device that cannot be made is still reported at the old
place, after the model has loaded, with the same text.

Tests: model_pack_dump takes an optional gfx11 argument, and test_model_pack_cpp packs
all twelve stage blobs of two synthetic models with drow_gfx11 against packing.py's
gfx11 table, byte for byte (the gfx11 and gfx12 blobs differ, so a wrong table cannot
pass either test). Red on purpose, then restored: drow_gfx11 returning the gfx12 table
fails it (seed 1, enc0: 4428 bytes differ). That the runtime picks the table from the
device is checked on the card by the whole-frame test in a later change.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ch>.<name> next to the code objects

build.py's gfx11 branch returned before the runtime step and refused --runtime-only, so
no libwg_runtime.so existed for a gfx11 build. The runtime reads its code objects as
<arch>.<name> (device.cpp, never the unqualified file), and a gfx12 build makes those
copies as it embeds the files; a gfx11 build had neither.

Unless --no-runtime or --sources is given (as before, neither builds a runtime), a gfx11
build now copies every code object to <arch>.<name> (link_arch_objects) and compiles the
runtime with embed=False: no embedded table, since the gfx11 set is not the gfx12 list
that write_embedded requires, and the runtime reads the files from the folder.
--runtime-only does the same step alone for a gfx11 target; --motion-outres-only stays
gfx12 only (new message: the gfx11 target has no motion code objects). gfx12 builds are
untouched: runtime-only still embeds, and the guard's --no-runtime build is the same.
The runtime's own sources are not changed here. clean/README.md follows.

Not done: nothing embeds a gfx11 code-object set (which files a release would ship for
gfx11, and whether it ships at all, is the maintainer's decision), and the Windows
runtime_command (D3D12 interop sources) is not exercised for gfx11.

Tests (test_build_targets, compiler replaced by a function that writes each output):
both gfx11 names make the runtime with no -DWG_EMBEDDED_TABLE and no embedded call, one
<arch>.<name> copy per code object with the same bytes and no other gfx* file;
--no-runtime and --sources make neither; --runtime-only links the existing objects and
does not embed for gfx11 but still embeds for gfx1201. Red on purpose, then restored:
the runtime step switched off (2 failures), the copy removed (3 errors), gfx11
--runtime-only embedding (1), gfx12 --runtime-only no longer embedding (1).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… built for gfx1100

With the runtime's gate, the packing rows and the build of the previous changes,
test_gpu_pipeline no longer skips on a gfx11 build that has its runtime: its 5 tests run
on the RX 7900 XT through HIP, default kernels (swin.co, swin_bneck6 for b0..b5,
post_recon2_r_band + post_filter_c1_s4_band, separate dec1 / dec0), synthetic weights and
frames only. The docstring no longer says the repository lacks the runtime.

New: test_the_runtime_picks_the_rows_of_the_cards_family. A game's interop never sets
WgRtConfig.d_row, so the runtime must choose the gfx11 table itself. Pipeline takes
d_row="runtime" (the config's table stays 0, the reference table of check_packing is the
build's family), and the test checks that every stage blob equals packing.py's on the
gfx11 table, that enc1's blob is not the gfx12 one, that the stage blobs equal those of a
pipeline given the table, and that a frame of two new pipelines is the same bits (new
ones: the cached pipelines have run frames and the frame counter is part of the result).
Red on purpose, then restored: a runtime that ignores the device's family fails it with
"runtime packing of enc0 differs from pack_weights".

Result on the card (OK, expected failures=1): output finite in all cases and frames;
stage by stage against the gfx11 lane emulator on the card's own inputs, one-code flips
0.000 to 2.368 % per stage (enc1.merged 45x29 the highest), more than one step at most
0.651 %, b0..b5 chain 0.000 %; the output pixels against the exact (gfx12-arithmetic)
emulator frame are 99.97 % different at 64x40 (the expected failure, unchanged, synthetic
network). Whole frame, median of 50, three runs: 64x40 -> 96x60 3.120 3.066 3.044 ms;
45x29 -> 67x43 3.073 3.182 3.169 ms; 1706x960 -> 2560x1440 18.664 18.822 19.757 ms (the
card is shared with another project's jobs; the scratch run of the previous change gave
17.8 to 18.0 ms).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…11 (S1)

wg_platform.hlsli takes WG_GFX (12 by default, unchanged; 11 for RDNA 3). Under 11:
no FP8 matrix op, no FP8 convert and no _ags pack (wg_ags.hlsli hides the FP8 codes);
wg_wmma_f16_k8 is one raw F16 instruction on fragments replicated across the half-waves,
wg_wmma_f16_k16 chains two of them (a K=16 step), wg_wmma_f16 keeps the gfx12-shaped
operands of HIP's helper on top of them, wg_d_row is the 2i+h row table, SW_UNPACK_AGS is 0,
and the clamp form of the GELU LUT converts with f32tof16 (RADV rounds a (half) cast to
nearest even). build_dx12.py --gfx 11 defines WG_GFX=11, writes <out>/dx12-gfx11/ with a
"gfx": 11 manifest key, builds only declarations with gfx=11 (default gfx=12) and skips RGA.

Tests: test_build_gfx.py (8, no tools needed) and test_gfx11_helpers.py: headless on the
card under Wine (vkd3d-proton) against numpy and a HIP gfx1100 kernel, bit for bit for the
raw op, the K=16 chain and wg_wmma_f16; convert is RNE; xor16; row table; rounding modes.
They skip without the tools named in the module doc. Red on purpose then restored: row
table i+8h, K=16 chain without its second op, swapped K halves, xor16 skipping a lane,
swapped convert dwords, FP8 code visible on gfx11, build without the WG_GFX define.

Guard: the 46 gfx12 DXIL files are identical to the baseline (manifest too).
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…sts, PSO sweep (S3)

wine_headless_run.sh runs a Windows program under Wine with a null-graphics prefix, a
private runtime folder and the GPU lock, and refuses anything that could open a window
(a prefix with a graphics driver, /tmp or /run/user scratch). build_kernel_replay.py
cross-builds kernel_replay.exe, pso_sweep.exe and gfx11_probe_host.exe with llvm-mingw.
pso_sweep.exe creates each kernel's pipeline with the runtime's buffer or direct root
signature (the runtime itself is not used: its device refuses a card that is not RDNA 4);
pso_sweep.py runs it over a built folder, records a driver crash per kernel and classifies
each DXIL by FP8 channel codes (dxc -dumpbin): a PSO with an FP8 op must fail on RDNA 3,
any other must be created.

Result on the RX 7900 XT (vkd3d-proton, extended emulation): the 46 gfx12 DXIL files give
23 created (all without FP8), 17 failed with 0x80070057 and 5 crashed the driver (the swin
c96 add_* blocks: a page fault inside the driver, reproducible alone), 1 tex kernel not
tried; none unexpected. The 8 gfx11 helper kernels of test_gfx11_helpers all create.
kernel_replay.exe builds and starts but its device refuses the card until the family gate
opens.

test_wine_headless_run.py: 4 tests; red on purpose: scratch check removed, null-graphics
check removed.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…out in the CPU model (S2)

native_driver_probe_f16 (gfx=11 only) runs D = A x B + C with FP16 operands as a K=16 step
of two chained instructions on the replicated fragments of the gfx11 layout. The gfx12
kernel native_driver_probe is untouched (its DXIL is byte for byte the same, and it is not
built for gfx11: it has an FP8 instruction). wmma_probe.h gets namespace gfx11: buffer
layout, fill, expected results (the same matrices, rows 2v+h) and a compare with a tolerance
of 1/4096, because the card's F16 accumulation differs from the exact sum in the last bit in
57 of 256 results (measured; equal to HIP's instruction).

test_wmma_probe.cpp: 20 new checks (the right layout passes, six wrong hardware variants
fail, replicated-half variants pass, all lane-pair swaps of A, B and C fail exactly when
they change what the instruction reads, all element-pair swaps fail, the tolerance accepts
last-bit differences and reports 2e-3 and NaN); the gfx12 checks are unchanged. Red on
purpose: fragment 1 given the K of fragment 0 (6 failures), d_entry rows v+8h (1).
test_gfx11_probe.py (3 tests): builds the kernel with build_dx12.py --gfx 11 and runs it on
the card under Wine with wmma_probe_dump's operands: results within the tolerance; the
gfx12 buffer fails. Red on purpose: the kernel with one instruction instead of two.

Guard: the 46 gfx12 DXIL files are identical to the baseline.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…y normal build (S4)

A compile-time macro, default off and never defined by clean/runner/build.py, lets the
native D3D12 device, the executor choice and the Wine log line take an RX 7900 series
board (PCI 1002:744C, Navi 31, gfx1100; the id read on the development card, shared by
the XT, XTX and GRE; no other RDNA 3 id, 7448 included). The mechanism is one of the
options listed for the maintainer in the RDNA 3 plan (a macro and a local build); the
maintainer may prefer another (environment variable, ini key, a PCI id rule in the
family function, or the gate left closed in main), and this change does not choose
among them. It exists so that the D3D12 kernels for gfx11 can be built and tested
before that decision.

Without the macro nothing a player runs changes: rdna4_identity is untouched (the Ray
Reconstruction runtime still asks only it), dev_gfx11_identity is constant false,
dx12_folder answers "dx12", the device reports no target, the probe is the gfx12 one,
and the order of the manifest read and the device is as before. With the macro on:
 - rdna4.h: kDevGfx11, gfx11_device_id, dev_gfx11_identity (same rule as rdna4_identity:
   the driver's ids, else the Wine fallback's), dx12_folder(arch), dev_gfx11_arch();
   the Wine line and describe_identity know a taken board.
 - executor_select.h: the default rule takes the board (D3D12 executor).
 - device_d3d12.cpp: both identity gates take it; the device reports target gfx1100,
   reads <build>/dx12-gfx11/manifest.json after the device (the family comes from the
   adapter) and runs native_driver_probe_f16 with wmma_probe::gfx11 (F16 only, compared
   within 1/4096).
 - runtime.cpp: the device is made before any manifest is read, the manifest of a build
   is read through dx12_manifest(), the accumulator row table comes from the device's
   target as for HIP (already so), and the native-direct check accepts the gfx11 table.
 The tolerance: the card's F16 accumulation is not exact (57 of 256 probe results differ
 from the exact FP32 sum, by at most 2^-15 here, equal to HIP's instruction). 1/4096 is
 two ulps of the largest expected value (1331) and 1/2500 of the smallest gap between
 distinct expected values (0.607), so a wrong layout cannot pass.

Tests: test_dev_gfx11.cpp (28 checks without the macro, 29 with; both builds pass) and
tests/test_dev_gfx11.py (builds both, 6 source pins incl. that no build or release file
mentions the macro). The existing test_executor_select.cpp: 234 pass without the macro;
with it exactly the 12 cases that assert the RX 7900 XTX id (744C) is refused fail, as
they must. On the card, headless under Wine with the system DXGI of DXVK in a scratch
prefix (the Wine fallback ids, as a player's Proton gives): native_probe_test (new target
of build_kernel_replay.py; the borrowed-device path) is refused without the macro and
passes the F16 probe with it; WG_TEST_DRIVER_PROBE_FAIL=1 fails at float 7; kernel_replay
runs native_driver_probe_f16 as a chain, 256 results within tolerance. Red on purpose,
then restored: 11 host mutations and 6 Wine mutations (see the test files' pins).
Guard: the 46 gfx12 DXIL files and the 42 HIP objects are unchanged (no kernel touched).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…fx12 kernels on RDNA 3, with a rounding check

Inside the WG_DEV_GFX11 opening a second developer-only choice lets a gfx1100 board run
the UNCHANGED gfx12 kernel set (dx12) through dxil-spirv's emulation of the FP8 matrix
operations in the raw-fragment path (DXIL_SPIRV_CONFIG=wmma_rdna3_raw_fp8 in a patched
vkd3d-proton). The mechanism is the environment variable WG_DEV_GFX11_SHORTCUT=1, read
only in a build made with the macro (rdna4.h: without it dev_gfx11_shortcut() is constant
false and the library never asks the environment). It is a placeholder: the maintainer
decides the mechanism later (ini key, option, a PCI id rule, or no shortcut at all), and
shipping a patched dxil-spirv is a product decision this change does not make.

Without the macro nothing changes for a player. With the macro and the variable unset
nothing changes either (the S4 behaviour). With both, the device reports no target (the
gfx12 folder dx12 and the gfx12 accumulator table, as on RDNA 4; the emulation keeps the
gfx12 accumulator layout), and:
 - the borrowed-device probe is the gfx12 one (native_driver_probe, FP16 and emulated
   FP8), compared within wmma_probe::gfx11::kTolerance: the emulation multiplies in the
   FP16 instruction, whose accumulation is not exact on gfx1100 (same reasoning as S4).
 - a rounding check runs at device creation, for owned and borrowed devices: the
   kernels round to FP16 through an F32 -> F16 -> F32 pair of the matrix conversion, and
   the driver's compiler folds such a pair away unless the conversions are exact (found
   with the shortcut probe: a fold gives encoder features one FP8 step off). The kernel
   native_round_probe (gfx=11 only, so the gfx12 set keeps its 46 files) converts 256
   inputs that all have bits below FP16's precision, including exact ties of both
   parities; the device compares bit for bit with the nearest-even result computed on the
   CPU (wmma_probe.h, round16) and refuses with a log line otherwise. The kernel is read
   from <build>/dx12-gfx11/manifest.json (read_manifest takes an optional kernel name).

Tests: test_shortcut_probe.cpp (27 checks, built with and without the macro) and
tests/test_dev_gfx11_shortcut.py (both builds plus 5 source pins). On the card, headless
under Wine: the shortcut passes the probe and the rounding check; with
DXIL_SPIRV_CONFIG unset the compiler folds the pair and the device is refused ("the
driver does not round F32 -> F16 -> F32"); WG_TEST_ROUNDING_FAIL=1 gives the same
refusal. Red on purpose, then restored: 8 mutations (see the test files' pins).
Guard: the 46 gfx12 DXIL files are unchanged.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ess Wine, compared with the HIP whole frame (S12)

frame_host.cpp is a small Windows host (cross-built with the other replay hosts by
build_kernel_replay.py) that loads wronggpu_runtime.dll, creates the native D3D12 backend
(backend 3) on a synthetic raw-tensor model, runs frames from input files, dumps the
named buffers of each frame and times frames with the runtime's own timestamps. Nothing
reads a model.wgcm. It exists because the whole-frame machinery (compare.py,
time_frames.py) drives the runtime library from Python on Windows, and a headless run
under Wine on Linux has no Python on the Windows side.

frame_parity.py prepares the model, the inputs and the host's spec (the same synthetic
frames as compare.py), runs the HIP or lane-emulator whole frame natively through
pipeline.py, and compares every named buffer and the output image of two runs: bytes
that differ per buffer (buffers that a fusion leaves unwritten, stale ones, and the
decoder's other format are listed and not counted, as in compare.py), the output in
FP16 steps and relative RMS, and the GPU times. It measures; it is not a gate, since the
output standard of an RDNA 3 route is the maintainer's decision.

Checked: a run against itself gives 0 differing bytes and byte-equal output; the
D3D12 shortcut frames at 64x40, 853x481 and 1706x960 run end to end (see the S12 report).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… text, and a guard checks it

The RDNA 3 changes in swin.hip, encoder.hip and wg_platform.h leave the gfx12 program the same, but not its text: the
gfx11 operand types, fragment loads, store forms and wave id helper were written for every target, and the patch
merge and expand stores gained braces. A different compiler can schedule different text differently: on the RDNA 3
line, ROCm 10.1 (clang 24) moved instructions in swin.co and swin__bmegahg.co for one restructured store, although the
program was unchanged. "The same instructions under today's compiler" is not a promise RDNA 4 can rely on; the text is.

Every gfx11 form now sits under WG_SWIN_G11 (gfx11 or the emulator) with main's earlier text in the #else. In
wg_platform.h the v4h / v16h types and wg_xchg16_u exist only for gfx11 and the emulator; encoder.hip gets the same
treatment for its gfx11 store. clean/tests/check_gfx12_source_unchanged.py preprocesses every shipped HIP compile for
gfx1201 (each tree's own build.py with its compiler calls recorded, FG's three objects, every other clean/**/*.hip)
and requires the text to match the base tree's token for token; its unit tests plant an edit to prove it can fail.
dx12/tests/gfx11_hip_ref.hip, a host program of the gfx11 helper tests that no gfx12 build compiles, is named as
unbuildable in the test and as a new form on the command line.

Verified (ROCm 10.1, clang 24; only HIP kernel sources and the guard changed, no D3D12 or runtime file):
- guard against the base on main (91436c4): 160 forms, 0 failed. The same guard with the unfixed branch head
  (0cd1a91) as the tree: 160 of 160 failed.
- gfx1201 objects (M, K, L, FG: 42) of the fixed tree and of 91436c4: byte-identical 42 of 42. (The unfixed head
  0cd1a91 also gave 42 of 42 identical objects under 10.1: the source differed, the code objects did not.)
- gfx1100 objects (--variants f16wm,mgx: 11) before and after: byte-identical 11 of 11.
- guard unit tests 14 OK; the emulator tests test_kernels_emu_gfx11, test_e4_gfx11_emu and test_kernels_emu: 36 OK.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
….hlsli and wg_ags.hlsli are main's again

On main aa85515, model K's softmax self-check records kSourceDigest, a SHA-256 over the
shader headers the softmax kernels include (wg_platform.hlsli and wg_ags.hlsli among them).
S1's gfx11 edits of those two files changed the digest and build_dx12.py refused to build.

Moved: everything gfx11 in wg_platform.hlsli (WG_GFX default and checks, wg_in_upper_half,
wg_xor16_4, wg_wmma_f16_k8 / _k16, the gfx11 wg_wmma_f16, wg_d_row, the gfx12 wg_d_row) and in
wg_ags.hlsli (the WG_GFX default and the #if hiding the FP8 codes) into the new
hlsl/wg_gfx11.hlsli. A gfx11 kernel includes it instead of wg_platform.hlsli (it includes
that one itself); gfx12 builds of it only include the platform layer. Under WG_GFX 11 it
renames the platform layer's gfx12-shaped wg_wmma_f16 while including it, and hides the
FP8 identifiers again with #undef / #define (a use does not compile, as before).
native_game_helpers.hlsl, tests/gfx11_helpers.hlsl and the compile check include it.
wg_gelu_lut.hlsli, swin_common.hlsli and build_dx12.py are not in the digest and stay.

Verified: smx_selfcheck_digest.digest() equals the committed header again; the two gfx11
probe DXIL files are byte-identical to the ones built before the move; test_build_gfx
(8) and TestGfx11Compile (4, including the FP8-is-hidden negative controls) pass; the
card tests skip (no GPU in this slice).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… of compiling it for gfx1201

Since the gfx12 source of every HIP kernel is main's text again, the gfx11 exchange wg_xchg16_u exists only for gfx11
and the emulator. test_exchange_is_one_permlane_on_gfx11_and_bpermute_on_gfx12 still compiled it for gfx1201 and so
failed ("use of undeclared identifier 'wg_xchg16_u'"). Its gfx12 half now checks what the design promises: wg_xor16 on
gfx1201 is the one ds_bpermute exchange of main's text (no permlane), and wg_xchg16_u does not compile there. The gfx11
half is unchanged.

Verified (ROCm 10.1, clang 24): the test is red before (the old test fails on the parent commit) and green after; with
the wg_platform.h of the commit before the gfx12-inert fix (wg_xchg16_u for every target) the new test fails, restored
it passes. Without a ROCm clang the whole module skips, as on CI's Windows runners.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…Wine runner test skips off Linux

build_dx12.py's digest label used os.path.relpath, which raises ValueError when a source (a test's temp
folder on C:) and the repository (D:) are on different drives. A source outside the repository is now
labelled ext/<file name>. A new test in test_build_gfx.py simulates the cross-drive error (red before the
fix, green after). test_wine_headless_run.py runs a bash script, so it is skipped unless on Linux.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants