Skip to content

Model K on RDNA 3: native narrow stages and faster encoders (draft, stacked on #259) - #276

Draft
j0nsh1n wants to merge 57 commits into
phantoslayer21:mainfrom
j0nsh1n:rdna3/k-native
Draft

j0nsh1n wants to merge 57 commits into
phantoslayer21:mainfrom
j0nsh1n:rdna3/k-native

Conversation

@j0nsh1n

@j0nsh1n j0nsh1n commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Draft, for visibility; not asking for a merge. Model K on RDNA 3 (RX 7900 XT, gfx1100) under Proton: the measured state, native RDNA 3 forms of K's narrow network stages (and faster encoders for the two-window stages) behind the existing developer gate, and one timing observation you may want to know about. Nothing here changes what a player gets: everything RDNA 3 sits behind WG_DEV_GFX11 (off in every normal build), and RDNA 4 is unchanged (see "RDNA 4" below). Stacked on #259 at c3f6b75 (rebased on 0.6.5, 2b31cbc); since #259's branch lives on my fork, this PR targets main and GitHub also lists #259's commits: this PR's own are the 16 from 9d176e7 on. Open decisions are listed at the end.

Testing still to do on my side: nothing has been re-run on an RX 7900 XT since the rebase onto 0.6.5 (the card is out of my machine right now). Before this is more than a draft I'll re-run the card checks (stage-level native vs shortcut, the pipeline sweep, the whole-frame byte comparison) and an in-game session on the rebased code. Details under "Not re-run after the rebase on 0.6.5".

Where Model K stands on the RX 7900 XT (1706x960 -> 2560x1440, synthetic model random_network_params(11), headless Wine, a local vkd3d-proton build with enable_extended_emulation plus the dxil-spirv change of #259; medians of 300-540 frames; measured before the rebase onto 0.6.5)

Route Whole frame, median Output
Unchanged RDNA 4 kernels through the translator ("shortcut") 2.39 ms inside K's limits against the NumPy reference (head mean 1.6-1.8e-3, max 1.7-2.1e-2 of RMS)
+ native RDNA 3 forms for four narrow stages 2.33 ms byte-identical to the shortcut on every stage buffer and image (216 of 216 files)
+ the faster encoders, native list with k_stage2_e1 added 0.069 ms less than the line above byte-identical to the shortcut (216 of 216 files, 9 of 9 rounds)
+ your opt-in k_approx=a1,a2 2.06 ms (before the encoders) inside K's head limits (compact head; a2 refuses k_head=full)

In game (Cyberpunk 2077 under Proton, 1706x960 -> 2560x1440, a real converted Model K, max settings, no RT, runtime GPU timestamps, median of 300-frame windows of one session each; developer builds, not a controlled benchmark):

Card Path Total per frame
RX 7900 XT RDNA 3 route, native list, exact 2.07 ms (median; input 0.44, network 1.11, output 0.50)
RX 7900 XT same + k_approx=a1 1.76 ms, with visibly more artifacts in game
RX 9070 XT main's RDNA 4 kernels, self-check passed 1.62 ms (input 0.32, network 0.76, output 0.53)

Image quality is not settled: on both cards K shows flicker on thin bright detail, some ghosting and occasional single-pixel specks, and it looks the same on both. Offline, captured real frames replayed through the runtime matched the in-game output byte for byte on the 7900 XT, and the NumPy reference of K reproduced the same specks at the same pixels, so it looks like K's own response to this game's inputs rather than a porting bug. k_approx=a1 looked clearly worse in game than the exact path. The game route records into the game's DIRECT queue, so the COMPUTE-queue observation below does not apply there.

What this changes and why

The shortcut runs K's RDNA 4 kernels on RDNA 3 by translating every 16x16x16 F16 matrix step, including an exchange between the two half-waves per call, because RDNA 3 lays out matrix operands differently. A native form can lay out its operands once and keep them, so it needs about 8 times fewer exchanges (e2: 74 against 578).

In order (16 commits on #259 at c3f6b75; all signed off). The move of the gfx11 helpers into wg_gfx11.hlsli (which keeps main's two headers and the #254 softmax digest untouched) is in #259 itself now:

  • 9d176e7 Native K = 16 operand helpers for gfx11 (16 halves per lane, laid out by the kernel) and a CPU emulation that runs the D3D12 kernel text itself, with one-hot probes.
  • 1ad1729 One template that gives 13 gfx11 kernels <port>__g11 for K's narrow stages (e0/e1/e2/d2/d1, the two-window forms with both heads, the three round-to-odd forms). An A-row map makes every result come out in the RDNA 4 layout, so the RDNA 4 weight blob is used unchanged. Each product sums K in the shortcut's order.
  • 1f151a9, c2f62a5, ca1822f k_native_check.py (native against shortcut on the same input bytes, GPU tool with a CPU dry run) and card tests that skip without a card.
  • ca15d25 A diagnostics-only k_smx_test=mix_unchecked hook (see decision 2).
  • 5b5ae65, d07ab51 WG_DEV_GFX11_NATIVE_K (read only in a WG_DEV_GFX11 build, on top of WG_DEV_GFX11_SHORTCUT=1): 1 launches every __g11 kernel the dx12-gfx11 manifest has, a comma list launches only those; the rest of the frame stays on the shortcut.
  • 0c0d25a The native K = 16 opcode is accepted by the translator only with the shortcut's configuration (it is the one that keeps the F32 -> F16 -> F32 rounding pairs unfolded).
  • 56fb35f Docs: the native K test's docstring names the list form.
  • d007c10, 1a6b3f6, 66b067e, 02382c0 Faster encoders for the native stages: register knobs for the two-window encoders (all off by default), KN_A16 (16-byte arena accesses as one 128-bit load or store, off by default), each stage kernel declares its measured register and load knobs, and KN_MERGE_STEP is limited to 1, 2 or 4 (a preprocessor error otherwise). k_stage2_e0 goes from 240 to 192 VGPRs (8 waves, the shortcut's) and k_stage2_e1 from 256 to 240 (6 waves, the shortcut's); stage times are 5 to 22 % lower on all 13 native kernels, and the whole frame is 0.069 ms lower with k_stage2_e1 added to the list. Output is byte-identical to the shortcut on every stage output and on all 216 whole-frame files.
  • e180608 Developer builds refuse Model K stage pointers that are not 16-byte aligned (the 128-bit accesses need them).
  • 5d3ddbe New in this rebase: the parser of the WG_DEV_GFX11_NATIVE_K list no longer uses std::string::find, because 0.6.5's R1: no in-game route to HIP; GPU identity from the driver; other cards use the front end's upscaler #225 test requires that rdna4.h has no find( outside comments. Entries, exact-name matching and stray-comma handling are unchanged.

The 15 commits before the last have the same patches as before the rebase on 0.6.5 (git range-diff shows = for each). The translator side (a developer-only opcode for "16 pre-laid-out halves, one K = 16 step") lives in the same local dxil-spirv change as #259, not in this repo.

RDNA 4

Source text equal to main's under the guard, and objects equal under ROCm 10.1, built on 2b31cbc:

Please repeat the guard on your toolchain.

How it was tested

  • CPU: the emulation of the kernel text matches the RDNA 4 ports byte for byte on all 39 stage outputs (three size sets with partial windows, three offsets) and stays inside spec 49's stage limits against NumPy. Negative controls (a wrong row table, a removed bound, the list separator) fail. After the rebase on 0.6.5: the native K, gfx11 emulation, knob, alignment, list-selection and softmax digest tests pass, and the rest of the CPU set fails only where it already fails without this stack (7 modules in the run before RDNA 3 (RX 7900 XT): Model M in HIP on gfx1100 and a developer-only D3D12 route under Proton (draft) #259 was finalised; the one left in test_gfx11_platform, test_the_faster_quantizer_is_what_spills, fails on RDNA 3 (RX 7900 XT): Model M in HIP on gfx1100 and a developer-only D3D12 route under Proton (draft) #259 itself: it expects a register spill that ROCm 10.1's compiler no longer produces, and CI does not run it).
  • Card, stage level (RX 7900 XT, before the rebase): native equal to the shortcut byte for byte on 78 cases with the staged kernel names, with a negative control that was caught; 16 of 16 gfx11 kernels create their pipelines.
  • Card, whole frame: shortcut, all native and the lists are byte-identical on 216 of 216 files.
  • An independent review of the native work (FIX-PASS, three fixes made and re-checked).
  • Game: Cyberpunk 2077 on Proton with a real converted Model K, native list: runs, 2.07 ms exact (above).

Not re-run after the rebase on 0.6.5

Nothing was run on an RDNA 3 card after the rebase. The card tests (stage check, PSO sweep, whole frame, k_native_check on the card), the Wine whole-frame runs and the in-game runs are from before it. After the rebase I checked, without an RDNA 3 card: the 16 kernel files of the dx12-gfx11 set are byte-identical to the set that was measured on the card, and the 16 commits' patches are unchanged. Not run at all: Windows, and an RDNA 4 card in game with these commits.

An observation about the queue (not changed in this PR)

On this card and driver (Mesa RADV 27.0-devel), identical K frames on the own-buffer executor's COMPUTE queue run at two speeds about 0.24 ms apart, in runs of 2-3 frames, at the same shader, memory and fabric clocks. It is not overlap of k_warm / k_out_color, not history ping-pong, not power state and not the timing queries (frames with no timing at all show it too). The same work on a DIRECT queue has no slow frames and is not slower (240 of 240 frames in one band; output byte-identical). A bottom-of-pipe timestamp mid-network or between all network stages also suppresses it, at a small cost. It looks like a scheduling bubble specific to the separate compute queue on this driver; I have not found its owner. A game recording into its own DIRECT queue would likely not see it. If you see anything similar on RDNA 4 I would like to know.

Open decisions (yours; none chosen here)

  1. Whether Model K gets RDNA 3 at all, and through which route (the shortcut alone, or with native stages).
  2. The Model K: softmax weight rounding in one v_fma_mix*_f16 (exact), with a probe and a logged fallback #254 create-time self-check refuses K on gfx11 with either route, because it compares against gfx1201 hashes and gfx11's matrix sums differ. Options: refuse (today), accept when the probes pass and the stage outputs are inside spec 49, or keep a gfx11 reference hash set. Every number above used the diagnostics-only mix_unchecked hook.
  3. The output standard for native K: equal to the shortcut's bits (designed for and measured), or inside spec 49.
  4. How the runtime finds native kernels: the <name>__g11 lookup used here, or a digest rule that skips gfx11-only declarations so the native files can use the port names.
  5. Whether a developer-only translator opcode is acceptable at all.
  6. Whether the own-buffer executor should use a DIRECT queue on RDNA 3 (or generally), given the observation above.

Checklist

The block kernel called __builtin_amdgcn_readfirstlane directly and
held a "+s" asm constraint for the RESIDENT kernels. Neither exists in
the host build (-DWG_EMU), so build.py --emu-only and every emulator test
failed with "use of undeclared identifier '__builtin_amdgcn_readfirstlane'"
and "invalid output constraint '+s' in asm".

The wave index now goes through wg_uniform, which is readfirstlane on the
device and the value itself on the host. The asm is left out of the host
build (#ifndef WG_EMU), the same form as upstream PR phantoslayer21#279 for this line.
Device code is unchanged.

Verified: build.py --emu-only fails with those two errors before and
builds after; TestLayoutIndependence passes (27 s); the 27 gfx1201 code
objects have identical SHA-256 before and after (clang 20, ROCm 7.1.1).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
build.py built for gfx12 only.

- build.py takes --arch (gfx1200, gfx1201, gfx1100, gfx1101) and --sources
  (build exactly these kernel sources and write target.json). gfx11 builds
  post, exposure and frame by default (FAMILY_SOURCES; the matrix kernels
  need gfx12 builtins) and no Model K / L kernels, variants, resource gate
  or runtime.
- family.py maps exact target names to gfx12 / gfx11 and refuses everything
  else: gfx1170-1172 are LLVM gfx11 names with the gfx12 matrix design, and
  gfx1102 / gfx1103 / gfx115x are untested. Runner takes its default
  accumulator row table from the build's target.json (WG_D_ROW and the
  d_row argument still win; no file means gfx12, as before).
- clean/README.md gains a short Linux section. --rocm names an SDK in the
  layout of AMD's installer; how the SDK is found is unchanged.

Verified: the new tests fail before and pass after; mapping gfx1170 to
gfx11 turns them red, restored they are green. build.py --arch gfx1100
--sources post.hip exposure.hip frame.hip exits 0. The gfx1201 code objects
(27 Model M, 8 Model K, 4 Model L) keep their SHA-256 (clang 20,
ROCm 7.1.1).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…U_TEST=1)

First run of the HIP host path on a Radeon RX 7900 XT (gfx1100): exposure,
post, filter and frame kernels through the spec 90 runner.

- test_gpu_stages.py (skipped unless WG_GPU_TEST=1; WG_BUILD, WG_ROCM) reuses
  the emulator test classes of exposure, post and filter with the runner
  replaced, so their cases, float64 references and limits are the same, and
  adds color_divide / scale_rgb of frame.hip against FP32 NumPy (bit equal,
  including zeros, denormals, infinities and NaN).
- test_exposure.py: the kernel cases move to KERNEL_CASES / kernel_color;
  test_post_emu.py: TestFilterEmu.KERNELS names the filter kernels. Both are
  behaviour-preserving and let the GPU test pick them up.
- Runner no longer needs swin.co in a HIP build made for a target without the
  Swin kernels (gfx11 so far); a gfx12 build still fails at construction.
- post_filter_quality9 is left out of the GPU filter test: it differs from
  reference/post.py in every value on the host emulator as well (the failure
  already on main); the card agrees with the emulator to 3e-7 relative, so it
  is not a gfx11 defect.

Verified on the card: 8 tests pass (exposure 10 cases, post 14 cases and the
reset test, filter 7 sizes x 6 kernels bit-identical across kernels, frame 50
cases). With the references changed on purpose (frame divisor x1.0001,
exposure alpha x0.8) 57 subtests fail; restored, green. The emulator tests
keep their result (same 7 failures as before, post_filter_quality9). The 27
gfx1201 code objects have the same SHA-256 as before (nothing built
differently).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ange, clear errors)

RDNA 3 (gfx1100, gfx1101) has no FP8 conversion or matrix instruction and
reads the other half of the wave with v_permlanex16 instead of ds_bpermute.
The header knows that now; the gfx12 branches are untouched.

- WG_GFX11 from __gfx1100__ / __gfx1101__ only (exact names; LLVM's gfx1170-
  1172 have the gfx12 matrix design). It forces WG_SOFT_FP8, so every gfx11
  kernel uses the software conversions.
- wg_xchg16_u / wg_xor16 on gfx11 are one v_permlanex16_b32 (selectors
  0x76543210 / 0xfedcba98, fetch-inactive on): lane l gets lane l ^ 16.
  wg_xchg16_u is the 32-bit word form on every target (ds_bpermute on gfx12,
  the lane emulator on the host); wg_mem.h's wg_xor16_u uses it on gfx11.
- wg_wmma_fp8, wg_wmma_f16, wg_dot4_fp8 (WG_NORM_DOT4) and the WG_PROFILE
  counters (wg_realtime, wg_cycles, wg_hwid, wg_drain) are declared with
  __attribute__((error(...))) on gfx11: a call that survives to code
  generation fails with a plain message, while the dead `if (WG_NORM_DOT4 &&
  ...)` branches of swin.hip still compile.
- probe.hip: the matrix kernels and the hardware conversions are not built
  for gfx11; probe_xchg16 (gfx11 and the emulator only, so the gfx12 probe.co
  keeps its hash). build.py's gfx11 default sources gain probe.hip.
- selftest.py: a gfx11 build skips the matrix and hardware-conversion probes,
  and runs the exchange probe (table, word and float forms, NaN payloads)
  plus the software conversion checks on the card.
- pipeline.py takes its default accumulator row table from the build's
  target, as stage_runner.py does (WG_D_ROW and d_row still win).

Verified on a Radeon RX 7900 XT: selftest.py on a gfx1100 build passes, with
the exchange table [16..31, 0..15] (lane l gets lane l ^ 16), 0 word and 0
float mismatches in 512 lanes, and the software conversions bit-exact against
reference/fp8.py (4274 encode, 256 decode, 8544 pack, 0 mismatches). New
tests: the exchange in the emulator (test_kernels_emu), the gfx11 compile
checks (test_gfx11_platform: macro, one v_permlanex16 on gfx11 and
ds_bpermute on gfx12, each error message, dead code still compiles) and
Pipeline's default rows (test_family). With the exchange replaced by the
lane's own value, the card probe reports 512 lanes wrong and the emulator
test fails; restored, both pass. The 27 gfx1201 code objects have the same
SHA-256 as before. The emulator, build, family and stage tests (87) keep
their result: only the 7 failures of post_filter_quality9 that predate this.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…16 form

On gfx11 every FP8 conversion is software. The plain form (rint for the
subnormal range, integer rounding for the rest, selects for the edges) costs
about 19 vector instructions per value. This adds a form with the same
results: z = |x| clamped to [2^-6, 448] gives the magic constant
C = 1.5 * 2^(E + 20); the FP32 sum |x| + C rounds to the E4M3 grid with ties
to even, and the difference sum - C is exact. The code comes from the sum's
bits, or the value goes to F16 bits for the next F16 matrix operand.

- wg_platform.h, gfx11 only (WG_FAST_Q8; -DWG_SLOW_Q8 selects the plain
  form): wg_f32_to_e4m3, wg_pack_q8, wg_pack_q8_inner and wg_pack_q8_nc (for
  |x| <= 464 the bytes of wg_pack_q8) replace the software forms there, and
  wg_q8_to_f16, _inner (NaN -> +-0) and _nc (|x| <= 464) are new. The gfx12
  path and the host emulator keep their forms.
- The sum must not be reassociated: the header refuses -ffast-math and
  -ffinite-math-only, build.py's compiler flags are checked for the others,
  and a compile test shows that the sums disappear with -fassociative-math
  -fno-signed-zeros (test_gfx11_platform.FastE4m3).
- runner/q8_exhaustive.py and probe_q8_sweep (probe.hip, gfx11): all 2^32
  FP32 patterns in 256 chunks of 2^24, converted on the card by the functions
  above and compared with a NumPy reference written from the definition
  (checked against reference/fp8.py, test_gpu_q8.TestReference). About 0.4 s
  per chunk, 1:42 in total, 1.9 GB of host memory, 150 MB on the card.
  test_gpu_q8 (WG_GPU_TEST=1) runs 11 chunks that cover every class of input.

Verified on a Radeon RX 7900 XT (gfx1100, clang 20, ROCm 7.1.1): all 2^32
patterns, 0 mismatches in each of the 7 functions; selftest.py still reports
the software conversions bit-exact against reference/fp8.py. The check can
fail: with the magic constant's mantissa made odd (ties go wrong) 87 patterns
of the 11 chunks differ in every function, and with the F16 saturation at 480
instead of 448 19.9 million differ; restored, 0. The 27 gfx1201 code objects
have the same SHA-256 as before.

Timing, 672 workgroups x 256 lanes, 4096 iterations x 16 values (11.3 G
values) per launch, median of 10 launches, the same loop without a
conversion subtracted (2.1 ms), two runs:
  FP32 -> 8 code bytes (wg_pack_q8): plain 13.6 / 13.7 ms, new 9.0 / 9.1 ms
    (1.5x; 1.21 -> 0.80 ps per value over the device)
  FP32 -> F16 operand: plain pack + widening 19.4 / 18.4 ms, new
    wg_q8_to_f16 6.9 / 6.9 ms (2.7x to 2.8x; 1.69 -> 0.61 ps per value)

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…e of the lane emulator

The Swin kernels call wg_wmma_fp8 and wg_wmma_f16 with the gfx12 operand
shape: a lane holds 8 K values of its half of the wave. On gfx11 these are
now made from the F16 matrix instruction, so the kernels and the weight
blob stay as they are.

- wg_platform.h, gfx11 and the emulator's gfx11 mode (-DWG_EMU_GFX11): the
  lane takes the 8 values of the other half with wg_xchg16_u, puts them
  first or second by its half (K slot (h, j) is index 8h + j, both
  operands alike), widens the 16 codes to F16 and calls wg_wmma16_f16
  (v_wmma_f32_16x16x16_f16). The widening puts the 7 magnitude bits under
  the F16 exponent and multiplies by 256: all 256 codes are exact, and the
  NaN code 0x7f gets bit 14 so that it stays a NaN (a compare and select
  per code made the block kernels spill; this form costs about 3 VGPRs).
  The accumulator rows are 2v + h: d_row "gfx11".
- emu/wg_emu.{h,cpp}: wmma16_f16 models the instruction as measured on an
  RX 7900 XT: output lanes 16h..16h+15 read A and B from the lanes of
  their own half. (The documented layout repeats lanes 0-15 in lanes
  16-31; the wrappers do.) build.py --emu-gfx11 builds wg_emu__gfx11.so;
  Runner(..., backend="emu", arch="gfx11") loads it and takes d_row "gfx11".
- probe.hip: probe_wmma_fp8 / probe_wmma_f16 run on gfx11 through the
  wrappers; probe_wmma16 calls the instruction on raw registers.
- build.py: swin.hip for gfx11 is built with -DWG_SLOW_Q8 (FAMILY_DEFINES).
  With the faster conversion of the previous change the C=128 block kernel
  spills 27 VGPRs and the other block kernels take 30 to 35 more; the
  speed slice budgets that.

Tests: test_kernels_emu_gfx11 runs enc1 (and the other stages of the
emulator tests' extra cases) through the wrappers against the float64
reference with the limits of test_kernels_emu, and the wrappers against
exact arithmetic (random E4M3 and F16 operands bit for bit, every code,
NaN codes). test_gfx11_platform checks the instruction mix of the wrappers
for gfx1100 and gfx1201, and gates the registers of the five block kernels:

  kernel               VGPRs  spills  waves/SIMD
  swin_block_c64        175     0        8
  swin_block_c64_add    175     0        8
  swin_block_c96        206     0        7
  swin_block_c96_add    206     0        7
  swin_block_c128       250     0        5
  (clang 20, ROCm 7.1.1, -DWG_SLOW_Q8; patch_merge_c64 237, 0, 6)

The comparisons can fail: swapping the half select for operand B only gives
91 % of the enc1 skip values more than one step off; using the lane's own
half twice or dropping the multiply by 256 fails the wrapper and enc1 tests;
restored, all pass. Swapping the select for both operands is not caught, by
design: both then take the same K order, and an exact sum does not depend on
it (the hardware rounding does). The 27 gfx1201 code objects have the same
SHA-256 as before; the gfx11 build of swin.hip compiles; no card run yet.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…es and the exchange in divergent code

runner/selftest.py now runs the matrix probes on a gfx11 build (through the
wrappers of the previous change) and has three new probes for gfx11 and the
gfx11 emulator mode (wg_emu__gfx11.so, --emu-library).

- probe_wmma (existing): on the RX 7900 XT it reports d_row
  [[0, 2, .., 14], [1, 3, .., 15]], name "gfx11", for FP8 and F16, with A
  row = lane % 16, B column = lane % 16, K slots paired, 1 x 1 = 1.
- probe_wmma_arithmetic: 32768 random problems per kind (8.4 M outputs)
  against exact float64 sums, whatever the K slot order; error in units of
  2^(E - 24), E the exponent of the largest product. Card, E4M3 (uniform
  random codes): 98.86 % within 4 units, 100 % within 16, largest 17.1
  (analysis: largest 23 on 33.5 M outputs); F16 (normal values): 87.8 %,
  99.96 %, largest 37.3; the same in a second seed. The emulator's exact
  model: 99.9999 % within 4, largest 8. Limits: none beyond 64; E4M3 at
  least 98 % within 4; F16 at least 99.9 % within 16 (the 99 % within 4 of
  the plan is the analysis' figure for the P1b-like data, 99.8 %; uniform
  random codes give 98.9 %).
- probe_wmma16_lanes (probe_wmma16, the instruction on raw registers):
  lanes 16-31 of A and B are not ignored. The lanes 16-31 of the result (odd
  rows 2v + 1) take A and B from lanes 16-31, lanes 0-15 (even rows) from
  lanes 0-15. The wrappers give both halves the same operands, so the sum
  does not depend on this; with other operands in the halves it does.
- probe_xchg16_divergent (probe_xchg16_div, gfx11): wg_xchg16_u in code that
  only some lanes execute (lanes 0-15 only, lanes 16-31 only, odd lanes,
  every fourth lane, a data-dependent half): every lane that executes gets
  the value of lane l ^ 16, also when that lane does not execute (its
  register is read all the same); the other lanes are untouched. 0 wrong
  values in all five modes.

Tests: test_gpu_probes (WG_GPU_TEST=1) asserts the card's report;
test_kernels_emu_gfx11 runs the same probes in the gfx11 emulator mode.
Wave64 post on gfx11 is still not tested.

The probes can fail: built with the multiply by 256 left out of the
widening, the card reports 1 x 1 = 1.5e-05 (2^-16) and the arithmetic probe
is off by up to 1.6e8 units; built without the exchange, 1 x 1 = 2 and the
K pairing fails; an emulator that reads lanes 0-15 for B in all rows fails
the selftest and test_the_instruction_reads_each_half_for_its_own_rows.
Restored, everything passes (card: 5 probe tests; A2-A4 card tests 27).
The 27 gfx1201 code objects have the same SHA-256 as before.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…n.hip in the gfx11 build)

The default build for gfx1100 / gfx1101 now includes swin.hip (built with
-DWG_SLOW_Q8, see the first change), so Runner on a gfx11 build runs the Swin
stages. test_gpu_enc1 (WG_GPU_TEST=1) runs enc1 (swin_block_c64, then
patch_merge_c64) on the card with the emulator tests' four enc1 cases.

Measured on a Radeon RX 7900 XT (gfx1100, clang 20, ROCm 7.1.1, headless):

- Against the float64 reference (the emulator tests' cases, d_row "gfx11"):
  spec 09 holds in all eight outputs: no non-finite value; 0.05 to 0.77 %
  of the values more than one FP8 step off (limit 2 %); mean absolute error
  0.005 to 0.115 % of the reference RMS (limit 1 %).
- The stricter limits of the emulator tests: below 5 % different per case
  holds (0.2 to 3.4 %). Below 1 % different over all cases does not:
  1.0076 % over the four enc1 cases (173,184 values). The gfx11 emulator
  equals the reference on these cases, so the same number is the distance
  card to emulator: 10,076 per million one-code flips, 2,292 per million
  more than one step (per case and output 2,000 to 34,000 per million; the
  analysis expects about 450 per million on realistic data, and random
  networks amplify). Swapping the K order of the odd rows on the card (a
  change of rounding only) flips 1,386 per million, so most of it is not
  the order of the sum. The limit is a product question (the plan's
  question 7); the test is an expected failure, so a change shows.
- Time, Runner.time, median of 50 launches, inputs on the card: 3.17 to
  3.23 ms at the 1706x960 render (480x864 tokens), 6.86 to 6.97 ms at
  2560x1440 (720x1280), over three runs. The tree gives enc1 0.35 ms at the
  first size for an RX 9070 XT (runner/sweep.py GAME_SIZES) and 0.94 ms for
  all of Swin at 1440p Quality (clean/l/SPEED.md): a different card, so about
  9 times only as a rough comparison.

Static ISA of swin_block_c64: 12,260 instructions, 10,519 vector ALU, 44
v_wmma_f32_16x16x16_f16, 124 v_permlanex16_b32, 432 v_pk_mul_f16. Without the
widening the kernel has 3,940 vector ALU instructions: the widening is about
6,600 of them, 150 per matrix instruction.

Checks that can fail: WG_D_ROW=gfx12 on the gfx11 build fails the spec 09
test in all eight outputs; a packer that ignores d_row fails
test_the_row_table_matters (0.0 differ, limit 0.2); restored, both pass.
The 27 gfx1201 code objects have the same SHA-256 as before.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On gfx11 every matrix instruction widened both E4M3 operands to F16 (about 6 600 of the 10 500 vector instructions of
swin_block_c64). The weights, which one wave uses once per window, are now widened when the staged unit is copied to
LDS: once per workgroup, the fragment stored as two planes of 16 rows, read back as 2 x 16 bytes.

Opt-in (WG_F16_W, variants.py "f16w", build.py --arch gfx1100 --variants f16w); the default gfx11 build and every
gfx12 object are unchanged. A unit of the plain block kernels holds half the code bytes, so the LDS stays the same; the
C = 128 kernel keeps codes (with F16 fragments it needs more than 256 registers). The C = 64 kernels take the 8-wave
occupancy floor (without it 256 registers, 27 spilled).

Verified: bit-identical to the default gfx11 build in the gfx11 emulator for the five block kernels and merge / expand
(test_kernels_emu_gfx11.TestVariantsGfx11); identity goes red when one FP32 sum order is changed in the variant and
green when restored. On the RX 7900 XT enc1 meets spec 09 as before; median of 50: 1.90 ms at 1706x960 (3.2 ms before)
and 4.04 ms at 2560x1440 (6.9 ms before).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On top of f16w: each activation operand (the quantized values of a 16-token tile, used by up to 16 matrix instructions)
is exchanged with the other half of the wave and widened to F16 where it is quantized (pack_act), 8 registers per K
group instead of 2, and the softmax probabilities are exchanged once for both value tiles. The register cost makes the
C = 96 kernels spill 30, so the variant applies to the C = 64 kernels only (the others stay as f16w); the C = 64 kernels
take an occupancy floor of 7 waves (8 spills 8).

Opt-in (WG_F16_A, variants.py "f16wa"); the default gfx11 build and every gfx12 object are unchanged.

Verified: bit-identical to the default gfx11 build in the gfx11 emulator (TestVariantsGfx11); the identity goes red when
one FP32 sum order is changed under WG_F16_A and green when restored. enc1 on the RX 7900 XT meets spec 09 as before;
median of 50: 1.63 ms at 1706x960 and 3.48 ms at 2560x1440 (f16w 1.90 and 4.04, default 3.2 and 6.8).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On top of f16wa: the activation operands come from the exact E4M3 rounding done as FP32 arithmetic and v_cvt_pk_rtz_f16_f32
(wg_q8_to_f16), not from code bytes widened again; the code bytes are made only for the stores. That takes the plain
software conversion (about 3 900 vector instructions per window of swin_block_c64) and the widening of the activations
out of the kernel: 6 060 to 3 620 vector instructions in swin_block_c64. The attention output o keeps the code form
(with the F16 form the C = 64 kernels spill 75 registers).

The F16 forms of the rounding (wg_q8m_round, wg_q8_to_f16*) move out of the WG_FAST_Q8 block of wg_platform.h so that they
exist in the default gfx11 build (plain code conversion, -DWG_SLOW_Q8) and in the emulator; the code is the same.
-DWG_SLOW_Q8=0 now selects the faster code conversion.

Opt-in (WG_F16_Q, variants.py "f16wq"); the default gfx11 build and every gfx12 object are unchanged.

Verified: bit-identical to the default gfx11 build in the gfx11 emulator (TestVariantsGfx11); the identity goes red when one
FP32 sum order is changed under WG_F16_Q and green when restored. enc1 on the RX 7900 XT meets spec 09 as before; median
of 50: 1.30 ms at 1706x960 and 2.79 ms at 2560x1440 (f16wa 1.63 and 3.48, default 3.2 and 6.8).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
runner/gfx11_decision.py measures stages enc1 and b0 (C = 128) for the default
gfx11 build and each variant of variants.FAMILY_VARIANTS["gfx11"], and prints
one markdown sheet: spec 09 numbers against the float64 reference, the verdict
word of equiv/compare.py on the same pair (its use on a stage output against our
own reference is marked INFERRED in the sheet), the one-code flip fraction
against the gfx11 emulator, VGPRs, spills and waves per SIMD from the compiler,
the median of 50 launches at 1706x960 and 2560x1440, and a per-frame Swin time
extrapolated from them (INFERRED; the RX 9070 XT figure of l/SPEED.md is a
different-card comparator only). It presents evidence and decides no output
standard. Synthetic weights and inputs only.

tests/test_gfx11_decision.py (CPU only): the verdict mapping on maps with a known
share of one-step differences, the spec 09 limits, the per-frame arithmetic, the
resource-report reading and the structure of the sheet. Checked red on purpose
by reporting DIFFERENT as CLOSE, loosening the 2 % limit and counting b0 five
times per frame; green again after restoring.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…r them (E1)

The FP16 route of the C = 96 (enc2, dec2), C = 128 (b0 to b5) and `_add` (dec1, dec2) block kernels and the weight-resident `_r`
kernels on the RX 7900 XT, each against the float64 reference with the acceptance of test_gpu_enc1 (provisional: the output
standard is the maintainer's decision): spec 09 per case and output, fewer than 5 % of the values different per case, the
"< 1 % over all cases" of the emulator tests. Worst case and output, default gfx11 build, over the cases of each test:

  stage  more than 1 step off   mean abs error (% RMS)   values different   over all cases
  enc2   0.977 %                0.217                    4.54 %             0.638 %
  b0-b5  0.187 %                0.037                    1.12 %             0.631 %
  dec2   0.181 %                0.033                    0.94 %             0.839 %
  dec1   0.225 %                0.027                    0.77 %             0.454 %

No non-finite value, no limit missed, and the overall bound holds for every group (enc1's 1.008 % stays the one expected
failure). Distance to the gfx11 emulator, reported: 4 500 to 6 900 one-code flips per million over all cases.

The `_r` kernels give the plain kernels' bytes for 1, 2 and 5 workgroups (each wave then takes several windows), and the same
reference numbers. Runner.resident = N launches them for enc1 and dec1 (the runtime's swin_resident); at 84 workgroups they
are slower than the plain kernels on this card (enc1 4.0 against 3.2 ms at 1706x960, dec1 4.0 against 3.4 ms).

Finding, pinned as an expected failure and left to slice E2: expand_c96 (107 spilled registers) writes NaN codes for valid
tokens when a wave's second 16-token tile is only partly valid (60, 84, 120 low-resolution tokens; 36, 40, 100, 288 are fine).
The emulator tests' case dec1 12x20 gives 5 120 non-finite values on the card for this reason, so the dec1 cases here use sizes
without that shape; the block kernel `swin_block_c64_add` is not involved (dec1 36x32 and 16x20 are within the limits).

gfx11_gates.py and gfx11_gates.json: compiler-reported VGPRs, spills and waves per SIMD of the 14 block kernels (`_nc` forms
included) in the default gfx11 build and in f16w, f16wa and f16wq. A gate is no spill, at most the recorded registers, at least
the recorded waves; a new table for the gfx11 builds, not clean/dx12/vgpr_limits.json. `python clean/runner/gfx11_gates.py
--rocm <ROCm> --scratch <folder>` checks, `--write` records (clang 20). The merge, expand and fused kernels are not gated: they
spill in these builds (E2).

Verified: the new tests and test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages, test_kernels_emu_gfx11,
test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass. Break on purpose, red then green:
c128 heads and the dec skip add (bottleneck, dec1, dec2 red, enc2 green); c96 heads (enc2 red); the resident loop skipping a
window (bit-identity red); the runner ignoring d_row and the resident option (row-table and launch tests red); one GELU
coefficient changed (the overall bound red, spec 09 still green); the gate check's spill test, and a recorded wave count one
too high (the compile-based gate test red).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…c128w7)

The default gfx11 build leaves swin_block_c128 without an occupancy floor: the allocator takes 250 registers and 5 waves per
SIMD. With a floor of 7 waves (WG_MINW_C128=7) it fits in 206 registers with no spill; a floor of 8 gives 192 and no spill too.
Nothing else changes: no source edit, one entry in variants.py.

On the RX 7900 XT, median of 50 launches, three runs (ms), at 1706x960 and 2560x1440:

  stage  default                         c128w7
  b0     1.089 1.117 1.171 (1.117)       0.911 0.913 0.910 (0.911)        2.139 2.130 2.154 (2.139)    1.887 1.744 1.743 (1.744)
  b1     1.092 1.085 1.094 (1.092)       0.831 0.827 0.830 (0.830)        2.086 2.090 2.073 (2.086)    1.616 1.618 1.615 (1.616)

That is 18 to 24 % less per launch, six launches per frame. A floor of 8 timed the same at 1706x960 and 1.65 ms at 2560x1440.

New on the card: TestVariantsGpu runs every gfx11 variant of the build against the default gfx11 build on seven cases of every
block shape and asks for the same bytes. They are the same for f16w, f16wa, f16wq and c128w7 (the F16 operands are exact and
the K order is the matrix instruction's own). The reference numbers of the stage tests for the c128w7 build equal the default's.
gfx11_gates.json gets the row of c128w7 (206 registers, no spill, 7 waves).

Verified: the new test red with a build whose c128w7 object has one GELU coefficient changed, green again with the real one;
the gate table test red with the c128w7 row deleted; test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages,
test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11 (c128w7 bit-identical to the default in the emulator),
test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
Timing-only builds (output wrong on purpose) show where the C = 128 kernel's 1.1 ms goes on the card: without the W1 and W2
multiplies it takes 0.41 ms, without the QKV multiplies 0.75 ms, with no matrix work at all (gather and store) 0.035 ms. The
matrix instructions themselves are a small part of that (INFERRED: about 770 of them a window); each call also widens its
operands from codes to F16, and f16wq already removed that for the C = 64 kernels, whose activation operands are made once as F16
(C < 96). The wider kernels kept codes because the F16 operands cost registers (C = 96: 30 spilled, then 12 with the attention
output left on codes).

Three knobs name the smallest C they apply to (0 = none, default build unchanged): WG_F16_AWIDE makes the activation operands
F16 operands in the kernels with C >= N, WG_F16_QWIDE makes them straight from the FP32 values there too, WG_SCHED_FRAGS puts a
scheduler barrier after each fragment step of the QKV, Wp, W1 and W2 loops of those kernels. Variant f16wk is f16wq with
AWIDE = QWIDE = SCHED = 128: only swin_block_c128 changes. 238 registers, 6 waves per SIMD, no spill (default: 250 registers,
5 waves). Without the barriers the register count is the same and the kernel is 20 % slower (0.85 against 0.70 ms at
1706x960; the cause was not looked into). F16 weights for
C = 128 (WG_F16_W on this kernel) still spill 94 registers with or without the barriers, so the weights stay codes there.

RX 7900 XT, median of 50 launches, three runs (ms), 1706x960 and 2560x1440:

  stage  default                      f16wk                         default                      f16wk
  b0     1.089 1.117 1.171 (1.117)    0.705 0.715 0.703 (0.705)     2.139 2.130 2.154 (2.139)    1.276 1.290 1.284 (1.284)
  b1     1.092 1.085 1.094 (1.092)    0.628 0.626 0.634 (0.628)     2.086 2.090 2.073 (2.086)    1.193 1.193 1.186 (1.193)

37 to 43 % less per launch; c128w7 (previous commit) gave 18 to 24 %. Same card bytes as the default build on seven cases
(TestVariantsGpu), same reference numbers as the default build (TestBottleneckGpu, TestEnc2Gpu, TestDec2Gpu with the f16wk build).

Verified: bit-identical to the default gfx11 build in the gfx11 emulator for every case (TestVariantsGfx11); the identity goes
red for f16wk, stage b1 only, when the C = 128 operand is scaled by 1.01 under WG_F16_QWIDE, green when restored; gate table
re-recorded with the f16wk row (c64 212 / 7 waves, c96 248 / 5, c128 238 / 6, no spill); test_gpu_enc1, test_gpu_probes,
test_gpu_q8, test_gpu_stages, test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11, test_gfx11_platform,
test_gfx11_decision, test_build_targets, test_family, test_hip_env pass.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…16wm)

The C = 96 block kernels (enc2, dec2) keep code activations under f16wq: F16 operands for the whole block spill 12 registers
(30 with the operands made from FP32). Their MLP is the larger half of the matrix work (W1 and W2 against the norm output and
the hidden values), so WG_F16_M makes only those two operands F16 operands, made once and used by every hidden tile, in the
kernels with C >= 96 that WG_F16_A leaves on codes. Variant f16wm is f16wk with WG_F16_M = 1 and the scheduler barriers of
WG_SCHED_FRAGS in the C = 96 kernels too (WG_SCHED_FRAGS = 96): without them the C = 96 kernels spill 23. The operands are made
from codes, not straight from the FP32 values (that spills 133); the C = 64 kernels are f16wq's and the C = 128 kernel f16wk's.
swin_block_c96[_add]: 252 registers, 5 waves per SIMD, no spill (f16wq: 248, 5 waves).

RX 7900 XT, median of 50 launches, three runs (ms), 1706x960 and 2560x1440:

  stage  f16wq                         f16wm                         f16wq                        f16wm
  enc2   1.523 1.509 1.505 (1.509)     1.328 1.347 1.322 (1.328)     3.002 3.016 2.995 (3.002)    2.648 2.697 2.688 (2.688)
  dec2   1.206 1.199 1.214 (1.206)     1.042 1.036 1.036 (1.036)     2.395 2.384 2.385 (2.385)    2.058 2.055 2.056 (2.056)

12 to 14 % less than f16wq, 44 % (enc2) and 49 % (dec2) less than the default gfx11 build (2.360 and 2.014 ms at 1706x960).
Same card bytes as the default build on seven cases (TestVariantsGpu); the reference numbers of enc2, dec2, dec1 and the
bottleneck with the f16wm build equal the default's (TestEnc2Gpu, TestDec2Gpu, TestDec1Gpu, TestBottleneckGpu).

Verified: bit-identical to the default gfx11 build in the gfx11 emulator for every case (TestVariantsGfx11); the identity goes
red for f16wm, stages enc2 and dec2 only, when the hidden values are scaled by 1.01 under WG_F16_M, green when restored; gate
table re-recorded with the f16wm row (c64 212 / 7 waves, c96 252 / 5, c128 238 / 6, no spill); test_gpu_enc1, test_gpu_probes,
test_gpu_q8, test_gpu_stages, test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11, test_gfx11_platform,
test_gfx11_decision, test_build_targets, test_family, test_hip_env pass.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…x11 (E2)

On gfx11 expand_c96 wrote NaN codes (or wrong values) for valid tokens when a
16-token tile of a wave was only partly valid (60, 84, 120 or 140 low-resolution
tokens). The store was `if (token < ntok)`; the compiler moved the whole
v_wmma chain of the result into that branch (its only user), so the matrix
instruction ran with EXEC partly off. v_wmma needs every lane executing; the
emulator runs all lanes, which is why it never showed. The same move made the
kernels spill (expand_c96 107 registers, patch_merge_c96 101) and patch_merge_c64
and c96 carried the same latent hazard.

Fix at the source, for the gfx11 family only (WG_TILE_STORES_UNGUARDED; the other
targets keep their code, gfx1201 objects are byte-identical): a lane past the last
token repeats the work of the last token and stores the same bytes to the same
place, so there is no branch to move work into. Results are unchanged: the
kernels are bit-identical in the gfx11 emulator.

Spills now 0 in every gfx11 build: patch_merge_c64 185 VGPRs, patch_merge_c96
251, expand_c96 199, expand_c128 248; they join the gate table. On the card
(1706x960): patch_merge_c96 0.639 -> 0.422 ms, expand_c96 0.688 -> 0.418 ms,
patch_merge_c64 and expand_c128 unchanged. The expected-failure test becomes a
passing test and a new test runs each kernel alone with partial tiles against
the gfx11 emulator (red with the branch put back, green with the fix).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… gfx11 (E2)

The default patch_merge widens the weight fragment and the token operand for every
matrix instruction (the half-wave exchange is not merged), so each v_wmma pays for
two widenings. In the variant the loops run K group outermost over 6 (C = 64) or 4
(C = 96) output tiles at a time: each token operand is widened once for all of them
and each weight fragment once for both token groups. Every accumulator adds its K
groups in the same order, so the result is bit-identical (gfx11 emulator, card).
Opt-in (variants.FAMILY_VARIANTS): the default build and the gfx12 objects are
unchanged.

No spills in any kernel of the build (patch_merge_c64 250 VGPRs, patch_merge_c96
143, expand_c96 199, expand_c128 248); the table of resource gates has the mgx build.
On the card (RX 7900 XT, median of 50, 1706x960): patch_merge_c64 0.402 -> 0.354 ms,
patch_merge_c96 0.422 -> 0.311 ms (2560x1440: see the report); expand is unchanged.
Tried and not adopted: 3 token groups per wave (merge c96 0.223 ms, expand_c128 0.342
ms but 9 spilled registers; splitting the expand into passes spills more or gains
nothing), 128-lane workgroups, 1 token group per wave.

Also: gfx11_gates.py --write no longer empties the table when a kernel spills (it
built the table after opening the file); the partial-tile test has a third case
size (36 tokens: a partial third tile with 3 groups per wave).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…rd (E3)

swin_v2.hip with swin.hip's arithmetic (swin__bmegahg.co, the runtime's default for
b0..b5 on gfx12) joins the default gfx11 build: swin_bneck6 (b0 to b5 in one launch) and
swin_mid (enc2 with its merge, b0 to b5, dec2 with its expand in one launch). No kernel
source changes: the file compiles for gfx1100 through the wrappers of wg_platform.h, and
the sync-word protocol is the one of gfx12. build.py builds the object with the family's
defines (variants.FAMILY_MEGA) and writes variants.json for it; the Runner takes it as a
gfx11 variant with the grids of the family's card (FAMILY_MEGA_GROUPS: 126 workgroups for
swin_bneck6, 84 for swin_mid; any grid is correct, these are the resident capacity).

The E2 hazard, checked: a control-flow scan of the compiler's assembly (every basic block
reachable under an s_and_saveexec before its join) finds no v_wmma and no v_permlanex16
under a lane-divergent exec in swin_bneck6, swin_mid or fused_merge. The kernels have no
branch round a store for the compiler to sink a matrix chain into.

Card (RX 7900 XT), against the plain launches of the default build: bit-identical for both
parities, 1 / 3 workgroups and the build's grid, maps with partial windows and one or two
window rows, clamping and clamp-free kernels; sync words zero after every launch. Block by
block against the float64 reference (each block fed the card's own output of the one
before): spec 09 and < 5 % different per block everywhere (worst: b1 12x16 1.27 % different,
0.31 % more than one step off). The six blocks chained against the reference chain compound
the one-code flips of the F16 matrix instruction: 26.5 % more than one step off at 12x16
(plain launches the same bytes). A reference chain given flips of 0.2 % of the values per
block shows the same amplification (34 % at the end), so this is the random synthetic
network, not the kernel.

swin_bneck6: 239 VGPRs, 0 spills, 6 waves per SIMD, 32 KB LDS (gate table, new build
"bmegahg"; the other recorded builds are untouched). swin_mid: 256 VGPRs, 143 VGPRs and 28
SGPRs spilled, 5 waves: correct on the card but NOT gated. Its two C = 96 phases (enc2 with
the merge, dec2 with the expand) spill about 70 registers each, the C = 128 phases none;
plain C = 96 phases spill 79 too, so it is the C = 96 streamed body in the 256-lane queue,
not the fusion. Two knobs tried without effect (WG_V2_PIPE=0: 131 spilled; no scheduling
fences: 141). Open.

Times, median of 50 launches, three runs (ms; 1706x960 | 2560x1440):
  bneck  plain 6.165 | 12.354   megakernel 4.600 | 9.369   plain f16wm 3.449 | 6.808
  mid    plain 10.037 | 20.439  swin_mid  10.127 | 22.272  plain f16wm 5.407 | 11.075
The megakernel is 25 % faster than the default plain blocks and 33 % slower than the f16wm
blocks; swin_mid is level with the plain launches at 1706x960 and 9 % slower at 2560x1440
(the spills), and 1.9 times f16wm. Grid sweep (swin_bneck6, 1706x960): 42 / 84 / 126 / 252
workgroups 12.0 / 6.2 / 4.5 / 4.5 ms. Header in LDS (bmegahg with
WG_V2_MEGA_HDR_GLOBAL=0) 4.40 ms, 128 lanes 8.0 ms: no variant is added.

Verified: TestBmegaGfx11 and TestMidGfx11 (the existing v2 emulator tests with a gfx11
mode) bit-identical to the plain gfx11 kernels; the gfx12 modes of those classes unchanged
and green; clean.tests.test_gpu_swin_mega (new, card) and test_gpu_swin_stages,
test_gpu_enc1 (known expected failure), test_gpu_probes, test_gpu_q8, test_gpu_stages,
test_kernels_emu_gfx11, test_gfx11_platform, test_gfx11_decision, test_gfx11_gates
(compiler check of all ten builds), test_build_targets, test_family, test_hip_env pass.
Break on purpose, red then green: a wrong ping-pong buffer in swin_bneck6 plus a wrong
expand slot in swin_mid (emulator tests: 14 failures; card tests red), a plain build with
the attention switched off (card bit-identity and both reference tests red), the gate
entry removed, the persistent build not built, a gfx12 build accepted as a gfx11 variant.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…t-build blocks on gfx11, and a scan that checks it

The scan of the gfx11 code objects found every v_wmma and v_permlanex16 of
swin_block_c64[_add][_nc] in swin__bmegahg.co under one lane-divergent mask
(s_and_saveexec on vcc): the window loop's `if (win0 + i >= nwin) break;`.
win0 is the wave's first window, a vector value to the compiler, so it could not
see that the test is the same for the whole wave. v_wmma and v_permlanex16 need
every lane executing. At run time the mask is all or nothing (win0 comes from
the wave index), so the kernels passed the card cases; they carried the pattern.

Fix for the gfx11 family only (WG_TILE_STORES_UNGUARDED, the switch of the E2
fix): win0 goes through wg_uniform (readfirstlane), the break becomes a scalar
branch, and the whole loop runs with EXEC full. Other targets keep their code.
Spills of the four kernels fall 48/46/108/90 -> 41/40/11/50 (still above zero:
these kernels are not in the gate table, the runtime uses swin_bneck6).

clean/runner/exec_scan.py: disassembles a code object and runs a forward
data-flow over the control-flow graph of each function: EXEC is partial after
s_and_saveexec, s_and/s_and_not1/s_xor into exec and v_cmpx, restored by
s_or/s_mov exec from a register saved from EXEC (and after a divergent loop),
joins are "partial if any path is". It lists every v_wmma / v_permlanex16 that
may run under a partial mask. test_exec_scan: 15 hand-written functions (each
rule, with a look-alike that must not be flagged) and, with a gfx11 build, a scan
of every kernel the runner launches (0 hits in the fixed build, 4 kernels red in
the build before this commit).

On the card the old code cannot give a wrong result (the mask is uniform), so
TestResidentBlockGpu (partly filled workgroups, 1 to 17 windows, both block
kinds, clamping and clamp-free) is green before and after; it goes red with the
window count off by one (checked). The scan test is the one that is red before.

Not changed, reported by the scan: swin_mid and the fused kernels (_mg, _ex,
fused_merge) still carry the pattern (swin_mid is parked for its owner);
post.co (v_permlanex16 in the reconstruction kernels) and probe_xchg16_div
(deliberate) are flagged too.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…under a lane mask (E4)

kernels/encoder.hip builds for gfx1100 with the gfx11 wrappers unchanged (the
packers and the weight blob are the ones of gfx12: only d_row differs). On a
partly valid 16-token tile the first card build was wrong: the features store
is `if (tok < ntok)`, the compiler moved the whole projection (the half-wave
exchange of the pixel codes, v_permlanex16, and the v_wmma chain) into that
branch, and both need every lane executing. Cases 3 and 13 of the encoder
cases (22x18 padded to 26x20, 4x6: 130 and 12 tokens) gave 8 % and 23 % wrong
features.

Fix, gfx11 family only (WG_ENC_STORE_UNGUARDED, the E2 rule): the features are
stored from every lane; a lane past the last token repeats the work of the last
token (its loads are already clamped to it) and stores the same bytes to the
same place. Other targets keep their code; gfx1201 objects are unchanged.

Resources (gfx1100, clang 20.0.0.rocm): input_encoder 142 VGPRs, 0 spills, 10
waves per SIMD (gate recorded: new "enc0" entry in gfx11_gates.json; nothing
else re-recorded). exec_scan.py: 0 of 113 matrix / exchange instructions under
a lane mask (122 of 126 before the fix); the scan learns the if / else shape
`s_and_not1_saveexec` that the encoder's code contains.

Card (RX 7900 XT), the 14 cases of test_encoder_emu against the float64
reference: 0.000 to 0.003 % more than one step off, mean error at most 0.0015 %
of RMS, 0.000 to 0.095 % of the features different per case (bound 5 %), 0.0575 %
over all cases (467 840 values; the 1 % overall bound of the provisional
acceptance); against the gfx11 emulator 0 to 946 per million one-code flips, the
motion surfaces equal. The emulator test's own limits (0.01 % overall) are an
expected failure on the card. Time, median of 50 launches, three runs:
1706x960 0.637 0.616 0.633 ms, 2560x1440 1.281 1.275 1.289 ms.

Tests: test_gpu_e4.TestEncoderGpu (card; red with the encoder object built before
this fix: cases 3 and 13 and the partly-valid-tile test; red with the store
moved by 16 bytes: every case fails), test_e4_gfx11_emu (the same cases in the gfx11
emulator; red with the same mutation), test_exec_scan (encoder.co joins the
kernels the card runs: red with the old object), test_gfx11_gates.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
kernels/decoder.hip needs no source change for gfx1100: its projection stays in
registers, the pixel stores are scalar-condition stores with no matrix work
behind them, and the scan finds 0 of its 68 matrix / exchange instructions under
a lane mask. The change is the build list (decoder.hip joins the default gfx11
sources), the gate and the tests.

Resources (gfx1100, clang 20.0.0.rocm): coef_decoder 103 VGPRs, 0 spills, 12
waves per SIMD, LDS 10 240 bytes (new "dec0" entry in gfx11_gates.json; nothing
else re-recorded).

Card (RX 7900 XT), the 5 cases of test_decoder_emu, the reset and the border
case, against the float64 reference: A, B and latent in 8-bit codes, C in FP16
steps; at most 0.139 % of a case's values more than one step off, mean error at
most 0.0052 % of RMS, at most 0.139 % different per case and output (bound
5 %), 0.0354 % over all cases (78 988 values; the 1 % overall bound of the
provisional acceptance). Against the gfx11 emulator 0 to 1 389 per million
flips. The emulator test's own limits (99.9 % identical) are an expected failure
on the card. Time, median of 50 launches, three runs: 1706x960 0.431 0.411
0.413 ms, 2560x1440 0.728 0.729 0.728 ms.

Tests: test_gpu_e4.TestDecoderGpu (card), test_e4_gfx11_emu.TestDecoderEmuGfx11
(gfx11 emulator, the emulator test's own limits), test_exec_scan (decoder.co
joins the kernels the card runs), test_gfx11_gates. Red with the pixel-pair store
condition `x0 + 1 < rw` changed to `x0 + 2 < rw`: the card cases and the
emulator cases fail (5 cases, reset and border each); green restored.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
kernels/dec1_dec0.hip builds for gfx1100 unchanged and needs no source change:
its epilogue (dec0's half-wave exchange and projection) runs after the swin
block's work, in every lane, and the scan finds 0 of its 480 matrix / exchange
instructions under a lane mask. The change is the build list (dec1_dec0.hip
joins the default gfx11 sources, with -DWG_SLOW_Q8 like swin.hip, whose block it
includes: 178 VGPRs against 214 with the faster conversion), the gate and the
tests.

Resources (gfx1100, clang 20.0.0.rocm): swin_block_c64_add_dec0 and _nc 178
VGPRs, 0 spills, 8 waves per SIMD, LDS 32 768 bytes (new "dec1dec0" entry in
gfx11_gates.json; nothing else re-recorded).

Card (RX 7900 XT): the fused kernel gives exactly the bytes of dec1 then dec0
(the kernel's contract; the 4 cases of test_dec1_dec0_emu, both clamping
variants). Against the float64 reference, dec0's reference fed the card's own
dec1 output: at most 0.104 % of a case's values more than one step off, mean
error at most 0.003 % of RMS, at most 0.243 % different per case and output
(bound 5 %), 0.0266 % over all cases (71 461 values; the 1 % overall bound of the
provisional acceptance); dec1 itself 0.065 to 0.169 % more than one step off.
Time, median of 50 launches, three runs: 1706x960 fused 3.412 3.446 3.434 ms,
two launches 3.593 3.433 3.446 ms; 2560x1440 fused 7.406 7.404 7.439 ms, two
launches 7.346 7.644 7.372 ms. The fusion is no faster than the two launches on
this card (dec1's block dominates), so no speed variant was added.

Tests: test_gpu_e4.TestDec1Dec0Gpu (card), test_e4_gfx11_emu.TestDec1Dec0EmuGfx11
(fused equals separate in the gfx11 emulator), test_exec_scan (dec1_dec0.co
joins the kernels the card runs), test_gfx11_gates, test_build_targets (red
with the define dropped). Red with dec0's pixel-pair store condition changed:
the three reference cases of the card test fail; green restored.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…sts no longer skip)

The tests of exec_scan ran with ROCM_PATH set to the link folder made for
build.py (it has the compiler, not llvm/bin/llvm-objdump). The tool lookup took
ROCM_PATH only and gave up, so TestExecScanBuild skipped every time in those
runs, the acceptance runs of E3 and E4 included: the "no matrix instruction
under a lane mask" test of the kernels the card runs was never evaluated there.
find_objdump() tries WG_ROCM, ROCM_PATH, /usr/lib64/rocm and /opt/rocm in that
order, then PATH.

Checked by hand on the E4 head build (gfx1100, all kernels the card runs): the
test now runs and passes, 0 findings in them.

Tests: TestFindObjdump (first candidate with the tool wins; a folder without it
is passed over). Red with the existence check dropped (the second case gets the
missing path); green restored.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… post kernels on the gfx11 card

The scan flags all 117 v_permlanex16 of post.co: 39 kernels (post_recon2 ..
post_recon6, post_fused, post_fused3) with three each, the `nb[c] = wg_xor16(hsum[c])`
of the reduced-history step, which the compiler put inside the store's branch
`if (active && has1 && wl < 16 && x + 1 < dw)`. There is no v_wmma in post.co.

Verdict: harmless, not a defect. In that branch only lanes 0 to 15 of a half-wave
pair run, so the lanes the exchange reads (16 to 31) are off, but v_permlanex16
reads the register of a lane whose EXEC bit is off all the same (wg_platform.h
says so; selftest probe_xchg16_div measures it on the card, mode 0 is this exact
shape), and hsum was summed by every lane before the branch. The partner's value
is the partner's real one. No source change, no change of any object: the
gfx1100 post.co of this head and of the E4 head are the same bytes.

The scan cannot prove this (it would need the reaching definitions of the source
register), so it is pinned as a known, explained exception: test_exec_scan
POST_EXCHANGE_KERNELS, exactly these 39 kernels with exactly three
v_permlanex16 and no v_wmma each; a new site, a missing one, or a matrix
instruction in a post kernel fails the test.

Card (RX 7900 XT), new test_gpu_post: the emulator test classes of the post
kernels (recon2 family, recon3 / 5 / 6, recon4, banded, output mode, fused,
fused3) run on the card with their own cases (odd D, partial 32-column tiles, one
D row or column, 3x ratio, NaN / inf inputs, reset): bit-identical to post_recon,
which has no exchange, and the float64 limits where the emulator test has them.
21 tests, 0 failures; 38 of the 39 kernels are launched (a test checks the
list). post_recon2_r_out_diet is not: the runtime's option post_diet=1 is its
only caller and the stage runner has no launch plan for it; its body is
post_recon2_r_out's.

Not used by the Model M path on this card, for the record: the C++ runtime has
no gfx11 build; its defaults would be post_recon2_r_band[_p12] with
post_filter_c1_s4_band (no exchange), post_recon as the fallback.

Red with the exchange removed (wg_xor16 returns its argument on gfx11, post.co
rebuilt): 191 failures in test_gpu_post (test_identical_to_post_recon,
test_identical_to_two_kernel_path, test_exact_variants, test_recon3,
test_recon6); green with the source restored. Red with one kernel dropped from
POST_EXCHANGE_KERNELS, and with a card class left out (the launched-kernels
check); green restored.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…; the _mg / _ex blocks on the gfx11 card

The scan flags swin_block_c64_add_ex, swin_block_c96_add_ex (and their _nc forms)
and the two fused_merge functions the _mg blocks call, in swin.co, swin__mgx.co
and swin__f16wm.co: 216 / 300 / 32 (240 in f16wm for C = 64) matrix and exchange
instructions per function, all under ONE mask. Its origin is the entry test of
the loop over the output tiles, `for (mt = wave; mt < tiles; mt += 8)`: `wave` =
tid >> 5 is a vector value to the compiler, so it tests `tid < 32 * tiles` per
lane (v_cmpx with 0x1ff, 0x2ff, 0xc0, 0x100). The bound is a multiple of 32, a
wave32 holds 32 consecutive thread indices from a multiple of 32, so every lane of
a wave is on one side of it: the mask is full or empty at run time, and an empty
one skips the loop. The stores behind `if (tok[n] >= 0)` are after the matrix
chain, not around it. Verdict: harmless as launched (256 threads), not live.

No source change, on purpose. Making `wave` scalar (wg_uniform, as the persistent
blocks got) does clear the scan, but it adds spilled registers to kernels that
already spill: swin.co _ex_nc c64 / c96 37 -> 47, f16wm 4 -> 29, mgx 37 -> 47
(fused_merge shared with swin_mid kept its code with a template argument; checked
by building it). That is a speed cost for a mask that never happens, so the
pattern is pinned instead: test_exec_scan FUSED_GUARDED lists these functions with
their exact counts, one origin each, a v_cmpx with an immediate that is a multiple
of 32. swin_mid (swin__bmegahg.co, parked) is outside the pin and keeps its result.

Card (RX 7900 XT), new test_gpu_swin_fused (the first card runs of these kernels,
variants base, f16wm, mgx): 23 cases of enc1, enc2, dec1, dec2 with window offset
0 or 2, grids from 4x4 to 36x28 tokens (windows that stick out of the grid, a
workgroup of 8 windows only partly filled), fused kernels against the separate
ones: every output byte equal, no NaN left from the 0x7F fill; against the
float64 reference with the spec 09 limits of the stage tests (5 % different per
case where an output has 2000 values or more). 3 tests per variant, all pass.

Red with fused_merge's store changed to `tok[n] > 0` (swin.co rebuilt):
test_fused_kernels_give_the_bytes_of_the_separate_ones fails in 12 cases and the
reference test in 12; green with the source restored. FUSED_GUARDED red with a
count changed (216 -> 217) and red on the build that has the wg_uniform fix
("not flagged any more"); green restored. Not used by the Model M path on this card:
the C++ runtime has no gfx11 build (it would run the _ex kernels for dec1 and
dec2, WG_SWIN_FUSE_STAGES default); the stage runner's `fuse` geometry runs the
_mg ones.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ime accepts gfx1100

clean/runner/pipeline.py needs libwg_runtime.so, and the repository has none for
a gfx11 target, so this test skips (with the runtime's own message) at this head:
  - build.py's gfx11 branch returns before the runtime step (its docs pass
    --no-runtime); build.runtime_command compiles the unchanged sources fine;
  - device.cpp:398 accepts gfx1200 / gfx1201 only ("no matching HIP device with
    packaged gfx1200/gfx1201 kernels"), and load_module reads <arch>/<name> or
    <arch>.<name> from the build folder.
Not changed here: that is the runtime's device policy, the maintainer's call.

Run in a scratch copy with exactly those three changes (nothing committed; synthetic
weights and frames only), RX 7900 XT, HIP, default kernels (swin.co, swin_bneck6 for
b0..b5, post_recon2_r_band + post_filter_c1_s4_band, separate dec1 / dec0):
 - whole frame, median of 50, three runs: 64x40 -> 96x60 2.993 3.048 2.999 ms;
   45x29 -> 67x43 3.008 2.999 2.995 ms; 1706x960 -> 2560x1440 17.761 18.018
   17.891 ms. Output finite in all cases and frames. Per-stage times are not
   exposed by Pipeline without the interop io (wg_rt_frame_io_timed); the stage
   timings are those of the stage tests.
 - stage by stage against the gfx11 lane emulator on the card's own stage inputs
   (64x40 / 45x29 render): one-code flips 0.000 to 2.368 % per stage (enc1.merged
   the highest), more than one step at most 0.651 %, b0..b5 chain 0.000 %.
   Under the 5 % per case and 2 % bounds.
 - output pixels against the exact (gfx12-arithmetic) emulator frame: 99.97 %
   different (64x40, first frame); per-stage compounding on the random synthetic
   network: enc2.merged 3.2 % of RMS, b5 8.0 %, output 28.3 %. Both the 5 % per
   case and the 1 % overall bound are missed: recorded as expectedFailure.

Tests: test_gpu_pipeline (4 tests: 3 pass, 1 expected failure on the scratch
runtime; 4 skipped at this head). Red with the per-stage flip bound lowered to
0.001 % (9 failures); green restored.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ames of family.py

device.cpp accepted gfx1200 and gfx1201 by two string comparisons, so a gfx11 card was
refused at create. The gate now asks wgrt::target_family (clean/runtime/target_family.h),
the exact-name map of clean/runner/family.py in C++: gfx1200 / gfx1201 -> gfx12,
gfx1100 / gfx1101 -> gfx11, every other name refused (gfx1102, gfx1103, gfx115x, and
gfx1170-1172, which are gfx11 names with the gfx12 matrix design, stay out). The
refusal text names the four targets. Device gets arch(), the exact target whose code
objects the device loads (empty for the emulator), for the runtime to derive the
accumulator row table in the next change.

Code objects are still looked up per exact target as <arch>.<name> or <arch>/<name>
(never the unqualified file), so a build without gfx1100 objects fails closed with
"missing gfx1100 code object". gfx12 devices take the same path as before. The D3D12
gates (rdna4.h, device_d3d12.cpp) are not touched. Whether players ever get a gfx11
build, and how that is gated, is the maintainer's decision; there is no option or
environment variable for it here.

Tests:
 - test_device_targets.cpp (Windows mock; run on Linux through a scratch windows.h
   shim and the mock built as libamdhip64.so, 27 checks): gfx1100 and gfx1101 now
   load their own gfx<arch>.<name> and report arch(); gfx1102, gfx1103, gfx1151,
   gfx1170, gfx1299, gfx11 and "unknown" are refused. The old expectation (gfx1100
   refused) is flipped because this change is the policy change.
 - test_target_family.py (new, Linux and Windows): the C++ map equals family.py for
   every name, plus 16 refused names (decorated, partial, case, other families).
 Red on purpose, then restored: a gfx11 prefix match turned test_device_targets red
 (exit 7: gfx1102 accepted) and test_target_family red; dropping gfx1100 from the map
 turned both red (exit 3 and a map difference).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ce's target family

The matrix instruction of gfx11 returns accumulator row 2 v + h in element v of lane
half h, that of gfx12 row v + 8 h, and the packed weights follow the table (A3:
packing.py D_ROW_LAYOUTS["gfx11"]). clean/runner/pipeline.py sets WgRtConfig.d_row from
the build's target, but clean/runtime/interop never sets it, so a game's runtime always
packed in gfx12 order and the gfx11 kernels would have read the wrong rows.

The runtime now makes its device before it packs, and when the config's d_row is all
zero it takes drow_gfx11() for a device whose arch() maps to gfx11 (target_family.h),
else the gfx12 table as before. A d_row given in the config still wins. No device-less
case changes: the emulator and D3D12 devices have no arch(), so they keep the gfx12
table, and on gfx12 the packed bytes are the ones of before. Creating the device is
the only step that moved up; a device that cannot be made is still reported at the old
place, after the model has loaded, with the same text.

Tests: model_pack_dump takes an optional gfx11 argument, and test_model_pack_cpp packs
all twelve stage blobs of two synthetic models with drow_gfx11 against packing.py's
gfx11 table, byte for byte (the gfx11 and gfx12 blobs differ, so a wrong table cannot
pass either test). Red on purpose, then restored: drow_gfx11 returning the gfx12 table
fails it (seed 1, enc0: 4428 bytes differ). That the runtime picks the table from the
device is checked on the card by the whole-frame test in a later change.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ch>.<name> next to the code objects

build.py's gfx11 branch returned before the runtime step and refused --runtime-only, so
no libwg_runtime.so existed for a gfx11 build. The runtime reads its code objects as
<arch>.<name> (device.cpp, never the unqualified file), and a gfx12 build makes those
copies as it embeds the files; a gfx11 build had neither.

Unless --no-runtime or --sources is given (as before, neither builds a runtime), a gfx11
build now copies every code object to <arch>.<name> (link_arch_objects) and compiles the
runtime with embed=False: no embedded table, since the gfx11 set is not the gfx12 list
that write_embedded requires, and the runtime reads the files from the folder.
--runtime-only does the same step alone for a gfx11 target; --motion-outres-only stays
gfx12 only (new message: the gfx11 target has no motion code objects). gfx12 builds are
untouched: runtime-only still embeds, and the guard's --no-runtime build is the same.
The runtime's own sources are not changed here. clean/README.md follows.

Not done: nothing embeds a gfx11 code-object set (which files a release would ship for
gfx11, and whether it ships at all, is the maintainer's decision), and the Windows
runtime_command (D3D12 interop sources) is not exercised for gfx11.

Tests (test_build_targets, compiler replaced by a function that writes each output):
both gfx11 names make the runtime with no -DWG_EMBEDDED_TABLE and no embedded call, one
<arch>.<name> copy per code object with the same bytes and no other gfx* file;
--no-runtime and --sources make neither; --runtime-only links the existing objects and
does not embed for gfx11 but still embeds for gfx1201. Red on purpose, then restored:
the runtime step switched off (2 failures), the copy removed (3 errors), gfx11
--runtime-only embedding (1), gfx12 --runtime-only no longer embedding (1).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… built for gfx1100

With the runtime's gate, the packing rows and the build of the previous changes,
test_gpu_pipeline no longer skips on a gfx11 build that has its runtime: its 5 tests run
on the RX 7900 XT through HIP, default kernels (swin.co, swin_bneck6 for b0..b5,
post_recon2_r_band + post_filter_c1_s4_band, separate dec1 / dec0), synthetic weights and
frames only. The docstring no longer says the repository lacks the runtime.

New: test_the_runtime_picks_the_rows_of_the_cards_family. A game's interop never sets
WgRtConfig.d_row, so the runtime must choose the gfx11 table itself. Pipeline takes
d_row="runtime" (the config's table stays 0, the reference table of check_packing is the
build's family), and the test checks that every stage blob equals packing.py's on the
gfx11 table, that enc1's blob is not the gfx12 one, that the stage blobs equal those of a
pipeline given the table, and that a frame of two new pipelines is the same bits (new
ones: the cached pipelines have run frames and the frame counter is part of the result).
Red on purpose, then restored: a runtime that ignores the device's family fails it with
"runtime packing of enc0 differs from pack_weights".

Result on the card (OK, expected failures=1): output finite in all cases and frames;
stage by stage against the gfx11 lane emulator on the card's own inputs, one-code flips
0.000 to 2.368 % per stage (enc1.merged 45x29 the highest), more than one step at most
0.651 %, b0..b5 chain 0.000 %; the output pixels against the exact (gfx12-arithmetic)
emulator frame are 99.97 % different at 64x40 (the expected failure, unchanged, synthetic
network). Whole frame, median of 50, three runs: 64x40 -> 96x60 3.120 3.066 3.044 ms;
45x29 -> 67x43 3.073 3.182 3.169 ms; 1706x960 -> 2560x1440 18.664 18.822 19.757 ms (the
card is shared with another project's jobs; the scratch run of the previous change gave
17.8 to 18.0 ms).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…11 (S1)

wg_platform.hlsli takes WG_GFX (12 by default, unchanged; 11 for RDNA 3). Under 11:
no FP8 matrix op, no FP8 convert and no _ags pack (wg_ags.hlsli hides the FP8 codes);
wg_wmma_f16_k8 is one raw F16 instruction on fragments replicated across the half-waves,
wg_wmma_f16_k16 chains two of them (a K=16 step), wg_wmma_f16 keeps the gfx12-shaped
operands of HIP's helper on top of them, wg_d_row is the 2i+h row table, SW_UNPACK_AGS is 0,
and the clamp form of the GELU LUT converts with f32tof16 (RADV rounds a (half) cast to
nearest even). build_dx12.py --gfx 11 defines WG_GFX=11, writes <out>/dx12-gfx11/ with a
"gfx": 11 manifest key, builds only declarations with gfx=11 (default gfx=12) and skips RGA.

Tests: test_build_gfx.py (8, no tools needed) and test_gfx11_helpers.py: headless on the
card under Wine (vkd3d-proton) against numpy and a HIP gfx1100 kernel, bit for bit for the
raw op, the K=16 chain and wg_wmma_f16; convert is RNE; xor16; row table; rounding modes.
They skip without the tools named in the module doc. Red on purpose then restored: row
table i+8h, K=16 chain without its second op, swapped K halves, xor16 skipping a lane,
swapped convert dwords, FP8 code visible on gfx11, build without the WG_GFX define.

Guard: the 46 gfx12 DXIL files are identical to the baseline (manifest too).
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…sts, PSO sweep (S3)

wine_headless_run.sh runs a Windows program under Wine with a null-graphics prefix, a
private runtime folder and the GPU lock, and refuses anything that could open a window
(a prefix with a graphics driver, /tmp or /run/user scratch). build_kernel_replay.py
cross-builds kernel_replay.exe, pso_sweep.exe and gfx11_probe_host.exe with llvm-mingw.
pso_sweep.exe creates each kernel's pipeline with the runtime's buffer or direct root
signature (the runtime itself is not used: its device refuses a card that is not RDNA 4);
pso_sweep.py runs it over a built folder, records a driver crash per kernel and classifies
each DXIL by FP8 channel codes (dxc -dumpbin): a PSO with an FP8 op must fail on RDNA 3,
any other must be created.

Result on the RX 7900 XT (vkd3d-proton, extended emulation): the 46 gfx12 DXIL files give
23 created (all without FP8), 17 failed with 0x80070057 and 5 crashed the driver (the swin
c96 add_* blocks: a page fault inside the driver, reproducible alone), 1 tex kernel not
tried; none unexpected. The 8 gfx11 helper kernels of test_gfx11_helpers all create.
kernel_replay.exe builds and starts but its device refuses the card until the family gate
opens.

test_wine_headless_run.py: 4 tests; red on purpose: scratch check removed, null-graphics
check removed.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…out in the CPU model (S2)

native_driver_probe_f16 (gfx=11 only) runs D = A x B + C with FP16 operands as a K=16 step
of two chained instructions on the replicated fragments of the gfx11 layout. The gfx12
kernel native_driver_probe is untouched (its DXIL is byte for byte the same, and it is not
built for gfx11: it has an FP8 instruction). wmma_probe.h gets namespace gfx11: buffer
layout, fill, expected results (the same matrices, rows 2v+h) and a compare with a tolerance
of 1/4096, because the card's F16 accumulation differs from the exact sum in the last bit in
57 of 256 results (measured; equal to HIP's instruction).

test_wmma_probe.cpp: 20 new checks (the right layout passes, six wrong hardware variants
fail, replicated-half variants pass, all lane-pair swaps of A, B and C fail exactly when
they change what the instruction reads, all element-pair swaps fail, the tolerance accepts
last-bit differences and reports 2e-3 and NaN); the gfx12 checks are unchanged. Red on
purpose: fragment 1 given the K of fragment 0 (6 failures), d_entry rows v+8h (1).
test_gfx11_probe.py (3 tests): builds the kernel with build_dx12.py --gfx 11 and runs it on
the card under Wine with wmma_probe_dump's operands: results within the tolerance; the
gfx12 buffer fails. Red on purpose: the kernel with one instruction instead of two.

Guard: the 46 gfx12 DXIL files are identical to the baseline.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…y normal build (S4)

A compile-time macro, default off and never defined by clean/runner/build.py, lets the
native D3D12 device, the executor choice and the Wine log line take an RX 7900 series
board (PCI 1002:744C, Navi 31, gfx1100; the id read on the development card, shared by
the XT, XTX and GRE; no other RDNA 3 id, 7448 included). The mechanism is one of the
options listed for the maintainer in the RDNA 3 plan (a macro and a local build); the
maintainer may prefer another (environment variable, ini key, a PCI id rule in the
family function, or the gate left closed in main), and this change does not choose
among them. It exists so that the D3D12 kernels for gfx11 can be built and tested
before that decision.

Without the macro nothing a player runs changes: rdna4_identity is untouched (the Ray
Reconstruction runtime still asks only it), dev_gfx11_identity is constant false,
dx12_folder answers "dx12", the device reports no target, the probe is the gfx12 one,
and the order of the manifest read and the device is as before. With the macro on:
 - rdna4.h: kDevGfx11, gfx11_device_id, dev_gfx11_identity (same rule as rdna4_identity:
   the driver's ids, else the Wine fallback's), dx12_folder(arch), dev_gfx11_arch();
   the Wine line and describe_identity know a taken board.
 - executor_select.h: the default rule takes the board (D3D12 executor).
 - device_d3d12.cpp: both identity gates take it; the device reports target gfx1100,
   reads <build>/dx12-gfx11/manifest.json after the device (the family comes from the
   adapter) and runs native_driver_probe_f16 with wmma_probe::gfx11 (F16 only, compared
   within 1/4096).
 - runtime.cpp: the device is made before any manifest is read, the manifest of a build
   is read through dx12_manifest(), the accumulator row table comes from the device's
   target as for HIP (already so), and the native-direct check accepts the gfx11 table.
 The tolerance: the card's F16 accumulation is not exact (57 of 256 probe results differ
 from the exact FP32 sum, by at most 2^-15 here, equal to HIP's instruction). 1/4096 is
 two ulps of the largest expected value (1331) and 1/2500 of the smallest gap between
 distinct expected values (0.607), so a wrong layout cannot pass.

Tests: test_dev_gfx11.cpp (28 checks without the macro, 29 with; both builds pass) and
tests/test_dev_gfx11.py (builds both, 6 source pins incl. that no build or release file
mentions the macro). The existing test_executor_select.cpp: 234 pass without the macro;
with it exactly the 12 cases that assert the RX 7900 XTX id (744C) is refused fail, as
they must. On the card, headless under Wine with the system DXGI of DXVK in a scratch
prefix (the Wine fallback ids, as a player's Proton gives): native_probe_test (new target
of build_kernel_replay.py; the borrowed-device path) is refused without the macro and
passes the F16 probe with it; WG_TEST_DRIVER_PROBE_FAIL=1 fails at float 7; kernel_replay
runs native_driver_probe_f16 as a chain, 256 results within tolerance. Red on purpose,
then restored: 11 host mutations and 6 Wine mutations (see the test files' pins).
Guard: the 46 gfx12 DXIL files and the 42 HIP objects are unchanged (no kernel touched).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…fx12 kernels on RDNA 3, with a rounding check

Inside the WG_DEV_GFX11 opening a second developer-only choice lets a gfx1100 board run
the UNCHANGED gfx12 kernel set (dx12) through dxil-spirv's emulation of the FP8 matrix
operations in the raw-fragment path (DXIL_SPIRV_CONFIG=wmma_rdna3_raw_fp8 in a patched
vkd3d-proton). The mechanism is the environment variable WG_DEV_GFX11_SHORTCUT=1, read
only in a build made with the macro (rdna4.h: without it dev_gfx11_shortcut() is constant
false and the library never asks the environment). It is a placeholder: the maintainer
decides the mechanism later (ini key, option, a PCI id rule, or no shortcut at all), and
shipping a patched dxil-spirv is a product decision this change does not make.

Without the macro nothing changes for a player. With the macro and the variable unset
nothing changes either (the S4 behaviour). With both, the device reports no target (the
gfx12 folder dx12 and the gfx12 accumulator table, as on RDNA 4; the emulation keeps the
gfx12 accumulator layout), and:
 - the borrowed-device probe is the gfx12 one (native_driver_probe, FP16 and emulated
   FP8), compared within wmma_probe::gfx11::kTolerance: the emulation multiplies in the
   FP16 instruction, whose accumulation is not exact on gfx1100 (same reasoning as S4).
 - a rounding check runs at device creation, for owned and borrowed devices: the
   kernels round to FP16 through an F32 -> F16 -> F32 pair of the matrix conversion, and
   the driver's compiler folds such a pair away unless the conversions are exact (found
   with the shortcut probe: a fold gives encoder features one FP8 step off). The kernel
   native_round_probe (gfx=11 only, so the gfx12 set keeps its 46 files) converts 256
   inputs that all have bits below FP16's precision, including exact ties of both
   parities; the device compares bit for bit with the nearest-even result computed on the
   CPU (wmma_probe.h, round16) and refuses with a log line otherwise. The kernel is read
   from <build>/dx12-gfx11/manifest.json (read_manifest takes an optional kernel name).

Tests: test_shortcut_probe.cpp (27 checks, built with and without the macro) and
tests/test_dev_gfx11_shortcut.py (both builds plus 5 source pins). On the card, headless
under Wine: the shortcut passes the probe and the rounding check; with
DXIL_SPIRV_CONFIG unset the compiler folds the pair and the device is refused ("the
driver does not round F32 -> F16 -> F32"); WG_TEST_ROUNDING_FAIL=1 gives the same
refusal. Red on purpose, then restored: 8 mutations (see the test files' pins).
Guard: the 46 gfx12 DXIL files are unchanged.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ess Wine, compared with the HIP whole frame (S12)

frame_host.cpp is a small Windows host (cross-built with the other replay hosts by
build_kernel_replay.py) that loads wronggpu_runtime.dll, creates the native D3D12 backend
(backend 3) on a synthetic raw-tensor model, runs frames from input files, dumps the
named buffers of each frame and times frames with the runtime's own timestamps. Nothing
reads a model.wgcm. It exists because the whole-frame machinery (compare.py,
time_frames.py) drives the runtime library from Python on Windows, and a headless run
under Wine on Linux has no Python on the Windows side.

frame_parity.py prepares the model, the inputs and the host's spec (the same synthetic
frames as compare.py), runs the HIP or lane-emulator whole frame natively through
pipeline.py, and compares every named buffer and the output image of two runs: bytes
that differ per buffer (buffers that a fusion leaves unwritten, stale ones, and the
decoder's other format are listed and not counted, as in compare.py), the output in
FP16 steps and relative RMS, and the GPU times. It measures; it is not a gate, since the
output standard of an RDNA 3 route is the maintainer's decision.

Checked: a run against itself gives 0 differing bytes and byte-equal output; the
D3D12 shortcut frames at 64x40, 853x481 and 1706x960 run end to end (see the S12 report).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… text, and a guard checks it

The RDNA 3 changes in swin.hip, encoder.hip and wg_platform.h leave the gfx12 program the same, but not its text: the
gfx11 operand types, fragment loads, store forms and wave id helper were written for every target, and the patch
merge and expand stores gained braces. A different compiler can schedule different text differently: on the RDNA 3
line, ROCm 10.1 (clang 24) moved instructions in swin.co and swin__bmegahg.co for one restructured store, although the
program was unchanged. "The same instructions under today's compiler" is not a promise RDNA 4 can rely on; the text is.

Every gfx11 form now sits under WG_SWIN_G11 (gfx11 or the emulator) with main's earlier text in the #else. In
wg_platform.h the v4h / v16h types and wg_xchg16_u exist only for gfx11 and the emulator; encoder.hip gets the same
treatment for its gfx11 store. clean/tests/check_gfx12_source_unchanged.py preprocesses every shipped HIP compile for
gfx1201 (each tree's own build.py with its compiler calls recorded, FG's three objects, every other clean/**/*.hip)
and requires the text to match the base tree's token for token; its unit tests plant an edit to prove it can fail.
dx12/tests/gfx11_hip_ref.hip, a host program of the gfx11 helper tests that no gfx12 build compiles, is named as
unbuildable in the test and as a new form on the command line.

Verified (ROCm 10.1, clang 24; only HIP kernel sources and the guard changed, no D3D12 or runtime file):
- guard against the base on main (91436c4): 160 forms, 0 failed. The same guard with the unfixed branch head
  (0cd1a91) as the tree: 160 of 160 failed.
- gfx1201 objects (M, K, L, FG: 42) of the fixed tree and of 91436c4: byte-identical 42 of 42. (The unfixed head
  0cd1a91 also gave 42 of 42 identical objects under 10.1: the source differed, the code objects did not.)
- gfx1100 objects (--variants f16wm,mgx: 11) before and after: byte-identical 11 of 11.
- guard unit tests 14 OK; the emulator tests test_kernels_emu_gfx11, test_e4_gfx11_emu and test_kernels_emu: 36 OK.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
….hlsli and wg_ags.hlsli are main's again

On main aa85515, model K's softmax self-check records kSourceDigest, a SHA-256 over the
shader headers the softmax kernels include (wg_platform.hlsli and wg_ags.hlsli among them).
S1's gfx11 edits of those two files changed the digest and build_dx12.py refused to build.

Moved: everything gfx11 in wg_platform.hlsli (WG_GFX default and checks, wg_in_upper_half,
wg_xor16_4, wg_wmma_f16_k8 / _k16, the gfx11 wg_wmma_f16, wg_d_row, the gfx12 wg_d_row) and in
wg_ags.hlsli (the WG_GFX default and the #if hiding the FP8 codes) into the new
hlsl/wg_gfx11.hlsli. A gfx11 kernel includes it instead of wg_platform.hlsli (it includes
that one itself); gfx12 builds of it only include the platform layer. Under WG_GFX 11 it
renames the platform layer's gfx12-shaped wg_wmma_f16 while including it, and hides the
FP8 identifiers again with #undef / #define (a use does not compile, as before).
native_game_helpers.hlsl, tests/gfx11_helpers.hlsl and the compile check include it.
wg_gelu_lut.hlsli, swin_common.hlsli and build_dx12.py are not in the digest and stay.

Verified: smx_selfcheck_digest.digest() equals the committed header again; the two gfx11
probe DXIL files are byte-identical to the ones built before the move; test_build_gfx
(8) and TestGfx11Compile (4, including the FP8-is-hidden negative controls) pass; the
card tests skip (no GPU in this slice).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… of compiling it for gfx1201

Since the gfx12 source of every HIP kernel is main's text again, the gfx11 exchange wg_xchg16_u exists only for gfx11
and the emulator. test_exchange_is_one_permlane_on_gfx11_and_bpermute_on_gfx12 still compiled it for gfx1201 and so
failed ("use of undeclared identifier 'wg_xchg16_u'"). Its gfx12 half now checks what the design promises: wg_xor16 on
gfx1201 is the one ds_bpermute exchange of main's text (no permlane), and wg_xchg16_u does not compile there. The gfx11
half is unchanged.

Verified (ROCm 10.1, clang 24): the test is red before (the old test fails on the parent commit) and green after; with
the wg_platform.h of the commit before the gfx12-inert fix (wg_xchg16_u for every target) the new test fails, restored
it passes. Without a ROCm clang the whole module skips, as on CI's Windows runners.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…Wine runner test skips off Linux

build_dx12.py's digest label used os.path.relpath, which raises ValueError when a source (a test's temp
folder on C:) and the repository (D:) are on different drives. A source outside the repository is now
labelled ext/<file name>. A new test in test_build_gfx.py simulates the cross-drive error (red before the
fix, green after). test_wine_headless_run.py runs a bash script, so it is skipped unless on Linux.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…U emulation of the kernel text

wg_gfx11.hlsli gets the operand layer of the native RDNA 3 form (WG_GFX 11 only; the gfx12 branch and every
RDNA 4 kernel are unchanged):
- wg_h16 (16 F16 values per lane: K 0..7 / 8..15 of a row or column, both half-waves alike) and the K = 16
  opcode WG_AGS_F16_F32_K16 (0x7e, outside AMD's opcode list; needs the translator's K = 16 patch, a stock
  translator refuses it): wg_wmma_f16_k16n / wg_mma16, one instruction per K = 16 step, K in index order;
- the row map wg_a_row(p) = 8 (p & 1) + p / 2: an A operand loaded with it leaves the accumulator in the gfx12
  layout, so the RDNA 4 weight blob and every step between products stay as they are;
- wg_ldA16 / wg_ldB16 (A row-mapped, B column, straight from a gfx12-order tile: no exchange), wg_b16 (a B
  operand from the lane's own gfx12-shape fragment: one half-wave exchange, reusable), wg_rows_to_gfx12 (the
  FP16 result of a product with unmapped rows into the gfx12 fragment order: one exchange of 2 dwords);
- WG_A_ROW_MUT: a wrong table for negative controls only.

clean/dx12/emu (test-only, never shipped): hlsl_emu.py compiles a D3D12 kernel file's declarations as C++
against hlsl_emu.h (stands in for wg_platform.hlsli / wg_ags.hlsli; the other headers are the real ones) and
runs them on the lane emulator of the HIP kernels (hlsl_emu_bridge.cpp): gfx12 matrix model for WG_GFX 12, the
measured gfx11 F16 instruction for WG_GFX 11. HlslEmu is a device with the stage runner's interface (arena
offsets as pointers, launch arguments flattened into root constants).

tests/gfx11_k16_probe.hlsl + test_gfx11_native.py: one-hot probes (512 cases: every A row and K position, every
B K position and column) of ldA16 / ldB16, b16, the accumulator start and the v^T form, exact equality on the
emulated instruction; the K = 16 channel equals the chained K = 8 form and HIP's exchanging wrapper there.
Negative controls: WG_A_ROW_MUT is caught on exactly the moved rows; the v^T form without wg_rows_to_gfx12 fails.
The translator checks are in the same file and skip until a patched dxil-spirv is configured.

Verified (CPU): test_gfx11_native 11 tests, 8 run, 3 skipped (translator); test_build_gfx, test_kernel_sets,
test_k_smx_selfcheck_digest, test_gfx11_helpers 38 OK (20 skipped: card); the K softmax digest is unchanged
(smx_selfcheck_digest.digest() equals k_smx_selfcheck.h); the RDNA 4 DXIL guard is in the report.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…11 only

kernels/k_block_g11.hlsl declares the narrow stages of k_block.hlsl (e0, e1, e2, d2, d1 one-window; e0, e1, d1,
d0 two-window incl. the hidden-history and compact-head d0; the three __rodd forms) gfx=11, named <port>__g11,
on hlsl/k_block_g11.hlsli: k_block.hlsli's template with every product as ONE K = 16 instruction on operands the
kernel lays out (wg_gfx11.hlsli S1 layer). Same arguments, launches and weight blob (RDNA 4 / d_row gfx12) as
the ports; every accumulator comes out in the gfx12 layout, so all code between products is the ports':
- A from memory or LDS (weights, K tiles, VT tiles): row-mapped 16-half loads, no exchange;
- B from memory or LDS (embed / expand inputs, Wv, the fixed mix table, the merge's x staging): direct loads;
- B made in registers (n, q, z, a, the hidden tile, x2 for the head, softmax weights): one exchange, reused
  for every output tile (n's B form is also v^T's A operand; KN_NB_HOIST 0 remakes it per head);
- v^T's result goes through wg_rows_to_gfx12 before LDS. Every product sums K in index order, so a stage
  should give the shortcut's bits (GPU test).
Names carry __g11 because main's softmax digest scans every declaration in clean/k/dx12/kernels for the
self-check names; the runtime does not launch these yet (open question in the report).

Verified (CPU): build_dx12.py --gfx 11 --model-k compiles all 13 (DXC 1.10, arithmetic-mode rewrite, dxv);
test_k_native_g11_emu (clean/dx12/emu): 39 stage outputs over 3 size sets (odd maps, partial windows, maps
smaller than a window) and 3 offsets against NumPy within spec 49's stage limits (worst max 2.8e-3 of RMS) and
byte for byte equal to the RDNA 4 ports emulated the same way; d0 full head, compact head and hidden history
equal to the ports' bytes; the __rodd forms equal; knobs KN_NB_HOIST 0, KB_GRPLATE, KB_W2LATE, KB_X1LDS keep
the bytes. Negative controls: WG_A_ROW_MUT puts every stage > 10x the limit; dropping wg_rows_to_gfx12 on
purpose made the stage test fail (e0 0.53 of RMS) and it passed again after restoring. Digest unchanged;
test_build_gfx / test_kernel_sets / test_k_smx_selfcheck_digest / test_k_plan_bridge / test_k_port_check_nan OK.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…same bytes (GPU tool, CPU dry run)

clean/k/dx12/k_native_check.py (developer tool of the native RDNA 3 work): --prepare writes, from the stage
runner's plans (k_plan_bridge.chain_lines), two kernel_replay case files over the same input files: the RDNA 4
ports (run through the shortcut) and the native __g11 kernels, 13 kernels x 6 block-level sizes x 5 even
offsets per seed, special values (infinity, +-65504, -0, subnormals, NaN) in two of every three cases, NumPy
references for the others, one negative control (a native chain with half its grid). --emulate runs both files
through clean/dx12/emu (a CPU dry run of the files and the comparison); --compare checks native = shortcut byte
for byte over the defined bytes (sign-only NaN differences counted apart), the 64-byte slack, spec 49's stage
limits against NumPy, and the negative control. --timing / --time N write the 1440p Quality levels with
kernel_replay's `time` for the timing round.

native_helpers.hlsl: native_fill is declared gfx=11,12. The D3D12 device's memset launches it, so kernel_replay
(and the runtime) in the native gfx11 mode needs it in <build>/dx12-gfx11. The gfx set is not part of the source
digest and the kernel is unchanged: the RDNA 4 DXIL and manifest stay byte-identical (guard in the report).

Verified (CPU): --prepare --emulate --compare with seed 0: 78 cases, 138 outputs, 0 differing, 0 slack
writes, worst 7.2e-5 mean / 3.2e-3 max of RMS, negative control caught; one flipped byte in a native output makes
--compare fail (exit 1) and passes again restored. test_k_native_check (a subset: 12 cases) OK incl. that flip;
test_build_gfx / test_kernel_sets OK; build_dx12.py --gfx 11 --model-k now writes 16 kernels incl. native_fill.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…kipped without a card)

test_gfx11_native.py gets TestNativeOnCard on the card harness of test_gfx11_helpers.py (gfx11_probe_host under
the headless Wine runner; WG_GFX11_VKD3D must hold the K = 16 patched vkd3d-proton): k16n equals the chained
K = 8 form bit for bit on random data with cancellation and on data with infinities, NaNs and signed zeros in
both operands, k16n matches HIP's v_wmma_f32_16x16x16_f16 (WG_GFX11_HIP_RUN), the one-hot probes of nat_ldA /
nat_b16 / nat_vt, and nat_ldA equals the chain's bits on random data (same K order). The emulator and the card
tests share one arena layout (arena_tiles / arena_channel).

Verified (CPU): test_gfx11_native 16 tests OK, 8 skipped without a card (the 3 translator checks run with the
patched and the stock dxil-spirv: one multiply per K = 16 step, no lane exchange, the stock tool refuses the
opcode); test_gfx11_helpers' DXC compile checks (TestGfx11Compile, 4) OK with WG_GFX11_WORK set.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…nd; files are closed

The unit subset used sizes and an order in which dropping the merge store's `my < mh` bound
(k_block_g11.hlsli) left every unit test green; only the full dry run caught it. The subset now
takes the first three sizes of the full list with seed 0, which places them at offsets that
expose a write into the slack. Verified: with the bound removed the test fails ("native wrote
into the slack", 4 cases); restored, it passes. k_native_check.py and its test close every file
(no ResourceWarning under -X dev -W error::ResourceWarning); the full seed-0 dry run still passes
(78 cases, 0 differ, 0 slack).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
A gfx11 board's softmax stage hashes differ from the gfx12 HIP reference, so the
create-time check refuses Model K there (MODEL-K-RDNA3.md section 4). The existing
rodd_unchecked hook takes the round-to-odd forms; this one takes the mix form,
which is the default path, so a whole frame can be timed on it. The hook is
compiled only with WG_DEV_DIAGNOSTICS, outside the self-check digest regions
(the digest and k_smx_selfcheck.h are untouched).

Verified: test_k_smx_selfcheck_digest, test_dev_gfx11*, release runtime
disassembly identical to the base commit.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…_g11) in a shortcut run

Option (a) of K-NATIVE-1.md 5.1. In a WG_DEV_GFX11 build, WG_DEV_GFX11_NATIVE_K=1 on
top of WG_DEV_GFX11_SHORTCUT=1 makes the D3D12 device load the __g11 entries of the
dx12-gfx11 manifest next to the gfx12 set, and launch/prepare run <name>__g11
instead of <name> wherever the manifest has it. Every other kernel (wide stages,
I/O, exposure, warm, helpers: no matrix work or no native source yet) still runs
as its gfx12 kernel through the shortcut, so a frame is mixed. The variable
without the shortcut makes create fail. One build times both modes: the same run
without the variable is the shortcut. Profile and launch-trace rows name the
kernel that ran.

Everything is under WG_DEV_GFX11: the release runtime's disassembly, symbols and
strings are identical to the parent commit's. Tests: the lookup picks __g11 only in
a WG_DEV_GFX11 build and only when the manifest has it (host build with and
without the macro, two negative controls), source pins, and each of the 13 native
declarations has its port's code object, block, wave and arguments.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…e chosen per kernel

GPU timing found native faster on some stages and slower on others. The value "1"
still means every kernel the manifest has a __g11 form of; a comma list of port
names (k_stage2_d0_hp,k_stage2_d1,k_stage_d2,k_stage_e2) launches __g11 only for
those ports (whole names) and runs the rest through the shortcut; empty or "0" is
off. Same gate as before: read only in a WG_DEV_GFX11 build, needs the shortcut.

Tests: the list form switches only listed ports (negative controls: a port not in
the list stays; a mutation that selects every name is caught by the host test),
prefixes do not match, stray commas select nothing. Release runtime: disassembly
and symbol list identical to the parent of this work, no new strings.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…uration

The patched translator now refuses opcode 0x7e without DXIL_SPIRV_CONFIG=wmma_rdna3_raw_fp8, since only
that configuration keeps the kernels' F32 -> F16 -> F32 rounding pairs unfolded. The count test runs with
the configuration only; a new test checks the refusal without it. Verified: green with the fixed
translator; with the refusal removed from the translator the new test fails, restored it passes.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…11_NATIVE_K

Review finding (Kimi, FIX-PASS): the module docstring only mentioned =1. Docstring only; the tests are unchanged and pass.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ult)

KN_NB_HOIST 2, KN_VSTREAM, KN_ZSTREAM, KN_AB 1/2, KN_XPIN, KN_HEAD_ROLL,
KN_MERGE_ROLL / KN_MERGE_STEP in k_block_g11.hlsli: where operands are
loaded or made, never an operand value or a sum's order. With every knob
at its default the 16 gfx11 DXIL files are byte-identical to the
wip/k-native-f3r2 build (scratch-kenc/v/dflt).

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… (off by default)

The template's 16-byte loads and stores go through kn_ld128 / kn_st8 /
kn_ldA16 / kn_ldB16. With KN_A16 1 the address has its low 4 bits
cleared: every address these kernels form is a multiple of 16 bytes from
a 256-byte aligned buffer or weight section, so the bytes are the same and
the driver issues global_load_b128 instead of four global_load_b32
(k_stage2_e0: 368 -> 92 vector loads). Off: the 16 gfx11 DXIL files are
byte-identical to the parent's build.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…nobs

KN_A16 on all 13 native kernels; e2 and e0 two-window KN_ZSTREAM, d2 and
d0 KB_GRPLATE (192 VGPRs, 8 waves), e1 two-window KN_XPIN + KB_GRPLATE +
KB_W2LATE (240 VGPRs, 6 waves, was 256 / 5), d1 two-window KB_GRPLATE +
KN_VSTREAM + KN_NB_HOIST 2 (51 spilled VGPRs, was 106). Stage times on the
RX 7900 XT, median of 6 interleaved rounds vs the parent native kernels:
e0 -6.5 %, e1 -6.0 %, e2 -6.0 %, d2 -22 %, d0 -5.6 %, d1 -5.0 %.

The emulation test and k_native_check.py --emulate build each native
kernel as its own library (the declarations' defines differ now); the knob
test also switches the new knobs. Checked: emulation of every stage form
byte-identical to the RDNA 4 ports; the test goes red with a 32-byte
address mask and green again.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…16-byte aligned

The native RDNA 3 kernels' KN_A16 form clears the low 4 bits of every
16-byte arena address, which is the requested address only when every
KBlkParams pointer is 16-byte aligned (review of K-ENC, fix 1). In builds
with WG_DEV_GFX11 or WG_DEV_DIAGNOSTICS, create checks the parameters
block_params gives every stage from the allocated buffers and weight
sections, and launch checks every k_stage* kernel's parameter block (the
softmax self-check path included); both fail naming the pointer. The
checks sit outside the self-check digest regions (block_params itself is
inside one, so it is unchanged and the digest with it).

Release build: llvm-mingw DLL of this tree vs wip/k-enc-1, 0 differing
disassembly lines of 367,575, identical symbol and string lists, 2
differing bytes (the PE timestamp and its copy); the check's message and
symbol only in the developer DLL.

Test: clean/tests/test_k_blk_align.py builds test_k_blk_align.cpp against
KBlkParams as runtime_k.cpp declares it (every pointer field + 1, 4, 8,
12 refused naming it), pins the wiring; red with the check removed from
the header, with a field dropped, and with the create check removed.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
The rolled merge loop has no tail and its per-iteration arrays hold 4
tiles: 0 never advanced, 3 read past the merge's K tiles, 8 past the
arrays (review of K-ENC, fix 2). No staged declaration rolls the merge:
the 16 gfx11 DXIL files are byte-identical to wip/k-enc-1's (only the
manifest's source_digest provenance changes with the header text).

Test: test_merge_step_is_1_2_or_4 (emulation build) refuses 0, 3 and 8
with the message and builds 1, 2 and 4; red with the guard removed. DXC
(build_dx12.py --define KN_MERGE_STEP=3) also stops with the message.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…hantoslayer21#225's no-name-rule pin

clean/tests/test_executor_select.py (CreateOrder.test_no_name_rule_left, phantoslayer21#225) requires that clean/runtime/rdna4.h
contains no find( outside comments, so that no GPU name rule can come back into the identity code. The parser of the
WG_DEV_GFX11_NATIVE_K list (developer builds only) used std::string::find to split at commas and broke that pin. It now
walks the characters; the entries, the exact-name match and the treatment of stray commas are unchanged.

Verified: test_executor_select and test_dev_gfx11_native_k (the host build of the list cases with and without
WG_DEV_GFX11): 20 tests OK. With the separator changed to ';' on purpose the list cases fail ("a listed port switches",
"the first and the last entry switch", "a list selects only its names"); restored, green.

Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant