Repository navigation
Conversation
j0nsh1n
force-pushed
the
rdna3/k-native
branch
from
November 17, 2026 21:01
fb9da1a to
5d3ddbe
Compare
The block kernel called __builtin_amdgcn_readfirstlane directly and held a "+s" asm constraint for the RESIDENT kernels. Neither exists in the host build (-DWG_EMU), so build.py --emu-only and every emulator test failed with "use of undeclared identifier '__builtin_amdgcn_readfirstlane'" and "invalid output constraint '+s' in asm". The wave index now goes through wg_uniform, which is readfirstlane on the device and the value itself on the host. The asm is left out of the host build (#ifndef WG_EMU), the same form as upstream PR phantoslayer21#279 for this line. Device code is unchanged. Verified: build.py --emu-only fails with those two errors before and builds after; TestLayoutIndependence passes (27 s); the 27 gfx1201 code objects have identical SHA-256 before and after (clang 20, ROCm 7.1.1). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
build.py built for gfx12 only. - build.py takes --arch (gfx1200, gfx1201, gfx1100, gfx1101) and --sources (build exactly these kernel sources and write target.json). gfx11 builds post, exposure and frame by default (FAMILY_SOURCES; the matrix kernels need gfx12 builtins) and no Model K / L kernels, variants, resource gate or runtime. - family.py maps exact target names to gfx12 / gfx11 and refuses everything else: gfx1170-1172 are LLVM gfx11 names with the gfx12 matrix design, and gfx1102 / gfx1103 / gfx115x are untested. Runner takes its default accumulator row table from the build's target.json (WG_D_ROW and the d_row argument still win; no file means gfx12, as before). - clean/README.md gains a short Linux section. --rocm names an SDK in the layout of AMD's installer; how the SDK is found is unchanged. Verified: the new tests fail before and pass after; mapping gfx1170 to gfx11 turns them red, restored they are green. build.py --arch gfx1100 --sources post.hip exposure.hip frame.hip exits 0. The gfx1201 code objects (27 Model M, 8 Model K, 4 Model L) keep their SHA-256 (clang 20, ROCm 7.1.1). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…U_TEST=1) First run of the HIP host path on a Radeon RX 7900 XT (gfx1100): exposure, post, filter and frame kernels through the spec 90 runner. - test_gpu_stages.py (skipped unless WG_GPU_TEST=1; WG_BUILD, WG_ROCM) reuses the emulator test classes of exposure, post and filter with the runner replaced, so their cases, float64 references and limits are the same, and adds color_divide / scale_rgb of frame.hip against FP32 NumPy (bit equal, including zeros, denormals, infinities and NaN). - test_exposure.py: the kernel cases move to KERNEL_CASES / kernel_color; test_post_emu.py: TestFilterEmu.KERNELS names the filter kernels. Both are behaviour-preserving and let the GPU test pick them up. - Runner no longer needs swin.co in a HIP build made for a target without the Swin kernels (gfx11 so far); a gfx12 build still fails at construction. - post_filter_quality9 is left out of the GPU filter test: it differs from reference/post.py in every value on the host emulator as well (the failure already on main); the card agrees with the emulator to 3e-7 relative, so it is not a gfx11 defect. Verified on the card: 8 tests pass (exposure 10 cases, post 14 cases and the reset test, filter 7 sizes x 6 kernels bit-identical across kernels, frame 50 cases). With the references changed on purpose (frame divisor x1.0001, exposure alpha x0.8) 57 subtests fail; restored, green. The emulator tests keep their result (same 7 failures as before, post_filter_quality9). The 27 gfx1201 code objects have the same SHA-256 as before (nothing built differently). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ange, clear errors) RDNA 3 (gfx1100, gfx1101) has no FP8 conversion or matrix instruction and reads the other half of the wave with v_permlanex16 instead of ds_bpermute. The header knows that now; the gfx12 branches are untouched. - WG_GFX11 from __gfx1100__ / __gfx1101__ only (exact names; LLVM's gfx1170- 1172 have the gfx12 matrix design). It forces WG_SOFT_FP8, so every gfx11 kernel uses the software conversions. - wg_xchg16_u / wg_xor16 on gfx11 are one v_permlanex16_b32 (selectors 0x76543210 / 0xfedcba98, fetch-inactive on): lane l gets lane l ^ 16. wg_xchg16_u is the 32-bit word form on every target (ds_bpermute on gfx12, the lane emulator on the host); wg_mem.h's wg_xor16_u uses it on gfx11. - wg_wmma_fp8, wg_wmma_f16, wg_dot4_fp8 (WG_NORM_DOT4) and the WG_PROFILE counters (wg_realtime, wg_cycles, wg_hwid, wg_drain) are declared with __attribute__((error(...))) on gfx11: a call that survives to code generation fails with a plain message, while the dead `if (WG_NORM_DOT4 && ...)` branches of swin.hip still compile. - probe.hip: the matrix kernels and the hardware conversions are not built for gfx11; probe_xchg16 (gfx11 and the emulator only, so the gfx12 probe.co keeps its hash). build.py's gfx11 default sources gain probe.hip. - selftest.py: a gfx11 build skips the matrix and hardware-conversion probes, and runs the exchange probe (table, word and float forms, NaN payloads) plus the software conversion checks on the card. - pipeline.py takes its default accumulator row table from the build's target, as stage_runner.py does (WG_D_ROW and d_row still win). Verified on a Radeon RX 7900 XT: selftest.py on a gfx1100 build passes, with the exchange table [16..31, 0..15] (lane l gets lane l ^ 16), 0 word and 0 float mismatches in 512 lanes, and the software conversions bit-exact against reference/fp8.py (4274 encode, 256 decode, 8544 pack, 0 mismatches). New tests: the exchange in the emulator (test_kernels_emu), the gfx11 compile checks (test_gfx11_platform: macro, one v_permlanex16 on gfx11 and ds_bpermute on gfx12, each error message, dead code still compiles) and Pipeline's default rows (test_family). With the exchange replaced by the lane's own value, the card probe reports 512 lanes wrong and the emulator test fails; restored, both pass. The 27 gfx1201 code objects have the same SHA-256 as before. The emulator, build, family and stage tests (87) keep their result: only the 7 failures of post_filter_quality9 that predate this. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…16 form
On gfx11 every FP8 conversion is software. The plain form (rint for the
subnormal range, integer rounding for the rest, selects for the edges) costs
about 19 vector instructions per value. This adds a form with the same
results: z = |x| clamped to [2^-6, 448] gives the magic constant
C = 1.5 * 2^(E + 20); the FP32 sum |x| + C rounds to the E4M3 grid with ties
to even, and the difference sum - C is exact. The code comes from the sum's
bits, or the value goes to F16 bits for the next F16 matrix operand.
- wg_platform.h, gfx11 only (WG_FAST_Q8; -DWG_SLOW_Q8 selects the plain
form): wg_f32_to_e4m3, wg_pack_q8, wg_pack_q8_inner and wg_pack_q8_nc (for
|x| <= 464 the bytes of wg_pack_q8) replace the software forms there, and
wg_q8_to_f16, _inner (NaN -> +-0) and _nc (|x| <= 464) are new. The gfx12
path and the host emulator keep their forms.
- The sum must not be reassociated: the header refuses -ffast-math and
-ffinite-math-only, build.py's compiler flags are checked for the others,
and a compile test shows that the sums disappear with -fassociative-math
-fno-signed-zeros (test_gfx11_platform.FastE4m3).
- runner/q8_exhaustive.py and probe_q8_sweep (probe.hip, gfx11): all 2^32
FP32 patterns in 256 chunks of 2^24, converted on the card by the functions
above and compared with a NumPy reference written from the definition
(checked against reference/fp8.py, test_gpu_q8.TestReference). About 0.4 s
per chunk, 1:42 in total, 1.9 GB of host memory, 150 MB on the card.
test_gpu_q8 (WG_GPU_TEST=1) runs 11 chunks that cover every class of input.
Verified on a Radeon RX 7900 XT (gfx1100, clang 20, ROCm 7.1.1): all 2^32
patterns, 0 mismatches in each of the 7 functions; selftest.py still reports
the software conversions bit-exact against reference/fp8.py. The check can
fail: with the magic constant's mantissa made odd (ties go wrong) 87 patterns
of the 11 chunks differ in every function, and with the F16 saturation at 480
instead of 448 19.9 million differ; restored, 0. The 27 gfx1201 code objects
have the same SHA-256 as before.
Timing, 672 workgroups x 256 lanes, 4096 iterations x 16 values (11.3 G
values) per launch, median of 10 launches, the same loop without a
conversion subtracted (2.1 ms), two runs:
FP32 -> 8 code bytes (wg_pack_q8): plain 13.6 / 13.7 ms, new 9.0 / 9.1 ms
(1.5x; 1.21 -> 0.80 ps per value over the device)
FP32 -> F16 operand: plain pack + widening 19.4 / 18.4 ms, new
wg_q8_to_f16 6.9 / 6.9 ms (2.7x to 2.8x; 1.69 -> 0.61 ps per value)
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…e of the lane emulator
The Swin kernels call wg_wmma_fp8 and wg_wmma_f16 with the gfx12 operand
shape: a lane holds 8 K values of its half of the wave. On gfx11 these are
now made from the F16 matrix instruction, so the kernels and the weight
blob stay as they are.
- wg_platform.h, gfx11 and the emulator's gfx11 mode (-DWG_EMU_GFX11): the
lane takes the 8 values of the other half with wg_xchg16_u, puts them
first or second by its half (K slot (h, j) is index 8h + j, both
operands alike), widens the 16 codes to F16 and calls wg_wmma16_f16
(v_wmma_f32_16x16x16_f16). The widening puts the 7 magnitude bits under
the F16 exponent and multiplies by 256: all 256 codes are exact, and the
NaN code 0x7f gets bit 14 so that it stays a NaN (a compare and select
per code made the block kernels spill; this form costs about 3 VGPRs).
The accumulator rows are 2v + h: d_row "gfx11".
- emu/wg_emu.{h,cpp}: wmma16_f16 models the instruction as measured on an
RX 7900 XT: output lanes 16h..16h+15 read A and B from the lanes of
their own half. (The documented layout repeats lanes 0-15 in lanes
16-31; the wrappers do.) build.py --emu-gfx11 builds wg_emu__gfx11.so;
Runner(..., backend="emu", arch="gfx11") loads it and takes d_row "gfx11".
- probe.hip: probe_wmma_fp8 / probe_wmma_f16 run on gfx11 through the
wrappers; probe_wmma16 calls the instruction on raw registers.
- build.py: swin.hip for gfx11 is built with -DWG_SLOW_Q8 (FAMILY_DEFINES).
With the faster conversion of the previous change the C=128 block kernel
spills 27 VGPRs and the other block kernels take 30 to 35 more; the
speed slice budgets that.
Tests: test_kernels_emu_gfx11 runs enc1 (and the other stages of the
emulator tests' extra cases) through the wrappers against the float64
reference with the limits of test_kernels_emu, and the wrappers against
exact arithmetic (random E4M3 and F16 operands bit for bit, every code,
NaN codes). test_gfx11_platform checks the instruction mix of the wrappers
for gfx1100 and gfx1201, and gates the registers of the five block kernels:
kernel VGPRs spills waves/SIMD
swin_block_c64 175 0 8
swin_block_c64_add 175 0 8
swin_block_c96 206 0 7
swin_block_c96_add 206 0 7
swin_block_c128 250 0 5
(clang 20, ROCm 7.1.1, -DWG_SLOW_Q8; patch_merge_c64 237, 0, 6)
The comparisons can fail: swapping the half select for operand B only gives
91 % of the enc1 skip values more than one step off; using the lane's own
half twice or dropping the multiply by 256 fails the wrapper and enc1 tests;
restored, all pass. Swapping the select for both operands is not caught, by
design: both then take the same K order, and an exact sum does not depend on
it (the hardware rounding does). The 27 gfx1201 code objects have the same
SHA-256 as before; the gfx11 build of swin.hip compiles; no card run yet.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…es and the exchange in divergent code runner/selftest.py now runs the matrix probes on a gfx11 build (through the wrappers of the previous change) and has three new probes for gfx11 and the gfx11 emulator mode (wg_emu__gfx11.so, --emu-library). - probe_wmma (existing): on the RX 7900 XT it reports d_row [[0, 2, .., 14], [1, 3, .., 15]], name "gfx11", for FP8 and F16, with A row = lane % 16, B column = lane % 16, K slots paired, 1 x 1 = 1. - probe_wmma_arithmetic: 32768 random problems per kind (8.4 M outputs) against exact float64 sums, whatever the K slot order; error in units of 2^(E - 24), E the exponent of the largest product. Card, E4M3 (uniform random codes): 98.86 % within 4 units, 100 % within 16, largest 17.1 (analysis: largest 23 on 33.5 M outputs); F16 (normal values): 87.8 %, 99.96 %, largest 37.3; the same in a second seed. The emulator's exact model: 99.9999 % within 4, largest 8. Limits: none beyond 64; E4M3 at least 98 % within 4; F16 at least 99.9 % within 16 (the 99 % within 4 of the plan is the analysis' figure for the P1b-like data, 99.8 %; uniform random codes give 98.9 %). - probe_wmma16_lanes (probe_wmma16, the instruction on raw registers): lanes 16-31 of A and B are not ignored. The lanes 16-31 of the result (odd rows 2v + 1) take A and B from lanes 16-31, lanes 0-15 (even rows) from lanes 0-15. The wrappers give both halves the same operands, so the sum does not depend on this; with other operands in the halves it does. - probe_xchg16_divergent (probe_xchg16_div, gfx11): wg_xchg16_u in code that only some lanes execute (lanes 0-15 only, lanes 16-31 only, odd lanes, every fourth lane, a data-dependent half): every lane that executes gets the value of lane l ^ 16, also when that lane does not execute (its register is read all the same); the other lanes are untouched. 0 wrong values in all five modes. Tests: test_gpu_probes (WG_GPU_TEST=1) asserts the card's report; test_kernels_emu_gfx11 runs the same probes in the gfx11 emulator mode. Wave64 post on gfx11 is still not tested. The probes can fail: built with the multiply by 256 left out of the widening, the card reports 1 x 1 = 1.5e-05 (2^-16) and the arithmetic probe is off by up to 1.6e8 units; built without the exchange, 1 x 1 = 2 and the K pairing fails; an emulator that reads lanes 0-15 for B in all rows fails the selftest and test_the_instruction_reads_each_half_for_its_own_rows. Restored, everything passes (card: 5 probe tests; A2-A4 card tests 27). The 27 gfx1201 code objects have the same SHA-256 as before. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…n.hip in the gfx11 build) The default build for gfx1100 / gfx1101 now includes swin.hip (built with -DWG_SLOW_Q8, see the first change), so Runner on a gfx11 build runs the Swin stages. test_gpu_enc1 (WG_GPU_TEST=1) runs enc1 (swin_block_c64, then patch_merge_c64) on the card with the emulator tests' four enc1 cases. Measured on a Radeon RX 7900 XT (gfx1100, clang 20, ROCm 7.1.1, headless): - Against the float64 reference (the emulator tests' cases, d_row "gfx11"): spec 09 holds in all eight outputs: no non-finite value; 0.05 to 0.77 % of the values more than one FP8 step off (limit 2 %); mean absolute error 0.005 to 0.115 % of the reference RMS (limit 1 %). - The stricter limits of the emulator tests: below 5 % different per case holds (0.2 to 3.4 %). Below 1 % different over all cases does not: 1.0076 % over the four enc1 cases (173,184 values). The gfx11 emulator equals the reference on these cases, so the same number is the distance card to emulator: 10,076 per million one-code flips, 2,292 per million more than one step (per case and output 2,000 to 34,000 per million; the analysis expects about 450 per million on realistic data, and random networks amplify). Swapping the K order of the odd rows on the card (a change of rounding only) flips 1,386 per million, so most of it is not the order of the sum. The limit is a product question (the plan's question 7); the test is an expected failure, so a change shows. - Time, Runner.time, median of 50 launches, inputs on the card: 3.17 to 3.23 ms at the 1706x960 render (480x864 tokens), 6.86 to 6.97 ms at 2560x1440 (720x1280), over three runs. The tree gives enc1 0.35 ms at the first size for an RX 9070 XT (runner/sweep.py GAME_SIZES) and 0.94 ms for all of Swin at 1440p Quality (clean/l/SPEED.md): a different card, so about 9 times only as a rough comparison. Static ISA of swin_block_c64: 12,260 instructions, 10,519 vector ALU, 44 v_wmma_f32_16x16x16_f16, 124 v_permlanex16_b32, 432 v_pk_mul_f16. Without the widening the kernel has 3,940 vector ALU instructions: the widening is about 6,600 of them, 150 per matrix instruction. Checks that can fail: WG_D_ROW=gfx12 on the gfx11 build fails the spec 09 test in all eight outputs; a packer that ignores d_row fails test_the_row_table_matters (0.0 differ, limit 0.2); restored, both pass. The 27 gfx1201 code objects have the same SHA-256 as before. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On gfx11 every matrix instruction widened both E4M3 operands to F16 (about 6 600 of the 10 500 vector instructions of swin_block_c64). The weights, which one wave uses once per window, are now widened when the staged unit is copied to LDS: once per workgroup, the fragment stored as two planes of 16 rows, read back as 2 x 16 bytes. Opt-in (WG_F16_W, variants.py "f16w", build.py --arch gfx1100 --variants f16w); the default gfx11 build and every gfx12 object are unchanged. A unit of the plain block kernels holds half the code bytes, so the LDS stays the same; the C = 128 kernel keeps codes (with F16 fragments it needs more than 256 registers). The C = 64 kernels take the 8-wave occupancy floor (without it 256 registers, 27 spilled). Verified: bit-identical to the default gfx11 build in the gfx11 emulator for the five block kernels and merge / expand (test_kernels_emu_gfx11.TestVariantsGfx11); identity goes red when one FP32 sum order is changed in the variant and green when restored. On the RX 7900 XT enc1 meets spec 09 as before; median of 50: 1.90 ms at 1706x960 (3.2 ms before) and 4.04 ms at 2560x1440 (6.9 ms before). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On top of f16w: each activation operand (the quantized values of a 16-token tile, used by up to 16 matrix instructions) is exchanged with the other half of the wave and widened to F16 where it is quantized (pack_act), 8 registers per K group instead of 2, and the softmax probabilities are exchanged once for both value tiles. The register cost makes the C = 96 kernels spill 30, so the variant applies to the C = 64 kernels only (the others stay as f16w); the C = 64 kernels take an occupancy floor of 7 waves (8 spills 8). Opt-in (WG_F16_A, variants.py "f16wa"); the default gfx11 build and every gfx12 object are unchanged. Verified: bit-identical to the default gfx11 build in the gfx11 emulator (TestVariantsGfx11); the identity goes red when one FP32 sum order is changed under WG_F16_A and green when restored. enc1 on the RX 7900 XT meets spec 09 as before; median of 50: 1.63 ms at 1706x960 and 3.48 ms at 2560x1440 (f16w 1.90 and 4.04, default 3.2 and 6.8). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On top of f16wa: the activation operands come from the exact E4M3 rounding done as FP32 arithmetic and v_cvt_pk_rtz_f16_f32 (wg_q8_to_f16), not from code bytes widened again; the code bytes are made only for the stores. That takes the plain software conversion (about 3 900 vector instructions per window of swin_block_c64) and the widening of the activations out of the kernel: 6 060 to 3 620 vector instructions in swin_block_c64. The attention output o keeps the code form (with the F16 form the C = 64 kernels spill 75 registers). The F16 forms of the rounding (wg_q8m_round, wg_q8_to_f16*) move out of the WG_FAST_Q8 block of wg_platform.h so that they exist in the default gfx11 build (plain code conversion, -DWG_SLOW_Q8) and in the emulator; the code is the same. -DWG_SLOW_Q8=0 now selects the faster code conversion. Opt-in (WG_F16_Q, variants.py "f16wq"); the default gfx11 build and every gfx12 object are unchanged. Verified: bit-identical to the default gfx11 build in the gfx11 emulator (TestVariantsGfx11); the identity goes red when one FP32 sum order is changed under WG_F16_Q and green when restored. enc1 on the RX 7900 XT meets spec 09 as before; median of 50: 1.30 ms at 1706x960 and 2.79 ms at 2560x1440 (f16wa 1.63 and 3.48, default 3.2 and 6.8). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
runner/gfx11_decision.py measures stages enc1 and b0 (C = 128) for the default gfx11 build and each variant of variants.FAMILY_VARIANTS["gfx11"], and prints one markdown sheet: spec 09 numbers against the float64 reference, the verdict word of equiv/compare.py on the same pair (its use on a stage output against our own reference is marked INFERRED in the sheet), the one-code flip fraction against the gfx11 emulator, VGPRs, spills and waves per SIMD from the compiler, the median of 50 launches at 1706x960 and 2560x1440, and a per-frame Swin time extrapolated from them (INFERRED; the RX 9070 XT figure of l/SPEED.md is a different-card comparator only). It presents evidence and decides no output standard. Synthetic weights and inputs only. tests/test_gfx11_decision.py (CPU only): the verdict mapping on maps with a known share of one-step differences, the spec 09 limits, the per-frame arithmetic, the resource-report reading and the structure of the sheet. Checked red on purpose by reporting DIFFERENT as CLOSE, loosening the 2 % limit and counting b0 five times per frame; green again after restoring. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…r them (E1) The FP16 route of the C = 96 (enc2, dec2), C = 128 (b0 to b5) and `_add` (dec1, dec2) block kernels and the weight-resident `_r` kernels on the RX 7900 XT, each against the float64 reference with the acceptance of test_gpu_enc1 (provisional: the output standard is the maintainer's decision): spec 09 per case and output, fewer than 5 % of the values different per case, the "< 1 % over all cases" of the emulator tests. Worst case and output, default gfx11 build, over the cases of each test: stage more than 1 step off mean abs error (% RMS) values different over all cases enc2 0.977 % 0.217 4.54 % 0.638 % b0-b5 0.187 % 0.037 1.12 % 0.631 % dec2 0.181 % 0.033 0.94 % 0.839 % dec1 0.225 % 0.027 0.77 % 0.454 % No non-finite value, no limit missed, and the overall bound holds for every group (enc1's 1.008 % stays the one expected failure). Distance to the gfx11 emulator, reported: 4 500 to 6 900 one-code flips per million over all cases. The `_r` kernels give the plain kernels' bytes for 1, 2 and 5 workgroups (each wave then takes several windows), and the same reference numbers. Runner.resident = N launches them for enc1 and dec1 (the runtime's swin_resident); at 84 workgroups they are slower than the plain kernels on this card (enc1 4.0 against 3.2 ms at 1706x960, dec1 4.0 against 3.4 ms). Finding, pinned as an expected failure and left to slice E2: expand_c96 (107 spilled registers) writes NaN codes for valid tokens when a wave's second 16-token tile is only partly valid (60, 84, 120 low-resolution tokens; 36, 40, 100, 288 are fine). The emulator tests' case dec1 12x20 gives 5 120 non-finite values on the card for this reason, so the dec1 cases here use sizes without that shape; the block kernel `swin_block_c64_add` is not involved (dec1 36x32 and 16x20 are within the limits). gfx11_gates.py and gfx11_gates.json: compiler-reported VGPRs, spills and waves per SIMD of the 14 block kernels (`_nc` forms included) in the default gfx11 build and in f16w, f16wa and f16wq. A gate is no spill, at most the recorded registers, at least the recorded waves; a new table for the gfx11 builds, not clean/dx12/vgpr_limits.json. `python clean/runner/gfx11_gates.py --rocm <ROCm> --scratch <folder>` checks, `--write` records (clang 20). The merge, expand and fused kernels are not gated: they spill in these builds (E2). Verified: the new tests and test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages, test_kernels_emu_gfx11, test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass. Break on purpose, red then green: c128 heads and the dec skip add (bottleneck, dec1, dec2 red, enc2 green); c96 heads (enc2 red); the resident loop skipping a window (bit-identity red); the runner ignoring d_row and the resident option (row-table and launch tests red); one GELU coefficient changed (the overall bound red, spec 09 still green); the gate check's spill test, and a recorded wave count one too high (the compile-based gate test red). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…c128w7) The default gfx11 build leaves swin_block_c128 without an occupancy floor: the allocator takes 250 registers and 5 waves per SIMD. With a floor of 7 waves (WG_MINW_C128=7) it fits in 206 registers with no spill; a floor of 8 gives 192 and no spill too. Nothing else changes: no source edit, one entry in variants.py. On the RX 7900 XT, median of 50 launches, three runs (ms), at 1706x960 and 2560x1440: stage default c128w7 b0 1.089 1.117 1.171 (1.117) 0.911 0.913 0.910 (0.911) 2.139 2.130 2.154 (2.139) 1.887 1.744 1.743 (1.744) b1 1.092 1.085 1.094 (1.092) 0.831 0.827 0.830 (0.830) 2.086 2.090 2.073 (2.086) 1.616 1.618 1.615 (1.616) That is 18 to 24 % less per launch, six launches per frame. A floor of 8 timed the same at 1706x960 and 1.65 ms at 2560x1440. New on the card: TestVariantsGpu runs every gfx11 variant of the build against the default gfx11 build on seven cases of every block shape and asks for the same bytes. They are the same for f16w, f16wa, f16wq and c128w7 (the F16 operands are exact and the K order is the matrix instruction's own). The reference numbers of the stage tests for the c128w7 build equal the default's. gfx11_gates.json gets the row of c128w7 (206 registers, no spill, 7 waves). Verified: the new test red with a build whose c128w7 object has one GELU coefficient changed, green again with the real one; the gate table test red with the c128w7 row deleted; test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages, test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11 (c128w7 bit-identical to the default in the emulator), test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
Timing-only builds (output wrong on purpose) show where the C = 128 kernel's 1.1 ms goes on the card: without the W1 and W2 multiplies it takes 0.41 ms, without the QKV multiplies 0.75 ms, with no matrix work at all (gather and store) 0.035 ms. The matrix instructions themselves are a small part of that (INFERRED: about 770 of them a window); each call also widens its operands from codes to F16, and f16wq already removed that for the C = 64 kernels, whose activation operands are made once as F16 (C < 96). The wider kernels kept codes because the F16 operands cost registers (C = 96: 30 spilled, then 12 with the attention output left on codes). Three knobs name the smallest C they apply to (0 = none, default build unchanged): WG_F16_AWIDE makes the activation operands F16 operands in the kernels with C >= N, WG_F16_QWIDE makes them straight from the FP32 values there too, WG_SCHED_FRAGS puts a scheduler barrier after each fragment step of the QKV, Wp, W1 and W2 loops of those kernels. Variant f16wk is f16wq with AWIDE = QWIDE = SCHED = 128: only swin_block_c128 changes. 238 registers, 6 waves per SIMD, no spill (default: 250 registers, 5 waves). Without the barriers the register count is the same and the kernel is 20 % slower (0.85 against 0.70 ms at 1706x960; the cause was not looked into). F16 weights for C = 128 (WG_F16_W on this kernel) still spill 94 registers with or without the barriers, so the weights stay codes there. RX 7900 XT, median of 50 launches, three runs (ms), 1706x960 and 2560x1440: stage default f16wk default f16wk b0 1.089 1.117 1.171 (1.117) 0.705 0.715 0.703 (0.705) 2.139 2.130 2.154 (2.139) 1.276 1.290 1.284 (1.284) b1 1.092 1.085 1.094 (1.092) 0.628 0.626 0.634 (0.628) 2.086 2.090 2.073 (2.086) 1.193 1.193 1.186 (1.193) 37 to 43 % less per launch; c128w7 (previous commit) gave 18 to 24 %. Same card bytes as the default build on seven cases (TestVariantsGpu), same reference numbers as the default build (TestBottleneckGpu, TestEnc2Gpu, TestDec2Gpu with the f16wk build). Verified: bit-identical to the default gfx11 build in the gfx11 emulator for every case (TestVariantsGfx11); the identity goes red for f16wk, stage b1 only, when the C = 128 operand is scaled by 1.01 under WG_F16_QWIDE, green when restored; gate table re-recorded with the f16wk row (c64 212 / 7 waves, c96 248 / 5, c128 238 / 6, no spill); test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages, test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11, test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…16wm) The C = 96 block kernels (enc2, dec2) keep code activations under f16wq: F16 operands for the whole block spill 12 registers (30 with the operands made from FP32). Their MLP is the larger half of the matrix work (W1 and W2 against the norm output and the hidden values), so WG_F16_M makes only those two operands F16 operands, made once and used by every hidden tile, in the kernels with C >= 96 that WG_F16_A leaves on codes. Variant f16wm is f16wk with WG_F16_M = 1 and the scheduler barriers of WG_SCHED_FRAGS in the C = 96 kernels too (WG_SCHED_FRAGS = 96): without them the C = 96 kernels spill 23. The operands are made from codes, not straight from the FP32 values (that spills 133); the C = 64 kernels are f16wq's and the C = 128 kernel f16wk's. swin_block_c96[_add]: 252 registers, 5 waves per SIMD, no spill (f16wq: 248, 5 waves). RX 7900 XT, median of 50 launches, three runs (ms), 1706x960 and 2560x1440: stage f16wq f16wm f16wq f16wm enc2 1.523 1.509 1.505 (1.509) 1.328 1.347 1.322 (1.328) 3.002 3.016 2.995 (3.002) 2.648 2.697 2.688 (2.688) dec2 1.206 1.199 1.214 (1.206) 1.042 1.036 1.036 (1.036) 2.395 2.384 2.385 (2.385) 2.058 2.055 2.056 (2.056) 12 to 14 % less than f16wq, 44 % (enc2) and 49 % (dec2) less than the default gfx11 build (2.360 and 2.014 ms at 1706x960). Same card bytes as the default build on seven cases (TestVariantsGpu); the reference numbers of enc2, dec2, dec1 and the bottleneck with the f16wm build equal the default's (TestEnc2Gpu, TestDec2Gpu, TestDec1Gpu, TestBottleneckGpu). Verified: bit-identical to the default gfx11 build in the gfx11 emulator for every case (TestVariantsGfx11); the identity goes red for f16wm, stages enc2 and dec2 only, when the hidden values are scaled by 1.01 under WG_F16_M, green when restored; gate table re-recorded with the f16wm row (c64 212 / 7 waves, c96 252 / 5, c128 238 / 6, no spill); test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages, test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11, test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…x11 (E2) On gfx11 expand_c96 wrote NaN codes (or wrong values) for valid tokens when a 16-token tile of a wave was only partly valid (60, 84, 120 or 140 low-resolution tokens). The store was `if (token < ntok)`; the compiler moved the whole v_wmma chain of the result into that branch (its only user), so the matrix instruction ran with EXEC partly off. v_wmma needs every lane executing; the emulator runs all lanes, which is why it never showed. The same move made the kernels spill (expand_c96 107 registers, patch_merge_c96 101) and patch_merge_c64 and c96 carried the same latent hazard. Fix at the source, for the gfx11 family only (WG_TILE_STORES_UNGUARDED; the other targets keep their code, gfx1201 objects are byte-identical): a lane past the last token repeats the work of the last token and stores the same bytes to the same place, so there is no branch to move work into. Results are unchanged: the kernels are bit-identical in the gfx11 emulator. Spills now 0 in every gfx11 build: patch_merge_c64 185 VGPRs, patch_merge_c96 251, expand_c96 199, expand_c128 248; they join the gate table. On the card (1706x960): patch_merge_c96 0.639 -> 0.422 ms, expand_c96 0.688 -> 0.418 ms, patch_merge_c64 and expand_c128 unchanged. The expected-failure test becomes a passing test and a new test runs each kernel alone with partial tiles against the gfx11 emulator (red with the branch put back, green with the fix). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… gfx11 (E2) The default patch_merge widens the weight fragment and the token operand for every matrix instruction (the half-wave exchange is not merged), so each v_wmma pays for two widenings. In the variant the loops run K group outermost over 6 (C = 64) or 4 (C = 96) output tiles at a time: each token operand is widened once for all of them and each weight fragment once for both token groups. Every accumulator adds its K groups in the same order, so the result is bit-identical (gfx11 emulator, card). Opt-in (variants.FAMILY_VARIANTS): the default build and the gfx12 objects are unchanged. No spills in any kernel of the build (patch_merge_c64 250 VGPRs, patch_merge_c96 143, expand_c96 199, expand_c128 248); the table of resource gates has the mgx build. On the card (RX 7900 XT, median of 50, 1706x960): patch_merge_c64 0.402 -> 0.354 ms, patch_merge_c96 0.422 -> 0.311 ms (2560x1440: see the report); expand is unchanged. Tried and not adopted: 3 token groups per wave (merge c96 0.223 ms, expand_c128 0.342 ms but 9 spilled registers; splitting the expand into passes spills more or gains nothing), 128-lane workgroups, 1 token group per wave. Also: gfx11_gates.py --write no longer empties the table when a kernel spills (it built the table after opening the file); the partial-tile test has a third case size (36 tokens: a partial third tile with 3 groups per wave). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…rd (E3) swin_v2.hip with swin.hip's arithmetic (swin__bmegahg.co, the runtime's default for b0..b5 on gfx12) joins the default gfx11 build: swin_bneck6 (b0 to b5 in one launch) and swin_mid (enc2 with its merge, b0 to b5, dec2 with its expand in one launch). No kernel source changes: the file compiles for gfx1100 through the wrappers of wg_platform.h, and the sync-word protocol is the one of gfx12. build.py builds the object with the family's defines (variants.FAMILY_MEGA) and writes variants.json for it; the Runner takes it as a gfx11 variant with the grids of the family's card (FAMILY_MEGA_GROUPS: 126 workgroups for swin_bneck6, 84 for swin_mid; any grid is correct, these are the resident capacity). The E2 hazard, checked: a control-flow scan of the compiler's assembly (every basic block reachable under an s_and_saveexec before its join) finds no v_wmma and no v_permlanex16 under a lane-divergent exec in swin_bneck6, swin_mid or fused_merge. The kernels have no branch round a store for the compiler to sink a matrix chain into. Card (RX 7900 XT), against the plain launches of the default build: bit-identical for both parities, 1 / 3 workgroups and the build's grid, maps with partial windows and one or two window rows, clamping and clamp-free kernels; sync words zero after every launch. Block by block against the float64 reference (each block fed the card's own output of the one before): spec 09 and < 5 % different per block everywhere (worst: b1 12x16 1.27 % different, 0.31 % more than one step off). The six blocks chained against the reference chain compound the one-code flips of the F16 matrix instruction: 26.5 % more than one step off at 12x16 (plain launches the same bytes). A reference chain given flips of 0.2 % of the values per block shows the same amplification (34 % at the end), so this is the random synthetic network, not the kernel. swin_bneck6: 239 VGPRs, 0 spills, 6 waves per SIMD, 32 KB LDS (gate table, new build "bmegahg"; the other recorded builds are untouched). swin_mid: 256 VGPRs, 143 VGPRs and 28 SGPRs spilled, 5 waves: correct on the card but NOT gated. Its two C = 96 phases (enc2 with the merge, dec2 with the expand) spill about 70 registers each, the C = 128 phases none; plain C = 96 phases spill 79 too, so it is the C = 96 streamed body in the 256-lane queue, not the fusion. Two knobs tried without effect (WG_V2_PIPE=0: 131 spilled; no scheduling fences: 141). Open. Times, median of 50 launches, three runs (ms; 1706x960 | 2560x1440): bneck plain 6.165 | 12.354 megakernel 4.600 | 9.369 plain f16wm 3.449 | 6.808 mid plain 10.037 | 20.439 swin_mid 10.127 | 22.272 plain f16wm 5.407 | 11.075 The megakernel is 25 % faster than the default plain blocks and 33 % slower than the f16wm blocks; swin_mid is level with the plain launches at 1706x960 and 9 % slower at 2560x1440 (the spills), and 1.9 times f16wm. Grid sweep (swin_bneck6, 1706x960): 42 / 84 / 126 / 252 workgroups 12.0 / 6.2 / 4.5 / 4.5 ms. Header in LDS (bmegahg with WG_V2_MEGA_HDR_GLOBAL=0) 4.40 ms, 128 lanes 8.0 ms: no variant is added. Verified: TestBmegaGfx11 and TestMidGfx11 (the existing v2 emulator tests with a gfx11 mode) bit-identical to the plain gfx11 kernels; the gfx12 modes of those classes unchanged and green; clean.tests.test_gpu_swin_mega (new, card) and test_gpu_swin_stages, test_gpu_enc1 (known expected failure), test_gpu_probes, test_gpu_q8, test_gpu_stages, test_kernels_emu_gfx11, test_gfx11_platform, test_gfx11_decision, test_gfx11_gates (compiler check of all ten builds), test_build_targets, test_family, test_hip_env pass. Break on purpose, red then green: a wrong ping-pong buffer in swin_bneck6 plus a wrong expand slot in swin_mid (emulator tests: 14 failures; card tests red), a plain build with the attention switched off (card bit-identity and both reference tests red), the gate entry removed, the persistent build not built, a gfx12 build accepted as a gfx11 variant. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…t-build blocks on gfx11, and a scan that checks it The scan of the gfx11 code objects found every v_wmma and v_permlanex16 of swin_block_c64[_add][_nc] in swin__bmegahg.co under one lane-divergent mask (s_and_saveexec on vcc): the window loop's `if (win0 + i >= nwin) break;`. win0 is the wave's first window, a vector value to the compiler, so it could not see that the test is the same for the whole wave. v_wmma and v_permlanex16 need every lane executing. At run time the mask is all or nothing (win0 comes from the wave index), so the kernels passed the card cases; they carried the pattern. Fix for the gfx11 family only (WG_TILE_STORES_UNGUARDED, the switch of the E2 fix): win0 goes through wg_uniform (readfirstlane), the break becomes a scalar branch, and the whole loop runs with EXEC full. Other targets keep their code. Spills of the four kernels fall 48/46/108/90 -> 41/40/11/50 (still above zero: these kernels are not in the gate table, the runtime uses swin_bneck6). clean/runner/exec_scan.py: disassembles a code object and runs a forward data-flow over the control-flow graph of each function: EXEC is partial after s_and_saveexec, s_and/s_and_not1/s_xor into exec and v_cmpx, restored by s_or/s_mov exec from a register saved from EXEC (and after a divergent loop), joins are "partial if any path is". It lists every v_wmma / v_permlanex16 that may run under a partial mask. test_exec_scan: 15 hand-written functions (each rule, with a look-alike that must not be flagged) and, with a gfx11 build, a scan of every kernel the runner launches (0 hits in the fixed build, 4 kernels red in the build before this commit). On the card the old code cannot give a wrong result (the mask is uniform), so TestResidentBlockGpu (partly filled workgroups, 1 to 17 windows, both block kinds, clamping and clamp-free) is green before and after; it goes red with the window count off by one (checked). The scan test is the one that is red before. Not changed, reported by the scan: swin_mid and the fused kernels (_mg, _ex, fused_merge) still carry the pattern (swin_mid is parked for its owner); post.co (v_permlanex16 in the reconstruction kernels) and probe_xchg16_div (deliberate) are flagged too. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…under a lane mask (E4) kernels/encoder.hip builds for gfx1100 with the gfx11 wrappers unchanged (the packers and the weight blob are the ones of gfx12: only d_row differs). On a partly valid 16-token tile the first card build was wrong: the features store is `if (tok < ntok)`, the compiler moved the whole projection (the half-wave exchange of the pixel codes, v_permlanex16, and the v_wmma chain) into that branch, and both need every lane executing. Cases 3 and 13 of the encoder cases (22x18 padded to 26x20, 4x6: 130 and 12 tokens) gave 8 % and 23 % wrong features. Fix, gfx11 family only (WG_ENC_STORE_UNGUARDED, the E2 rule): the features are stored from every lane; a lane past the last token repeats the work of the last token (its loads are already clamped to it) and stores the same bytes to the same place. Other targets keep their code; gfx1201 objects are unchanged. Resources (gfx1100, clang 20.0.0.rocm): input_encoder 142 VGPRs, 0 spills, 10 waves per SIMD (gate recorded: new "enc0" entry in gfx11_gates.json; nothing else re-recorded). exec_scan.py: 0 of 113 matrix / exchange instructions under a lane mask (122 of 126 before the fix); the scan learns the if / else shape `s_and_not1_saveexec` that the encoder's code contains. Card (RX 7900 XT), the 14 cases of test_encoder_emu against the float64 reference: 0.000 to 0.003 % more than one step off, mean error at most 0.0015 % of RMS, 0.000 to 0.095 % of the features different per case (bound 5 %), 0.0575 % over all cases (467 840 values; the 1 % overall bound of the provisional acceptance); against the gfx11 emulator 0 to 946 per million one-code flips, the motion surfaces equal. The emulator test's own limits (0.01 % overall) are an expected failure on the card. Time, median of 50 launches, three runs: 1706x960 0.637 0.616 0.633 ms, 2560x1440 1.281 1.275 1.289 ms. Tests: test_gpu_e4.TestEncoderGpu (card; red with the encoder object built before this fix: cases 3 and 13 and the partly-valid-tile test; red with the store moved by 16 bytes: every case fails), test_e4_gfx11_emu (the same cases in the gfx11 emulator; red with the same mutation), test_exec_scan (encoder.co joins the kernels the card runs: red with the old object), test_gfx11_gates. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
kernels/decoder.hip needs no source change for gfx1100: its projection stays in registers, the pixel stores are scalar-condition stores with no matrix work behind them, and the scan finds 0 of its 68 matrix / exchange instructions under a lane mask. The change is the build list (decoder.hip joins the default gfx11 sources), the gate and the tests. Resources (gfx1100, clang 20.0.0.rocm): coef_decoder 103 VGPRs, 0 spills, 12 waves per SIMD, LDS 10 240 bytes (new "dec0" entry in gfx11_gates.json; nothing else re-recorded). Card (RX 7900 XT), the 5 cases of test_decoder_emu, the reset and the border case, against the float64 reference: A, B and latent in 8-bit codes, C in FP16 steps; at most 0.139 % of a case's values more than one step off, mean error at most 0.0052 % of RMS, at most 0.139 % different per case and output (bound 5 %), 0.0354 % over all cases (78 988 values; the 1 % overall bound of the provisional acceptance). Against the gfx11 emulator 0 to 1 389 per million flips. The emulator test's own limits (99.9 % identical) are an expected failure on the card. Time, median of 50 launches, three runs: 1706x960 0.431 0.411 0.413 ms, 2560x1440 0.728 0.729 0.728 ms. Tests: test_gpu_e4.TestDecoderGpu (card), test_e4_gfx11_emu.TestDecoderEmuGfx11 (gfx11 emulator, the emulator test's own limits), test_exec_scan (decoder.co joins the kernels the card runs), test_gfx11_gates. Red with the pixel-pair store condition `x0 + 1 < rw` changed to `x0 + 2 < rw`: the card cases and the emulator cases fail (5 cases, reset and border each); green restored. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
kernels/dec1_dec0.hip builds for gfx1100 unchanged and needs no source change: its epilogue (dec0's half-wave exchange and projection) runs after the swin block's work, in every lane, and the scan finds 0 of its 480 matrix / exchange instructions under a lane mask. The change is the build list (dec1_dec0.hip joins the default gfx11 sources, with -DWG_SLOW_Q8 like swin.hip, whose block it includes: 178 VGPRs against 214 with the faster conversion), the gate and the tests. Resources (gfx1100, clang 20.0.0.rocm): swin_block_c64_add_dec0 and _nc 178 VGPRs, 0 spills, 8 waves per SIMD, LDS 32 768 bytes (new "dec1dec0" entry in gfx11_gates.json; nothing else re-recorded). Card (RX 7900 XT): the fused kernel gives exactly the bytes of dec1 then dec0 (the kernel's contract; the 4 cases of test_dec1_dec0_emu, both clamping variants). Against the float64 reference, dec0's reference fed the card's own dec1 output: at most 0.104 % of a case's values more than one step off, mean error at most 0.003 % of RMS, at most 0.243 % different per case and output (bound 5 %), 0.0266 % over all cases (71 461 values; the 1 % overall bound of the provisional acceptance); dec1 itself 0.065 to 0.169 % more than one step off. Time, median of 50 launches, three runs: 1706x960 fused 3.412 3.446 3.434 ms, two launches 3.593 3.433 3.446 ms; 2560x1440 fused 7.406 7.404 7.439 ms, two launches 7.346 7.644 7.372 ms. The fusion is no faster than the two launches on this card (dec1's block dominates), so no speed variant was added. Tests: test_gpu_e4.TestDec1Dec0Gpu (card), test_e4_gfx11_emu.TestDec1Dec0EmuGfx11 (fused equals separate in the gfx11 emulator), test_exec_scan (dec1_dec0.co joins the kernels the card runs), test_gfx11_gates, test_build_targets (red with the define dropped). Red with dec0's pixel-pair store condition changed: the three reference cases of the card test fail; green restored. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…sts no longer skip) The tests of exec_scan ran with ROCM_PATH set to the link folder made for build.py (it has the compiler, not llvm/bin/llvm-objdump). The tool lookup took ROCM_PATH only and gave up, so TestExecScanBuild skipped every time in those runs, the acceptance runs of E3 and E4 included: the "no matrix instruction under a lane mask" test of the kernels the card runs was never evaluated there. find_objdump() tries WG_ROCM, ROCM_PATH, /usr/lib64/rocm and /opt/rocm in that order, then PATH. Checked by hand on the E4 head build (gfx1100, all kernels the card runs): the test now runs and passes, 0 findings in them. Tests: TestFindObjdump (first candidate with the tool wins; a folder without it is passed over). Red with the existence check dropped (the second case gets the missing path); green restored. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… post kernels on the gfx11 card The scan flags all 117 v_permlanex16 of post.co: 39 kernels (post_recon2 .. post_recon6, post_fused, post_fused3) with three each, the `nb[c] = wg_xor16(hsum[c])` of the reduced-history step, which the compiler put inside the store's branch `if (active && has1 && wl < 16 && x + 1 < dw)`. There is no v_wmma in post.co. Verdict: harmless, not a defect. In that branch only lanes 0 to 15 of a half-wave pair run, so the lanes the exchange reads (16 to 31) are off, but v_permlanex16 reads the register of a lane whose EXEC bit is off all the same (wg_platform.h says so; selftest probe_xchg16_div measures it on the card, mode 0 is this exact shape), and hsum was summed by every lane before the branch. The partner's value is the partner's real one. No source change, no change of any object: the gfx1100 post.co of this head and of the E4 head are the same bytes. The scan cannot prove this (it would need the reaching definitions of the source register), so it is pinned as a known, explained exception: test_exec_scan POST_EXCHANGE_KERNELS, exactly these 39 kernels with exactly three v_permlanex16 and no v_wmma each; a new site, a missing one, or a matrix instruction in a post kernel fails the test. Card (RX 7900 XT), new test_gpu_post: the emulator test classes of the post kernels (recon2 family, recon3 / 5 / 6, recon4, banded, output mode, fused, fused3) run on the card with their own cases (odd D, partial 32-column tiles, one D row or column, 3x ratio, NaN / inf inputs, reset): bit-identical to post_recon, which has no exchange, and the float64 limits where the emulator test has them. 21 tests, 0 failures; 38 of the 39 kernels are launched (a test checks the list). post_recon2_r_out_diet is not: the runtime's option post_diet=1 is its only caller and the stage runner has no launch plan for it; its body is post_recon2_r_out's. Not used by the Model M path on this card, for the record: the C++ runtime has no gfx11 build; its defaults would be post_recon2_r_band[_p12] with post_filter_c1_s4_band (no exchange), post_recon as the fallback. Red with the exchange removed (wg_xor16 returns its argument on gfx11, post.co rebuilt): 191 failures in test_gpu_post (test_identical_to_post_recon, test_identical_to_two_kernel_path, test_exact_variants, test_recon3, test_recon6); green with the source restored. Red with one kernel dropped from POST_EXCHANGE_KERNELS, and with a card class left out (the launched-kernels check); green restored. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…; the _mg / _ex blocks on the gfx11 card
The scan flags swin_block_c64_add_ex, swin_block_c96_add_ex (and their _nc forms)
and the two fused_merge functions the _mg blocks call, in swin.co, swin__mgx.co
and swin__f16wm.co: 216 / 300 / 32 (240 in f16wm for C = 64) matrix and exchange
instructions per function, all under ONE mask. Its origin is the entry test of
the loop over the output tiles, `for (mt = wave; mt < tiles; mt += 8)`: `wave` =
tid >> 5 is a vector value to the compiler, so it tests `tid < 32 * tiles` per
lane (v_cmpx with 0x1ff, 0x2ff, 0xc0, 0x100). The bound is a multiple of 32, a
wave32 holds 32 consecutive thread indices from a multiple of 32, so every lane of
a wave is on one side of it: the mask is full or empty at run time, and an empty
one skips the loop. The stores behind `if (tok[n] >= 0)` are after the matrix
chain, not around it. Verdict: harmless as launched (256 threads), not live.
No source change, on purpose. Making `wave` scalar (wg_uniform, as the persistent
blocks got) does clear the scan, but it adds spilled registers to kernels that
already spill: swin.co _ex_nc c64 / c96 37 -> 47, f16wm 4 -> 29, mgx 37 -> 47
(fused_merge shared with swin_mid kept its code with a template argument; checked
by building it). That is a speed cost for a mask that never happens, so the
pattern is pinned instead: test_exec_scan FUSED_GUARDED lists these functions with
their exact counts, one origin each, a v_cmpx with an immediate that is a multiple
of 32. swin_mid (swin__bmegahg.co, parked) is outside the pin and keeps its result.
Card (RX 7900 XT), new test_gpu_swin_fused (the first card runs of these kernels,
variants base, f16wm, mgx): 23 cases of enc1, enc2, dec1, dec2 with window offset
0 or 2, grids from 4x4 to 36x28 tokens (windows that stick out of the grid, a
workgroup of 8 windows only partly filled), fused kernels against the separate
ones: every output byte equal, no NaN left from the 0x7F fill; against the
float64 reference with the spec 09 limits of the stage tests (5 % different per
case where an output has 2000 values or more). 3 tests per variant, all pass.
Red with fused_merge's store changed to `tok[n] > 0` (swin.co rebuilt):
test_fused_kernels_give_the_bytes_of_the_separate_ones fails in 12 cases and the
reference test in 12; green with the source restored. FUSED_GUARDED red with a
count changed (216 -> 217) and red on the build that has the wg_uniform fix
("not flagged any more"); green restored. Not used by the Model M path on this card:
the C++ runtime has no gfx11 build (it would run the _ex kernels for dec1 and
dec2, WG_SWIN_FUSE_STAGES default); the stage runner's `fuse` geometry runs the
_mg ones.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ime accepts gfx1100
clean/runner/pipeline.py needs libwg_runtime.so, and the repository has none for
a gfx11 target, so this test skips (with the runtime's own message) at this head:
- build.py's gfx11 branch returns before the runtime step (its docs pass
--no-runtime); build.runtime_command compiles the unchanged sources fine;
- device.cpp:398 accepts gfx1200 / gfx1201 only ("no matching HIP device with
packaged gfx1200/gfx1201 kernels"), and load_module reads <arch>/<name> or
<arch>.<name> from the build folder.
Not changed here: that is the runtime's device policy, the maintainer's call.
Run in a scratch copy with exactly those three changes (nothing committed; synthetic
weights and frames only), RX 7900 XT, HIP, default kernels (swin.co, swin_bneck6 for
b0..b5, post_recon2_r_band + post_filter_c1_s4_band, separate dec1 / dec0):
- whole frame, median of 50, three runs: 64x40 -> 96x60 2.993 3.048 2.999 ms;
45x29 -> 67x43 3.008 2.999 2.995 ms; 1706x960 -> 2560x1440 17.761 18.018
17.891 ms. Output finite in all cases and frames. Per-stage times are not
exposed by Pipeline without the interop io (wg_rt_frame_io_timed); the stage
timings are those of the stage tests.
- stage by stage against the gfx11 lane emulator on the card's own stage inputs
(64x40 / 45x29 render): one-code flips 0.000 to 2.368 % per stage (enc1.merged
the highest), more than one step at most 0.651 %, b0..b5 chain 0.000 %.
Under the 5 % per case and 2 % bounds.
- output pixels against the exact (gfx12-arithmetic) emulator frame: 99.97 %
different (64x40, first frame); per-stage compounding on the random synthetic
network: enc2.merged 3.2 % of RMS, b5 8.0 %, output 28.3 %. Both the 5 % per
case and the 1 % overall bound are missed: recorded as expectedFailure.
Tests: test_gpu_pipeline (4 tests: 3 pass, 1 expected failure on the scratch
runtime; 4 skipped at this head). Red with the per-stage flip bound lowered to
0.001 % (9 failures); green restored.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ames of family.py device.cpp accepted gfx1200 and gfx1201 by two string comparisons, so a gfx11 card was refused at create. The gate now asks wgrt::target_family (clean/runtime/target_family.h), the exact-name map of clean/runner/family.py in C++: gfx1200 / gfx1201 -> gfx12, gfx1100 / gfx1101 -> gfx11, every other name refused (gfx1102, gfx1103, gfx115x, and gfx1170-1172, which are gfx11 names with the gfx12 matrix design, stay out). The refusal text names the four targets. Device gets arch(), the exact target whose code objects the device loads (empty for the emulator), for the runtime to derive the accumulator row table in the next change. Code objects are still looked up per exact target as <arch>.<name> or <arch>/<name> (never the unqualified file), so a build without gfx1100 objects fails closed with "missing gfx1100 code object". gfx12 devices take the same path as before. The D3D12 gates (rdna4.h, device_d3d12.cpp) are not touched. Whether players ever get a gfx11 build, and how that is gated, is the maintainer's decision; there is no option or environment variable for it here. Tests: - test_device_targets.cpp (Windows mock; run on Linux through a scratch windows.h shim and the mock built as libamdhip64.so, 27 checks): gfx1100 and gfx1101 now load their own gfx<arch>.<name> and report arch(); gfx1102, gfx1103, gfx1151, gfx1170, gfx1299, gfx11 and "unknown" are refused. The old expectation (gfx1100 refused) is flipped because this change is the policy change. - test_target_family.py (new, Linux and Windows): the C++ map equals family.py for every name, plus 16 refused names (decorated, partial, case, other families). Red on purpose, then restored: a gfx11 prefix match turned test_device_targets red (exit 7: gfx1102 accepted) and test_target_family red; dropping gfx1100 from the map turned both red (exit 3 and a map difference). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ce's target family The matrix instruction of gfx11 returns accumulator row 2 v + h in element v of lane half h, that of gfx12 row v + 8 h, and the packed weights follow the table (A3: packing.py D_ROW_LAYOUTS["gfx11"]). clean/runner/pipeline.py sets WgRtConfig.d_row from the build's target, but clean/runtime/interop never sets it, so a game's runtime always packed in gfx12 order and the gfx11 kernels would have read the wrong rows. The runtime now makes its device before it packs, and when the config's d_row is all zero it takes drow_gfx11() for a device whose arch() maps to gfx11 (target_family.h), else the gfx12 table as before. A d_row given in the config still wins. No device-less case changes: the emulator and D3D12 devices have no arch(), so they keep the gfx12 table, and on gfx12 the packed bytes are the ones of before. Creating the device is the only step that moved up; a device that cannot be made is still reported at the old place, after the model has loaded, with the same text. Tests: model_pack_dump takes an optional gfx11 argument, and test_model_pack_cpp packs all twelve stage blobs of two synthetic models with drow_gfx11 against packing.py's gfx11 table, byte for byte (the gfx11 and gfx12 blobs differ, so a wrong table cannot pass either test). Red on purpose, then restored: drow_gfx11 returning the gfx12 table fails it (seed 1, enc0: 4428 bytes differ). That the runtime picks the table from the device is checked on the card by the whole-frame test in a later change. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ch>.<name> next to the code objects build.py's gfx11 branch returned before the runtime step and refused --runtime-only, so no libwg_runtime.so existed for a gfx11 build. The runtime reads its code objects as <arch>.<name> (device.cpp, never the unqualified file), and a gfx12 build makes those copies as it embeds the files; a gfx11 build had neither. Unless --no-runtime or --sources is given (as before, neither builds a runtime), a gfx11 build now copies every code object to <arch>.<name> (link_arch_objects) and compiles the runtime with embed=False: no embedded table, since the gfx11 set is not the gfx12 list that write_embedded requires, and the runtime reads the files from the folder. --runtime-only does the same step alone for a gfx11 target; --motion-outres-only stays gfx12 only (new message: the gfx11 target has no motion code objects). gfx12 builds are untouched: runtime-only still embeds, and the guard's --no-runtime build is the same. The runtime's own sources are not changed here. clean/README.md follows. Not done: nothing embeds a gfx11 code-object set (which files a release would ship for gfx11, and whether it ships at all, is the maintainer's decision), and the Windows runtime_command (D3D12 interop sources) is not exercised for gfx11. Tests (test_build_targets, compiler replaced by a function that writes each output): both gfx11 names make the runtime with no -DWG_EMBEDDED_TABLE and no embedded call, one <arch>.<name> copy per code object with the same bytes and no other gfx* file; --no-runtime and --sources make neither; --runtime-only links the existing objects and does not embed for gfx11 but still embeds for gfx1201. Red on purpose, then restored: the runtime step switched off (2 failures), the copy removed (3 errors), gfx11 --runtime-only embedding (1), gfx12 --runtime-only no longer embedding (1). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… built for gfx1100 With the runtime's gate, the packing rows and the build of the previous changes, test_gpu_pipeline no longer skips on a gfx11 build that has its runtime: its 5 tests run on the RX 7900 XT through HIP, default kernels (swin.co, swin_bneck6 for b0..b5, post_recon2_r_band + post_filter_c1_s4_band, separate dec1 / dec0), synthetic weights and frames only. The docstring no longer says the repository lacks the runtime. New: test_the_runtime_picks_the_rows_of_the_cards_family. A game's interop never sets WgRtConfig.d_row, so the runtime must choose the gfx11 table itself. Pipeline takes d_row="runtime" (the config's table stays 0, the reference table of check_packing is the build's family), and the test checks that every stage blob equals packing.py's on the gfx11 table, that enc1's blob is not the gfx12 one, that the stage blobs equal those of a pipeline given the table, and that a frame of two new pipelines is the same bits (new ones: the cached pipelines have run frames and the frame counter is part of the result). Red on purpose, then restored: a runtime that ignores the device's family fails it with "runtime packing of enc0 differs from pack_weights". Result on the card (OK, expected failures=1): output finite in all cases and frames; stage by stage against the gfx11 lane emulator on the card's own inputs, one-code flips 0.000 to 2.368 % per stage (enc1.merged 45x29 the highest), more than one step at most 0.651 %, b0..b5 chain 0.000 %; the output pixels against the exact (gfx12-arithmetic) emulator frame are 99.97 % different at 64x40 (the expected failure, unchanged, synthetic network). Whole frame, median of 50, three runs: 64x40 -> 96x60 3.120 3.066 3.044 ms; 45x29 -> 67x43 3.073 3.182 3.169 ms; 1706x960 -> 2560x1440 18.664 18.822 19.757 ms (the card is shared with another project's jobs; the scratch run of the previous change gave 17.8 to 18.0 ms). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…11 (S1) wg_platform.hlsli takes WG_GFX (12 by default, unchanged; 11 for RDNA 3). Under 11: no FP8 matrix op, no FP8 convert and no _ags pack (wg_ags.hlsli hides the FP8 codes); wg_wmma_f16_k8 is one raw F16 instruction on fragments replicated across the half-waves, wg_wmma_f16_k16 chains two of them (a K=16 step), wg_wmma_f16 keeps the gfx12-shaped operands of HIP's helper on top of them, wg_d_row is the 2i+h row table, SW_UNPACK_AGS is 0, and the clamp form of the GELU LUT converts with f32tof16 (RADV rounds a (half) cast to nearest even). build_dx12.py --gfx 11 defines WG_GFX=11, writes <out>/dx12-gfx11/ with a "gfx": 11 manifest key, builds only declarations with gfx=11 (default gfx=12) and skips RGA. Tests: test_build_gfx.py (8, no tools needed) and test_gfx11_helpers.py: headless on the card under Wine (vkd3d-proton) against numpy and a HIP gfx1100 kernel, bit for bit for the raw op, the K=16 chain and wg_wmma_f16; convert is RNE; xor16; row table; rounding modes. They skip without the tools named in the module doc. Red on purpose then restored: row table i+8h, K=16 chain without its second op, swapped K halves, xor16 skipping a lane, swapped convert dwords, FP8 code visible on gfx11, build without the WG_GFX define. Guard: the 46 gfx12 DXIL files are identical to the baseline (manifest too). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…sts, PSO sweep (S3) wine_headless_run.sh runs a Windows program under Wine with a null-graphics prefix, a private runtime folder and the GPU lock, and refuses anything that could open a window (a prefix with a graphics driver, /tmp or /run/user scratch). build_kernel_replay.py cross-builds kernel_replay.exe, pso_sweep.exe and gfx11_probe_host.exe with llvm-mingw. pso_sweep.exe creates each kernel's pipeline with the runtime's buffer or direct root signature (the runtime itself is not used: its device refuses a card that is not RDNA 4); pso_sweep.py runs it over a built folder, records a driver crash per kernel and classifies each DXIL by FP8 channel codes (dxc -dumpbin): a PSO with an FP8 op must fail on RDNA 3, any other must be created. Result on the RX 7900 XT (vkd3d-proton, extended emulation): the 46 gfx12 DXIL files give 23 created (all without FP8), 17 failed with 0x80070057 and 5 crashed the driver (the swin c96 add_* blocks: a page fault inside the driver, reproducible alone), 1 tex kernel not tried; none unexpected. The 8 gfx11 helper kernels of test_gfx11_helpers all create. kernel_replay.exe builds and starts but its device refuses the card until the family gate opens. test_wine_headless_run.py: 4 tests; red on purpose: scratch check removed, null-graphics check removed. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…out in the CPU model (S2) native_driver_probe_f16 (gfx=11 only) runs D = A x B + C with FP16 operands as a K=16 step of two chained instructions on the replicated fragments of the gfx11 layout. The gfx12 kernel native_driver_probe is untouched (its DXIL is byte for byte the same, and it is not built for gfx11: it has an FP8 instruction). wmma_probe.h gets namespace gfx11: buffer layout, fill, expected results (the same matrices, rows 2v+h) and a compare with a tolerance of 1/4096, because the card's F16 accumulation differs from the exact sum in the last bit in 57 of 256 results (measured; equal to HIP's instruction). test_wmma_probe.cpp: 20 new checks (the right layout passes, six wrong hardware variants fail, replicated-half variants pass, all lane-pair swaps of A, B and C fail exactly when they change what the instruction reads, all element-pair swaps fail, the tolerance accepts last-bit differences and reports 2e-3 and NaN); the gfx12 checks are unchanged. Red on purpose: fragment 1 given the K of fragment 0 (6 failures), d_entry rows v+8h (1). test_gfx11_probe.py (3 tests): builds the kernel with build_dx12.py --gfx 11 and runs it on the card under Wine with wmma_probe_dump's operands: results within the tolerance; the gfx12 buffer fails. Red on purpose: the kernel with one instruction instead of two. Guard: the 46 gfx12 DXIL files are identical to the baseline. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…y normal build (S4) A compile-time macro, default off and never defined by clean/runner/build.py, lets the native D3D12 device, the executor choice and the Wine log line take an RX 7900 series board (PCI 1002:744C, Navi 31, gfx1100; the id read on the development card, shared by the XT, XTX and GRE; no other RDNA 3 id, 7448 included). The mechanism is one of the options listed for the maintainer in the RDNA 3 plan (a macro and a local build); the maintainer may prefer another (environment variable, ini key, a PCI id rule in the family function, or the gate left closed in main), and this change does not choose among them. It exists so that the D3D12 kernels for gfx11 can be built and tested before that decision. Without the macro nothing a player runs changes: rdna4_identity is untouched (the Ray Reconstruction runtime still asks only it), dev_gfx11_identity is constant false, dx12_folder answers "dx12", the device reports no target, the probe is the gfx12 one, and the order of the manifest read and the device is as before. With the macro on: - rdna4.h: kDevGfx11, gfx11_device_id, dev_gfx11_identity (same rule as rdna4_identity: the driver's ids, else the Wine fallback's), dx12_folder(arch), dev_gfx11_arch(); the Wine line and describe_identity know a taken board. - executor_select.h: the default rule takes the board (D3D12 executor). - device_d3d12.cpp: both identity gates take it; the device reports target gfx1100, reads <build>/dx12-gfx11/manifest.json after the device (the family comes from the adapter) and runs native_driver_probe_f16 with wmma_probe::gfx11 (F16 only, compared within 1/4096). - runtime.cpp: the device is made before any manifest is read, the manifest of a build is read through dx12_manifest(), the accumulator row table comes from the device's target as for HIP (already so), and the native-direct check accepts the gfx11 table. The tolerance: the card's F16 accumulation is not exact (57 of 256 probe results differ from the exact FP32 sum, by at most 2^-15 here, equal to HIP's instruction). 1/4096 is two ulps of the largest expected value (1331) and 1/2500 of the smallest gap between distinct expected values (0.607), so a wrong layout cannot pass. Tests: test_dev_gfx11.cpp (28 checks without the macro, 29 with; both builds pass) and tests/test_dev_gfx11.py (builds both, 6 source pins incl. that no build or release file mentions the macro). The existing test_executor_select.cpp: 234 pass without the macro; with it exactly the 12 cases that assert the RX 7900 XTX id (744C) is refused fail, as they must. On the card, headless under Wine with the system DXGI of DXVK in a scratch prefix (the Wine fallback ids, as a player's Proton gives): native_probe_test (new target of build_kernel_replay.py; the borrowed-device path) is refused without the macro and passes the F16 probe with it; WG_TEST_DRIVER_PROBE_FAIL=1 fails at float 7; kernel_replay runs native_driver_probe_f16 as a chain, 256 results within tolerance. Red on purpose, then restored: 11 host mutations and 6 Wine mutations (see the test files' pins). Guard: the 46 gfx12 DXIL files and the 42 HIP objects are unchanged (no kernel touched). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…fx12 kernels on RDNA 3, with a rounding check
Inside the WG_DEV_GFX11 opening a second developer-only choice lets a gfx1100 board run
the UNCHANGED gfx12 kernel set (dx12) through dxil-spirv's emulation of the FP8 matrix
operations in the raw-fragment path (DXIL_SPIRV_CONFIG=wmma_rdna3_raw_fp8 in a patched
vkd3d-proton). The mechanism is the environment variable WG_DEV_GFX11_SHORTCUT=1, read
only in a build made with the macro (rdna4.h: without it dev_gfx11_shortcut() is constant
false and the library never asks the environment). It is a placeholder: the maintainer
decides the mechanism later (ini key, option, a PCI id rule, or no shortcut at all), and
shipping a patched dxil-spirv is a product decision this change does not make.
Without the macro nothing changes for a player. With the macro and the variable unset
nothing changes either (the S4 behaviour). With both, the device reports no target (the
gfx12 folder dx12 and the gfx12 accumulator table, as on RDNA 4; the emulation keeps the
gfx12 accumulator layout), and:
- the borrowed-device probe is the gfx12 one (native_driver_probe, FP16 and emulated
FP8), compared within wmma_probe::gfx11::kTolerance: the emulation multiplies in the
FP16 instruction, whose accumulation is not exact on gfx1100 (same reasoning as S4).
- a rounding check runs at device creation, for owned and borrowed devices: the
kernels round to FP16 through an F32 -> F16 -> F32 pair of the matrix conversion, and
the driver's compiler folds such a pair away unless the conversions are exact (found
with the shortcut probe: a fold gives encoder features one FP8 step off). The kernel
native_round_probe (gfx=11 only, so the gfx12 set keeps its 46 files) converts 256
inputs that all have bits below FP16's precision, including exact ties of both
parities; the device compares bit for bit with the nearest-even result computed on the
CPU (wmma_probe.h, round16) and refuses with a log line otherwise. The kernel is read
from <build>/dx12-gfx11/manifest.json (read_manifest takes an optional kernel name).
Tests: test_shortcut_probe.cpp (27 checks, built with and without the macro) and
tests/test_dev_gfx11_shortcut.py (both builds plus 5 source pins). On the card, headless
under Wine: the shortcut passes the probe and the rounding check; with
DXIL_SPIRV_CONFIG unset the compiler folds the pair and the device is refused ("the
driver does not round F32 -> F16 -> F32"); WG_TEST_ROUNDING_FAIL=1 gives the same
refusal. Red on purpose, then restored: 8 mutations (see the test files' pins).
Guard: the 46 gfx12 DXIL files are unchanged.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ess Wine, compared with the HIP whole frame (S12) frame_host.cpp is a small Windows host (cross-built with the other replay hosts by build_kernel_replay.py) that loads wronggpu_runtime.dll, creates the native D3D12 backend (backend 3) on a synthetic raw-tensor model, runs frames from input files, dumps the named buffers of each frame and times frames with the runtime's own timestamps. Nothing reads a model.wgcm. It exists because the whole-frame machinery (compare.py, time_frames.py) drives the runtime library from Python on Windows, and a headless run under Wine on Linux has no Python on the Windows side. frame_parity.py prepares the model, the inputs and the host's spec (the same synthetic frames as compare.py), runs the HIP or lane-emulator whole frame natively through pipeline.py, and compares every named buffer and the output image of two runs: bytes that differ per buffer (buffers that a fusion leaves unwritten, stale ones, and the decoder's other format are listed and not counted, as in compare.py), the output in FP16 steps and relative RMS, and the GPU times. It measures; it is not a gate, since the output standard of an RDNA 3 route is the maintainer's decision. Checked: a run against itself gives 0 differing bytes and byte-equal output; the D3D12 shortcut frames at 64x40, 853x481 and 1706x960 run end to end (see the S12 report). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… text, and a guard checks it The RDNA 3 changes in swin.hip, encoder.hip and wg_platform.h leave the gfx12 program the same, but not its text: the gfx11 operand types, fragment loads, store forms and wave id helper were written for every target, and the patch merge and expand stores gained braces. A different compiler can schedule different text differently: on the RDNA 3 line, ROCm 10.1 (clang 24) moved instructions in swin.co and swin__bmegahg.co for one restructured store, although the program was unchanged. "The same instructions under today's compiler" is not a promise RDNA 4 can rely on; the text is. Every gfx11 form now sits under WG_SWIN_G11 (gfx11 or the emulator) with main's earlier text in the #else. In wg_platform.h the v4h / v16h types and wg_xchg16_u exist only for gfx11 and the emulator; encoder.hip gets the same treatment for its gfx11 store. clean/tests/check_gfx12_source_unchanged.py preprocesses every shipped HIP compile for gfx1201 (each tree's own build.py with its compiler calls recorded, FG's three objects, every other clean/**/*.hip) and requires the text to match the base tree's token for token; its unit tests plant an edit to prove it can fail. dx12/tests/gfx11_hip_ref.hip, a host program of the gfx11 helper tests that no gfx12 build compiles, is named as unbuildable in the test and as a new form on the command line. Verified (ROCm 10.1, clang 24; only HIP kernel sources and the guard changed, no D3D12 or runtime file): - guard against the base on main (91436c4): 160 forms, 0 failed. The same guard with the unfixed branch head (0cd1a91) as the tree: 160 of 160 failed. - gfx1201 objects (M, K, L, FG: 42) of the fixed tree and of 91436c4: byte-identical 42 of 42. (The unfixed head 0cd1a91 also gave 42 of 42 identical objects under 10.1: the source differed, the code objects did not.) - gfx1100 objects (--variants f16wm,mgx: 11) before and after: byte-identical 11 of 11. - guard unit tests 14 OK; the emulator tests test_kernels_emu_gfx11, test_e4_gfx11_emu and test_kernels_emu: 36 OK. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
….hlsli and wg_ags.hlsli are main's again On main aa85515, model K's softmax self-check records kSourceDigest, a SHA-256 over the shader headers the softmax kernels include (wg_platform.hlsli and wg_ags.hlsli among them). S1's gfx11 edits of those two files changed the digest and build_dx12.py refused to build. Moved: everything gfx11 in wg_platform.hlsli (WG_GFX default and checks, wg_in_upper_half, wg_xor16_4, wg_wmma_f16_k8 / _k16, the gfx11 wg_wmma_f16, wg_d_row, the gfx12 wg_d_row) and in wg_ags.hlsli (the WG_GFX default and the #if hiding the FP8 codes) into the new hlsl/wg_gfx11.hlsli. A gfx11 kernel includes it instead of wg_platform.hlsli (it includes that one itself); gfx12 builds of it only include the platform layer. Under WG_GFX 11 it renames the platform layer's gfx12-shaped wg_wmma_f16 while including it, and hides the FP8 identifiers again with #undef / #define (a use does not compile, as before). native_game_helpers.hlsl, tests/gfx11_helpers.hlsl and the compile check include it. wg_gelu_lut.hlsli, swin_common.hlsli and build_dx12.py are not in the digest and stay. Verified: smx_selfcheck_digest.digest() equals the committed header again; the two gfx11 probe DXIL files are byte-identical to the ones built before the move; test_build_gfx (8) and TestGfx11Compile (4, including the FP8-is-hidden negative controls) pass; the card tests skip (no GPU in this slice). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… of compiling it for gfx1201
Since the gfx12 source of every HIP kernel is main's text again, the gfx11 exchange wg_xchg16_u exists only for gfx11
and the emulator. test_exchange_is_one_permlane_on_gfx11_and_bpermute_on_gfx12 still compiled it for gfx1201 and so
failed ("use of undeclared identifier 'wg_xchg16_u'"). Its gfx12 half now checks what the design promises: wg_xor16 on
gfx1201 is the one ds_bpermute exchange of main's text (no permlane), and wg_xchg16_u does not compile there. The gfx11
half is unchanged.
Verified (ROCm 10.1, clang 24): the test is red before (the old test fails on the parent commit) and green after; with
the wg_platform.h of the commit before the gfx12-inert fix (wg_xchg16_u for every target) the new test fails, restored
it passes. Without a ROCm clang the whole module skips, as on CI's Windows runners.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…Wine runner test skips off Linux build_dx12.py's digest label used os.path.relpath, which raises ValueError when a source (a test's temp folder on C:) and the repository (D:) are on different drives. A source outside the repository is now labelled ext/<file name>. A new test in test_build_gfx.py simulates the cross-drive error (red before the fix, green after). test_wine_headless_run.py runs a bash script, so it is skipped unless on Linux. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…U emulation of the kernel text wg_gfx11.hlsli gets the operand layer of the native RDNA 3 form (WG_GFX 11 only; the gfx12 branch and every RDNA 4 kernel are unchanged): - wg_h16 (16 F16 values per lane: K 0..7 / 8..15 of a row or column, both half-waves alike) and the K = 16 opcode WG_AGS_F16_F32_K16 (0x7e, outside AMD's opcode list; needs the translator's K = 16 patch, a stock translator refuses it): wg_wmma_f16_k16n / wg_mma16, one instruction per K = 16 step, K in index order; - the row map wg_a_row(p) = 8 (p & 1) + p / 2: an A operand loaded with it leaves the accumulator in the gfx12 layout, so the RDNA 4 weight blob and every step between products stay as they are; - wg_ldA16 / wg_ldB16 (A row-mapped, B column, straight from a gfx12-order tile: no exchange), wg_b16 (a B operand from the lane's own gfx12-shape fragment: one half-wave exchange, reusable), wg_rows_to_gfx12 (the FP16 result of a product with unmapped rows into the gfx12 fragment order: one exchange of 2 dwords); - WG_A_ROW_MUT: a wrong table for negative controls only. clean/dx12/emu (test-only, never shipped): hlsl_emu.py compiles a D3D12 kernel file's declarations as C++ against hlsl_emu.h (stands in for wg_platform.hlsli / wg_ags.hlsli; the other headers are the real ones) and runs them on the lane emulator of the HIP kernels (hlsl_emu_bridge.cpp): gfx12 matrix model for WG_GFX 12, the measured gfx11 F16 instruction for WG_GFX 11. HlslEmu is a device with the stage runner's interface (arena offsets as pointers, launch arguments flattened into root constants). tests/gfx11_k16_probe.hlsl + test_gfx11_native.py: one-hot probes (512 cases: every A row and K position, every B K position and column) of ldA16 / ldB16, b16, the accumulator start and the v^T form, exact equality on the emulated instruction; the K = 16 channel equals the chained K = 8 form and HIP's exchanging wrapper there. Negative controls: WG_A_ROW_MUT is caught on exactly the moved rows; the v^T form without wg_rows_to_gfx12 fails. The translator checks are in the same file and skip until a patched dxil-spirv is configured. Verified (CPU): test_gfx11_native 11 tests, 8 run, 3 skipped (translator); test_build_gfx, test_kernel_sets, test_k_smx_selfcheck_digest, test_gfx11_helpers 38 OK (20 skipped: card); the K softmax digest is unchanged (smx_selfcheck_digest.digest() equals k_smx_selfcheck.h); the RDNA 4 DXIL guard is in the report. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…11 only kernels/k_block_g11.hlsl declares the narrow stages of k_block.hlsl (e0, e1, e2, d2, d1 one-window; e0, e1, d1, d0 two-window incl. the hidden-history and compact-head d0; the three __rodd forms) gfx=11, named <port>__g11, on hlsl/k_block_g11.hlsli: k_block.hlsli's template with every product as ONE K = 16 instruction on operands the kernel lays out (wg_gfx11.hlsli S1 layer). Same arguments, launches and weight blob (RDNA 4 / d_row gfx12) as the ports; every accumulator comes out in the gfx12 layout, so all code between products is the ports': - A from memory or LDS (weights, K tiles, VT tiles): row-mapped 16-half loads, no exchange; - B from memory or LDS (embed / expand inputs, Wv, the fixed mix table, the merge's x staging): direct loads; - B made in registers (n, q, z, a, the hidden tile, x2 for the head, softmax weights): one exchange, reused for every output tile (n's B form is also v^T's A operand; KN_NB_HOIST 0 remakes it per head); - v^T's result goes through wg_rows_to_gfx12 before LDS. Every product sums K in index order, so a stage should give the shortcut's bits (GPU test). Names carry __g11 because main's softmax digest scans every declaration in clean/k/dx12/kernels for the self-check names; the runtime does not launch these yet (open question in the report). Verified (CPU): build_dx12.py --gfx 11 --model-k compiles all 13 (DXC 1.10, arithmetic-mode rewrite, dxv); test_k_native_g11_emu (clean/dx12/emu): 39 stage outputs over 3 size sets (odd maps, partial windows, maps smaller than a window) and 3 offsets against NumPy within spec 49's stage limits (worst max 2.8e-3 of RMS) and byte for byte equal to the RDNA 4 ports emulated the same way; d0 full head, compact head and hidden history equal to the ports' bytes; the __rodd forms equal; knobs KN_NB_HOIST 0, KB_GRPLATE, KB_W2LATE, KB_X1LDS keep the bytes. Negative controls: WG_A_ROW_MUT puts every stage > 10x the limit; dropping wg_rows_to_gfx12 on purpose made the stage test fail (e0 0.53 of RMS) and it passed again after restoring. Digest unchanged; test_build_gfx / test_kernel_sets / test_k_smx_selfcheck_digest / test_k_plan_bridge / test_k_port_check_nan OK. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…same bytes (GPU tool, CPU dry run) clean/k/dx12/k_native_check.py (developer tool of the native RDNA 3 work): --prepare writes, from the stage runner's plans (k_plan_bridge.chain_lines), two kernel_replay case files over the same input files: the RDNA 4 ports (run through the shortcut) and the native __g11 kernels, 13 kernels x 6 block-level sizes x 5 even offsets per seed, special values (infinity, +-65504, -0, subnormals, NaN) in two of every three cases, NumPy references for the others, one negative control (a native chain with half its grid). --emulate runs both files through clean/dx12/emu (a CPU dry run of the files and the comparison); --compare checks native = shortcut byte for byte over the defined bytes (sign-only NaN differences counted apart), the 64-byte slack, spec 49's stage limits against NumPy, and the negative control. --timing / --time N write the 1440p Quality levels with kernel_replay's `time` for the timing round. native_helpers.hlsl: native_fill is declared gfx=11,12. The D3D12 device's memset launches it, so kernel_replay (and the runtime) in the native gfx11 mode needs it in <build>/dx12-gfx11. The gfx set is not part of the source digest and the kernel is unchanged: the RDNA 4 DXIL and manifest stay byte-identical (guard in the report). Verified (CPU): --prepare --emulate --compare with seed 0: 78 cases, 138 outputs, 0 differing, 0 slack writes, worst 7.2e-5 mean / 3.2e-3 max of RMS, negative control caught; one flipped byte in a native output makes --compare fail (exit 1) and passes again restored. test_k_native_check (a subset: 12 cases) OK incl. that flip; test_build_gfx / test_kernel_sets OK; build_dx12.py --gfx 11 --model-k now writes 16 kernels incl. native_fill. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…kipped without a card) test_gfx11_native.py gets TestNativeOnCard on the card harness of test_gfx11_helpers.py (gfx11_probe_host under the headless Wine runner; WG_GFX11_VKD3D must hold the K = 16 patched vkd3d-proton): k16n equals the chained K = 8 form bit for bit on random data with cancellation and on data with infinities, NaNs and signed zeros in both operands, k16n matches HIP's v_wmma_f32_16x16x16_f16 (WG_GFX11_HIP_RUN), the one-hot probes of nat_ldA / nat_b16 / nat_vt, and nat_ldA equals the chain's bits on random data (same K order). The emulator and the card tests share one arena layout (arena_tiles / arena_channel). Verified (CPU): test_gfx11_native 16 tests OK, 8 skipped without a card (the 3 translator checks run with the patched and the stock dxil-spirv: one multiply per K = 16 step, no lane exchange, the stock tool refuses the opcode); test_gfx11_helpers' DXC compile checks (TestGfx11Compile, 4) OK with WG_GFX11_WORK set. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…nd; files are closed
The unit subset used sizes and an order in which dropping the merge store's `my < mh` bound
(k_block_g11.hlsli) left every unit test green; only the full dry run caught it. The subset now
takes the first three sizes of the full list with seed 0, which places them at offsets that
expose a write into the slack. Verified: with the bound removed the test fails ("native wrote
into the slack", 4 cases); restored, it passes. k_native_check.py and its test close every file
(no ResourceWarning under -X dev -W error::ResourceWarning); the full seed-0 dry run still passes
(78 cases, 0 differ, 0 slack).
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
A gfx11 board's softmax stage hashes differ from the gfx12 HIP reference, so the create-time check refuses Model K there (MODEL-K-RDNA3.md section 4). The existing rodd_unchecked hook takes the round-to-odd forms; this one takes the mix form, which is the default path, so a whole frame can be timed on it. The hook is compiled only with WG_DEV_DIAGNOSTICS, outside the self-check digest regions (the digest and k_smx_selfcheck.h are untouched). Verified: test_k_smx_selfcheck_digest, test_dev_gfx11*, release runtime disassembly identical to the base commit. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…_g11) in a shortcut run Option (a) of K-NATIVE-1.md 5.1. In a WG_DEV_GFX11 build, WG_DEV_GFX11_NATIVE_K=1 on top of WG_DEV_GFX11_SHORTCUT=1 makes the D3D12 device load the __g11 entries of the dx12-gfx11 manifest next to the gfx12 set, and launch/prepare run <name>__g11 instead of <name> wherever the manifest has it. Every other kernel (wide stages, I/O, exposure, warm, helpers: no matrix work or no native source yet) still runs as its gfx12 kernel through the shortcut, so a frame is mixed. The variable without the shortcut makes create fail. One build times both modes: the same run without the variable is the shortcut. Profile and launch-trace rows name the kernel that ran. Everything is under WG_DEV_GFX11: the release runtime's disassembly, symbols and strings are identical to the parent commit's. Tests: the lookup picks __g11 only in a WG_DEV_GFX11 build and only when the manifest has it (host build with and without the macro, two negative controls), source pins, and each of the 13 native declarations has its port's code object, block, wave and arguments. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…e chosen per kernel GPU timing found native faster on some stages and slower on others. The value "1" still means every kernel the manifest has a __g11 form of; a comma list of port names (k_stage2_d0_hp,k_stage2_d1,k_stage_d2,k_stage_e2) launches __g11 only for those ports (whole names) and runs the rest through the shortcut; empty or "0" is off. Same gate as before: read only in a WG_DEV_GFX11 build, needs the shortcut. Tests: the list form switches only listed ports (negative controls: a port not in the list stays; a mutation that selects every name is caught by the host test), prefixes do not match, stray commas select nothing. Release runtime: disassembly and symbol list identical to the parent of this work, no new strings. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…uration The patched translator now refuses opcode 0x7e without DXIL_SPIRV_CONFIG=wmma_rdna3_raw_fp8, since only that configuration keeps the kernels' F32 -> F16 -> F32 rounding pairs unfolded. The count test runs with the configuration only; a new test checks the refusal without it. Verified: green with the fixed translator; with the refusal removed from the translator the new test fails, restored it passes. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…11_NATIVE_K Review finding (Kimi, FIX-PASS): the module docstring only mentioned =1. Docstring only; the tests are unchanged and pass. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ult) KN_NB_HOIST 2, KN_VSTREAM, KN_ZSTREAM, KN_AB 1/2, KN_XPIN, KN_HEAD_ROLL, KN_MERGE_ROLL / KN_MERGE_STEP in k_block_g11.hlsli: where operands are loaded or made, never an operand value or a sum's order. With every knob at its default the 16 gfx11 DXIL files are byte-identical to the wip/k-native-f3r2 build (scratch-kenc/v/dflt). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… (off by default) The template's 16-byte loads and stores go through kn_ld128 / kn_st8 / kn_ldA16 / kn_ldB16. With KN_A16 1 the address has its low 4 bits cleared: every address these kernels form is a multiple of 16 bytes from a 256-byte aligned buffer or weight section, so the bytes are the same and the driver issues global_load_b128 instead of four global_load_b32 (k_stage2_e0: 368 -> 92 vector loads). Off: the 16 gfx11 DXIL files are byte-identical to the parent's build. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…nobs KN_A16 on all 13 native kernels; e2 and e0 two-window KN_ZSTREAM, d2 and d0 KB_GRPLATE (192 VGPRs, 8 waves), e1 two-window KN_XPIN + KB_GRPLATE + KB_W2LATE (240 VGPRs, 6 waves, was 256 / 5), d1 two-window KB_GRPLATE + KN_VSTREAM + KN_NB_HOIST 2 (51 spilled VGPRs, was 106). Stage times on the RX 7900 XT, median of 6 interleaved rounds vs the parent native kernels: e0 -6.5 %, e1 -6.0 %, e2 -6.0 %, d2 -22 %, d0 -5.6 %, d1 -5.0 %. The emulation test and k_native_check.py --emulate build each native kernel as its own library (the declarations' defines differ now); the knob test also switches the new knobs. Checked: emulation of every stage form byte-identical to the RDNA 4 ports; the test goes red with a 32-byte address mask and green again. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…16-byte aligned The native RDNA 3 kernels' KN_A16 form clears the low 4 bits of every 16-byte arena address, which is the requested address only when every KBlkParams pointer is 16-byte aligned (review of K-ENC, fix 1). In builds with WG_DEV_GFX11 or WG_DEV_DIAGNOSTICS, create checks the parameters block_params gives every stage from the allocated buffers and weight sections, and launch checks every k_stage* kernel's parameter block (the softmax self-check path included); both fail naming the pointer. The checks sit outside the self-check digest regions (block_params itself is inside one, so it is unchanged and the digest with it). Release build: llvm-mingw DLL of this tree vs wip/k-enc-1, 0 differing disassembly lines of 367,575, identical symbol and string lists, 2 differing bytes (the PE timestamp and its copy); the check's message and symbol only in the developer DLL. Test: clean/tests/test_k_blk_align.py builds test_k_blk_align.cpp against KBlkParams as runtime_k.cpp declares it (every pointer field + 1, 4, 8, 12 refused naming it), pins the wiring; red with the check removed from the header, with a field dropped, and with the create check removed. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
The rolled merge loop has no tail and its per-iteration arrays hold 4 tiles: 0 never advanced, 3 read past the merge's K tiles, 8 past the arrays (review of K-ENC, fix 2). No staged declaration rolls the merge: the 16 gfx11 DXIL files are byte-identical to wip/k-enc-1's (only the manifest's source_digest provenance changes with the header text). Test: test_merge_step_is_1_2_or_4 (emulation build) refuses 0, 3 and 8 with the message and builds 1, 2 and 4; red with the guard removed. DXC (build_dx12.py --define KN_MERGE_STEP=3) also stops with the message. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…hantoslayer21#225's no-name-rule pin clean/tests/test_executor_select.py (CreateOrder.test_no_name_rule_left, phantoslayer21#225) requires that clean/runtime/rdna4.h contains no find( outside comments, so that no GPU name rule can come back into the identity code. The parser of the WG_DEV_GFX11_NATIVE_K list (developer builds only) used std::string::find to split at commas and broke that pin. It now walks the characters; the entries, the exact-name match and the treatment of stray commas are unchanged. Verified: test_executor_select and test_dev_gfx11_native_k (the host build of the list cases with and without WG_DEV_GFX11): 20 tests OK. With the separator changed to ';' on purpose the list cases fail ("a listed port switches", "the first and the last entry switch", "a list selects only its names"); restored, green. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
j0nsh1n
force-pushed
the
rdna3/k-native
branch
from
November 18, 2026 02:16
5d3ddbe to
d877391
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, for visibility; not asking for a merge. Model K on RDNA 3 (RX 7900 XT, gfx1100) under Proton: the measured state, native RDNA 3 forms of K's narrow network stages (and faster encoders for the two-window stages) behind the existing developer gate, and one timing observation you may want to know about. Nothing here changes what a player gets: everything RDNA 3 sits behind
WG_DEV_GFX11(off in every normal build), and RDNA 4 is unchanged (see "RDNA 4" below). Stacked on #259 atc3f6b75(rebased on 0.6.5,2b31cbc); since #259's branch lives on my fork, this PR targets main and GitHub also lists #259's commits: this PR's own are the 16 from9d176e7on. Open decisions are listed at the end.Testing still to do on my side: nothing has been re-run on an RX 7900 XT since the rebase onto 0.6.5 (the card is out of my machine right now). Before this is more than a draft I'll re-run the card checks (stage-level native vs shortcut, the pipeline sweep, the whole-frame byte comparison) and an in-game session on the rebased code. Details under "Not re-run after the rebase on 0.6.5".
Where Model K stands on the RX 7900 XT (1706x960 -> 2560x1440, synthetic model
random_network_params(11), headless Wine, a local vkd3d-proton build withenable_extended_emulationplus the dxil-spirv change of #259; medians of 300-540 frames; measured before the rebase onto 0.6.5)k_stage2_e1addedk_approx=a1,a2a2refusesk_head=full)In game (Cyberpunk 2077 under Proton, 1706x960 -> 2560x1440, a real converted Model K, max settings, no RT, runtime GPU timestamps, median of 300-frame windows of one session each; developer builds, not a controlled benchmark):
k_approx=a1Image quality is not settled: on both cards K shows flicker on thin bright detail, some ghosting and occasional single-pixel specks, and it looks the same on both. Offline, captured real frames replayed through the runtime matched the in-game output byte for byte on the 7900 XT, and the NumPy reference of K reproduced the same specks at the same pixels, so it looks like K's own response to this game's inputs rather than a porting bug.
k_approx=a1looked clearly worse in game than the exact path. The game route records into the game's DIRECT queue, so the COMPUTE-queue observation below does not apply there.What this changes and why
The shortcut runs K's RDNA 4 kernels on RDNA 3 by translating every 16x16x16 F16 matrix step, including an exchange between the two half-waves per call, because RDNA 3 lays out matrix operands differently. A native form can lay out its operands once and keep them, so it needs about 8 times fewer exchanges (e2: 74 against 578).
In order (16 commits on #259 at
c3f6b75; all signed off). The move of the gfx11 helpers intowg_gfx11.hlsli(which keeps main's two headers and the #254 softmax digest untouched) is in #259 itself now:9d176e7Native K = 16 operand helpers for gfx11 (16 halves per lane, laid out by the kernel) and a CPU emulation that runs the D3D12 kernel text itself, with one-hot probes.1ad1729One template that gives 13 gfx11 kernels<port>__g11for K's narrow stages (e0/e1/e2/d2/d1, the two-window forms with both heads, the three round-to-odd forms). An A-row map makes every result come out in the RDNA 4 layout, so the RDNA 4 weight blob is used unchanged. Each product sums K in the shortcut's order.1f151a9,c2f62a5,ca1822fk_native_check.py(native against shortcut on the same input bytes, GPU tool with a CPU dry run) and card tests that skip without a card.ca15d25A diagnostics-onlyk_smx_test=mix_uncheckedhook (see decision 2).5b5ae65,d07ab51WG_DEV_GFX11_NATIVE_K(read only in aWG_DEV_GFX11build, on top ofWG_DEV_GFX11_SHORTCUT=1):1launches every__g11kernel thedx12-gfx11manifest has, a comma list launches only those; the rest of the frame stays on the shortcut.0c0d25aThe native K = 16 opcode is accepted by the translator only with the shortcut's configuration (it is the one that keeps the F32 -> F16 -> F32 rounding pairs unfolded).56fb35fDocs: the native K test's docstring names the list form.d007c10,1a6b3f6,66b067e,02382c0Faster encoders for the native stages: register knobs for the two-window encoders (all off by default),KN_A16(16-byte arena accesses as one 128-bit load or store, off by default), each stage kernel declares its measured register and load knobs, andKN_MERGE_STEPis limited to 1, 2 or 4 (a preprocessor error otherwise).k_stage2_e0goes from 240 to 192 VGPRs (8 waves, the shortcut's) andk_stage2_e1from 256 to 240 (6 waves, the shortcut's); stage times are 5 to 22 % lower on all 13 native kernels, and the whole frame is 0.069 ms lower withk_stage2_e1added to the list. Output is byte-identical to the shortcut on every stage output and on all 216 whole-frame files.e180608Developer builds refuse Model K stage pointers that are not 16-byte aligned (the 128-bit accesses need them).5d3ddbeNew in this rebase: the parser of theWG_DEV_GFX11_NATIVE_Klist no longer usesstd::string::find, because 0.6.5's R1: no in-game route to HIP; GPU identity from the driver; other cards use the front end's upscaler #225 test requires thatrdna4.hhas nofind(outside comments. Entries, exact-name matching and stray-comma handling are unchanged.The 15 commits before the last have the same patches as before the rebase on 0.6.5 (
git range-diffshows=for each). The translator side (a developer-only opcode for "16 pre-laid-out halves, one K = 16 step") lives in the same local dxil-spirv change as #259, not in this repo.RDNA 4
Source text equal to main's under the guard, and objects equal under ROCm 10.1, built on
2b31cbc:check_gfx12_source_unchanged.pyagainst main: 0 failed (172 forms preprocessed for gfx1201);k_smx_selfcheck.hare unchanged, and its test passes;test_k_runtime_gpu(20 tests: Model K through the per-frame runtime on main's RDNA 4 kernels, against the reference on the same inputs) passes on this tree and on main, so both give the same bits.WG_DEV_*) built from the last commit is identical in disassembly, symbols and strings to one built from RDNA 3 (RX 7900 XT): Model M in HIP on gfx1100 and a developer-only D3D12 route under Proton (draft) #259's header-move commit (f03c2c2andc3f6b75after it only change tests and a path label for sources outside the repo). Compared with main's release runtime it is not identical, because RDNA 3 (RX 7900 XT): Model M in HIP on gfx1100 and a developer-only D3D12 route under Proton (draft) #259 adds the (off) gfx11 code to it.Please repeat the guard on your toolchain.
How it was tested
test_gfx11_platform,test_the_faster_quantizer_is_what_spills, fails on RDNA 3 (RX 7900 XT): Model M in HIP on gfx1100 and a developer-only D3D12 route under Proton (draft) #259 itself: it expects a register spill that ROCm 10.1's compiler no longer produces, and CI does not run it).Not re-run after the rebase on 0.6.5
Nothing was run on an RDNA 3 card after the rebase. The card tests (stage check, PSO sweep, whole frame, k_native_check on the card), the Wine whole-frame runs and the in-game runs are from before it. After the rebase I checked, without an RDNA 3 card: the 16 kernel files of the
dx12-gfx11set are byte-identical to the set that was measured on the card, and the 16 commits' patches are unchanged. Not run at all: Windows, and an RDNA 4 card in game with these commits.An observation about the queue (not changed in this PR)
On this card and driver (Mesa RADV 27.0-devel), identical K frames on the own-buffer executor's COMPUTE queue run at two speeds about 0.24 ms apart, in runs of 2-3 frames, at the same shader, memory and fabric clocks. It is not overlap of
k_warm/k_out_color, not history ping-pong, not power state and not the timing queries (frames with no timing at all show it too). The same work on a DIRECT queue has no slow frames and is not slower (240 of 240 frames in one band; output byte-identical). A bottom-of-pipe timestamp mid-network or between all network stages also suppresses it, at a small cost. It looks like a scheduling bubble specific to the separate compute queue on this driver; I have not found its owner. A game recording into its own DIRECT queue would likely not see it. If you see anything similar on RDNA 4 I would like to know.Open decisions (yours; none chosen here)
mix_uncheckedhook.<name>__g11lookup used here, or a digest rule that skips gfx11-only declarations so the native files can use the port names.Checklist
Signed-off-by:line (DCO 1.1).REUSE.toml.