Repository navigation
RDNA 3 (RX 7900 XT): Model M in HIP on gfx1100 and a developer-only D3D12 route under Proton (draft) - #259
RDNA 3 (RX 7900 XT): Model M in HIP on gfx1100 and a developer-only D3D12 route under Proton (draft)#259j0nsh1n wants to merge 41 commits into
Conversation
|
Thanks for this, and for the RX 7900 XT work behind it. It will get a proper review after 0.6.5 ships; until then the release work comes first, so expect a delay. One thing to know up front: the route the game uses is D3D12 only, and that stays so. A HIP path for gfx1100 can land as a developer or test route, not as the in-game one, so please keep it behind the developer and test entry points. I've put it on the 0.6.6 milestone. |
|
RDNA 4 source text. This PR leaves the gfx12 program the same, but not its text: the gfx11 operand types, fragment Fix (one commit on top of the head, Guard. Evidence (ROCm 10.1, clang 24).
Rebase note. Main has moved since this PR's base (the hybrid form, #256, touches the same three places in |
4ea9f74 to
f03c2c2
Compare
|
Rebased onto 0.6.5 (
Checked with ROCm 10.1: the gfx12 source guard (172 compile forms, 0 differ), all 154 gfx12 DXIL files and all 52 gfx1201 code objects byte-identical to main's, the release runtime builds. Not run here: 0.6.5's RGA post-build scan (Windows tool; the DXIL bytes it reads are main's), and nothing on an RDNA 3 card since the rebase. |
The block kernel called __builtin_amdgcn_readfirstlane directly and held a "+s" asm constraint for the RESIDENT kernels. Neither exists in the host build (-DWG_EMU), so build.py --emu-only and every emulator test failed with "use of undeclared identifier '__builtin_amdgcn_readfirstlane'" and "invalid output constraint '+s' in asm". The wave index now goes through wg_uniform, which is readfirstlane on the device and the value itself on the host. The asm is left out of the host build (#ifndef WG_EMU), the same form as upstream PR phantoslayer21#279 for this line. Device code is unchanged. Verified: build.py --emu-only fails with those two errors before and builds after; TestLayoutIndependence passes (27 s); the 27 gfx1201 code objects have identical SHA-256 before and after (clang 20, ROCm 7.1.1). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
build.py built for gfx12 only. - build.py takes --arch (gfx1200, gfx1201, gfx1100, gfx1101) and --sources (build exactly these kernel sources and write target.json). gfx11 builds post, exposure and frame by default (FAMILY_SOURCES; the matrix kernels need gfx12 builtins) and no Model K / L kernels, variants, resource gate or runtime. - family.py maps exact target names to gfx12 / gfx11 and refuses everything else: gfx1170-1172 are LLVM gfx11 names with the gfx12 matrix design, and gfx1102 / gfx1103 / gfx115x are untested. Runner takes its default accumulator row table from the build's target.json (WG_D_ROW and the d_row argument still win; no file means gfx12, as before). - clean/README.md gains a short Linux section. --rocm names an SDK in the layout of AMD's installer; how the SDK is found is unchanged. Verified: the new tests fail before and pass after; mapping gfx1170 to gfx11 turns them red, restored they are green. build.py --arch gfx1100 --sources post.hip exposure.hip frame.hip exits 0. The gfx1201 code objects (27 Model M, 8 Model K, 4 Model L) keep their SHA-256 (clang 20, ROCm 7.1.1). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…U_TEST=1) First run of the HIP host path on a Radeon RX 7900 XT (gfx1100): exposure, post, filter and frame kernels through the spec 90 runner. - test_gpu_stages.py (skipped unless WG_GPU_TEST=1; WG_BUILD, WG_ROCM) reuses the emulator test classes of exposure, post and filter with the runner replaced, so their cases, float64 references and limits are the same, and adds color_divide / scale_rgb of frame.hip against FP32 NumPy (bit equal, including zeros, denormals, infinities and NaN). - test_exposure.py: the kernel cases move to KERNEL_CASES / kernel_color; test_post_emu.py: TestFilterEmu.KERNELS names the filter kernels. Both are behaviour-preserving and let the GPU test pick them up. - Runner no longer needs swin.co in a HIP build made for a target without the Swin kernels (gfx11 so far); a gfx12 build still fails at construction. - post_filter_quality9 is left out of the GPU filter test: it differs from reference/post.py in every value on the host emulator as well (the failure already on main); the card agrees with the emulator to 3e-7 relative, so it is not a gfx11 defect. Verified on the card: 8 tests pass (exposure 10 cases, post 14 cases and the reset test, filter 7 sizes x 6 kernels bit-identical across kernels, frame 50 cases). With the references changed on purpose (frame divisor x1.0001, exposure alpha x0.8) 57 subtests fail; restored, green. The emulator tests keep their result (same 7 failures as before, post_filter_quality9). The 27 gfx1201 code objects have the same SHA-256 as before (nothing built differently). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ange, clear errors) RDNA 3 (gfx1100, gfx1101) has no FP8 conversion or matrix instruction and reads the other half of the wave with v_permlanex16 instead of ds_bpermute. The header knows that now; the gfx12 branches are untouched. - WG_GFX11 from __gfx1100__ / __gfx1101__ only (exact names; LLVM's gfx1170- 1172 have the gfx12 matrix design). It forces WG_SOFT_FP8, so every gfx11 kernel uses the software conversions. - wg_xchg16_u / wg_xor16 on gfx11 are one v_permlanex16_b32 (selectors 0x76543210 / 0xfedcba98, fetch-inactive on): lane l gets lane l ^ 16. wg_xchg16_u is the 32-bit word form on every target (ds_bpermute on gfx12, the lane emulator on the host); wg_mem.h's wg_xor16_u uses it on gfx11. - wg_wmma_fp8, wg_wmma_f16, wg_dot4_fp8 (WG_NORM_DOT4) and the WG_PROFILE counters (wg_realtime, wg_cycles, wg_hwid, wg_drain) are declared with __attribute__((error(...))) on gfx11: a call that survives to code generation fails with a plain message, while the dead `if (WG_NORM_DOT4 && ...)` branches of swin.hip still compile. - probe.hip: the matrix kernels and the hardware conversions are not built for gfx11; probe_xchg16 (gfx11 and the emulator only, so the gfx12 probe.co keeps its hash). build.py's gfx11 default sources gain probe.hip. - selftest.py: a gfx11 build skips the matrix and hardware-conversion probes, and runs the exchange probe (table, word and float forms, NaN payloads) plus the software conversion checks on the card. - pipeline.py takes its default accumulator row table from the build's target, as stage_runner.py does (WG_D_ROW and d_row still win). Verified on a Radeon RX 7900 XT: selftest.py on a gfx1100 build passes, with the exchange table [16..31, 0..15] (lane l gets lane l ^ 16), 0 word and 0 float mismatches in 512 lanes, and the software conversions bit-exact against reference/fp8.py (4274 encode, 256 decode, 8544 pack, 0 mismatches). New tests: the exchange in the emulator (test_kernels_emu), the gfx11 compile checks (test_gfx11_platform: macro, one v_permlanex16 on gfx11 and ds_bpermute on gfx12, each error message, dead code still compiles) and Pipeline's default rows (test_family). With the exchange replaced by the lane's own value, the card probe reports 512 lanes wrong and the emulator test fails; restored, both pass. The 27 gfx1201 code objects have the same SHA-256 as before. The emulator, build, family and stage tests (87) keep their result: only the 7 failures of post_filter_quality9 that predate this. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…16 form
On gfx11 every FP8 conversion is software. The plain form (rint for the
subnormal range, integer rounding for the rest, selects for the edges) costs
about 19 vector instructions per value. This adds a form with the same
results: z = |x| clamped to [2^-6, 448] gives the magic constant
C = 1.5 * 2^(E + 20); the FP32 sum |x| + C rounds to the E4M3 grid with ties
to even, and the difference sum - C is exact. The code comes from the sum's
bits, or the value goes to F16 bits for the next F16 matrix operand.
- wg_platform.h, gfx11 only (WG_FAST_Q8; -DWG_SLOW_Q8 selects the plain
form): wg_f32_to_e4m3, wg_pack_q8, wg_pack_q8_inner and wg_pack_q8_nc (for
|x| <= 464 the bytes of wg_pack_q8) replace the software forms there, and
wg_q8_to_f16, _inner (NaN -> +-0) and _nc (|x| <= 464) are new. The gfx12
path and the host emulator keep their forms.
- The sum must not be reassociated: the header refuses -ffast-math and
-ffinite-math-only, build.py's compiler flags are checked for the others,
and a compile test shows that the sums disappear with -fassociative-math
-fno-signed-zeros (test_gfx11_platform.FastE4m3).
- runner/q8_exhaustive.py and probe_q8_sweep (probe.hip, gfx11): all 2^32
FP32 patterns in 256 chunks of 2^24, converted on the card by the functions
above and compared with a NumPy reference written from the definition
(checked against reference/fp8.py, test_gpu_q8.TestReference). About 0.4 s
per chunk, 1:42 in total, 1.9 GB of host memory, 150 MB on the card.
test_gpu_q8 (WG_GPU_TEST=1) runs 11 chunks that cover every class of input.
Verified on a Radeon RX 7900 XT (gfx1100, clang 20, ROCm 7.1.1): all 2^32
patterns, 0 mismatches in each of the 7 functions; selftest.py still reports
the software conversions bit-exact against reference/fp8.py. The check can
fail: with the magic constant's mantissa made odd (ties go wrong) 87 patterns
of the 11 chunks differ in every function, and with the F16 saturation at 480
instead of 448 19.9 million differ; restored, 0. The 27 gfx1201 code objects
have the same SHA-256 as before.
Timing, 672 workgroups x 256 lanes, 4096 iterations x 16 values (11.3 G
values) per launch, median of 10 launches, the same loop without a
conversion subtracted (2.1 ms), two runs:
FP32 -> 8 code bytes (wg_pack_q8): plain 13.6 / 13.7 ms, new 9.0 / 9.1 ms
(1.5x; 1.21 -> 0.80 ps per value over the device)
FP32 -> F16 operand: plain pack + widening 19.4 / 18.4 ms, new
wg_q8_to_f16 6.9 / 6.9 ms (2.7x to 2.8x; 1.69 -> 0.61 ps per value)
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…e of the lane emulator
The Swin kernels call wg_wmma_fp8 and wg_wmma_f16 with the gfx12 operand
shape: a lane holds 8 K values of its half of the wave. On gfx11 these are
now made from the F16 matrix instruction, so the kernels and the weight
blob stay as they are.
- wg_platform.h, gfx11 and the emulator's gfx11 mode (-DWG_EMU_GFX11): the
lane takes the 8 values of the other half with wg_xchg16_u, puts them
first or second by its half (K slot (h, j) is index 8h + j, both
operands alike), widens the 16 codes to F16 and calls wg_wmma16_f16
(v_wmma_f32_16x16x16_f16). The widening puts the 7 magnitude bits under
the F16 exponent and multiplies by 256: all 256 codes are exact, and the
NaN code 0x7f gets bit 14 so that it stays a NaN (a compare and select
per code made the block kernels spill; this form costs about 3 VGPRs).
The accumulator rows are 2v + h: d_row "gfx11".
- emu/wg_emu.{h,cpp}: wmma16_f16 models the instruction as measured on an
RX 7900 XT: output lanes 16h..16h+15 read A and B from the lanes of
their own half. (The documented layout repeats lanes 0-15 in lanes
16-31; the wrappers do.) build.py --emu-gfx11 builds wg_emu__gfx11.so;
Runner(..., backend="emu", arch="gfx11") loads it and takes d_row "gfx11".
- probe.hip: probe_wmma_fp8 / probe_wmma_f16 run on gfx11 through the
wrappers; probe_wmma16 calls the instruction on raw registers.
- build.py: swin.hip for gfx11 is built with -DWG_SLOW_Q8 (FAMILY_DEFINES).
With the faster conversion of the previous change the C=128 block kernel
spills 27 VGPRs and the other block kernels take 30 to 35 more; the
speed slice budgets that.
Tests: test_kernels_emu_gfx11 runs enc1 (and the other stages of the
emulator tests' extra cases) through the wrappers against the float64
reference with the limits of test_kernels_emu, and the wrappers against
exact arithmetic (random E4M3 and F16 operands bit for bit, every code,
NaN codes). test_gfx11_platform checks the instruction mix of the wrappers
for gfx1100 and gfx1201, and gates the registers of the five block kernels:
kernel VGPRs spills waves/SIMD
swin_block_c64 175 0 8
swin_block_c64_add 175 0 8
swin_block_c96 206 0 7
swin_block_c96_add 206 0 7
swin_block_c128 250 0 5
(clang 20, ROCm 7.1.1, -DWG_SLOW_Q8; patch_merge_c64 237, 0, 6)
The comparisons can fail: swapping the half select for operand B only gives
91 % of the enc1 skip values more than one step off; using the lane's own
half twice or dropping the multiply by 256 fails the wrapper and enc1 tests;
restored, all pass. Swapping the select for both operands is not caught, by
design: both then take the same K order, and an exact sum does not depend on
it (the hardware rounding does). The 27 gfx1201 code objects have the same
SHA-256 as before; the gfx11 build of swin.hip compiles; no card run yet.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…es and the exchange in divergent code runner/selftest.py now runs the matrix probes on a gfx11 build (through the wrappers of the previous change) and has three new probes for gfx11 and the gfx11 emulator mode (wg_emu__gfx11.so, --emu-library). - probe_wmma (existing): on the RX 7900 XT it reports d_row [[0, 2, .., 14], [1, 3, .., 15]], name "gfx11", for FP8 and F16, with A row = lane % 16, B column = lane % 16, K slots paired, 1 x 1 = 1. - probe_wmma_arithmetic: 32768 random problems per kind (8.4 M outputs) against exact float64 sums, whatever the K slot order; error in units of 2^(E - 24), E the exponent of the largest product. Card, E4M3 (uniform random codes): 98.86 % within 4 units, 100 % within 16, largest 17.1 (analysis: largest 23 on 33.5 M outputs); F16 (normal values): 87.8 %, 99.96 %, largest 37.3; the same in a second seed. The emulator's exact model: 99.9999 % within 4, largest 8. Limits: none beyond 64; E4M3 at least 98 % within 4; F16 at least 99.9 % within 16 (the 99 % within 4 of the plan is the analysis' figure for the P1b-like data, 99.8 %; uniform random codes give 98.9 %). - probe_wmma16_lanes (probe_wmma16, the instruction on raw registers): lanes 16-31 of A and B are not ignored. The lanes 16-31 of the result (odd rows 2v + 1) take A and B from lanes 16-31, lanes 0-15 (even rows) from lanes 0-15. The wrappers give both halves the same operands, so the sum does not depend on this; with other operands in the halves it does. - probe_xchg16_divergent (probe_xchg16_div, gfx11): wg_xchg16_u in code that only some lanes execute (lanes 0-15 only, lanes 16-31 only, odd lanes, every fourth lane, a data-dependent half): every lane that executes gets the value of lane l ^ 16, also when that lane does not execute (its register is read all the same); the other lanes are untouched. 0 wrong values in all five modes. Tests: test_gpu_probes (WG_GPU_TEST=1) asserts the card's report; test_kernels_emu_gfx11 runs the same probes in the gfx11 emulator mode. Wave64 post on gfx11 is still not tested. The probes can fail: built with the multiply by 256 left out of the widening, the card reports 1 x 1 = 1.5e-05 (2^-16) and the arithmetic probe is off by up to 1.6e8 units; built without the exchange, 1 x 1 = 2 and the K pairing fails; an emulator that reads lanes 0-15 for B in all rows fails the selftest and test_the_instruction_reads_each_half_for_its_own_rows. Restored, everything passes (card: 5 probe tests; A2-A4 card tests 27). The 27 gfx1201 code objects have the same SHA-256 as before. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…n.hip in the gfx11 build) The default build for gfx1100 / gfx1101 now includes swin.hip (built with -DWG_SLOW_Q8, see the first change), so Runner on a gfx11 build runs the Swin stages. test_gpu_enc1 (WG_GPU_TEST=1) runs enc1 (swin_block_c64, then patch_merge_c64) on the card with the emulator tests' four enc1 cases. Measured on a Radeon RX 7900 XT (gfx1100, clang 20, ROCm 7.1.1, headless): - Against the float64 reference (the emulator tests' cases, d_row "gfx11"): spec 09 holds in all eight outputs: no non-finite value; 0.05 to 0.77 % of the values more than one FP8 step off (limit 2 %); mean absolute error 0.005 to 0.115 % of the reference RMS (limit 1 %). - The stricter limits of the emulator tests: below 5 % different per case holds (0.2 to 3.4 %). Below 1 % different over all cases does not: 1.0076 % over the four enc1 cases (173,184 values). The gfx11 emulator equals the reference on these cases, so the same number is the distance card to emulator: 10,076 per million one-code flips, 2,292 per million more than one step (per case and output 2,000 to 34,000 per million; the analysis expects about 450 per million on realistic data, and random networks amplify). Swapping the K order of the odd rows on the card (a change of rounding only) flips 1,386 per million, so most of it is not the order of the sum. The limit is a product question (the plan's question 7); the test is an expected failure, so a change shows. - Time, Runner.time, median of 50 launches, inputs on the card: 3.17 to 3.23 ms at the 1706x960 render (480x864 tokens), 6.86 to 6.97 ms at 2560x1440 (720x1280), over three runs. The tree gives enc1 0.35 ms at the first size for an RX 9070 XT (runner/sweep.py GAME_SIZES) and 0.94 ms for all of Swin at 1440p Quality (clean/l/SPEED.md): a different card, so about 9 times only as a rough comparison. Static ISA of swin_block_c64: 12,260 instructions, 10,519 vector ALU, 44 v_wmma_f32_16x16x16_f16, 124 v_permlanex16_b32, 432 v_pk_mul_f16. Without the widening the kernel has 3,940 vector ALU instructions: the widening is about 6,600 of them, 150 per matrix instruction. Checks that can fail: WG_D_ROW=gfx12 on the gfx11 build fails the spec 09 test in all eight outputs; a packer that ignores d_row fails test_the_row_table_matters (0.0 differ, limit 0.2); restored, both pass. The 27 gfx1201 code objects have the same SHA-256 as before. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On gfx11 every matrix instruction widened both E4M3 operands to F16 (about 6 600 of the 10 500 vector instructions of swin_block_c64). The weights, which one wave uses once per window, are now widened when the staged unit is copied to LDS: once per workgroup, the fragment stored as two planes of 16 rows, read back as 2 x 16 bytes. Opt-in (WG_F16_W, variants.py "f16w", build.py --arch gfx1100 --variants f16w); the default gfx11 build and every gfx12 object are unchanged. A unit of the plain block kernels holds half the code bytes, so the LDS stays the same; the C = 128 kernel keeps codes (with F16 fragments it needs more than 256 registers). The C = 64 kernels take the 8-wave occupancy floor (without it 256 registers, 27 spilled). Verified: bit-identical to the default gfx11 build in the gfx11 emulator for the five block kernels and merge / expand (test_kernels_emu_gfx11.TestVariantsGfx11); identity goes red when one FP32 sum order is changed in the variant and green when restored. On the RX 7900 XT enc1 meets spec 09 as before; median of 50: 1.90 ms at 1706x960 (3.2 ms before) and 4.04 ms at 2560x1440 (6.9 ms before). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On top of f16w: each activation operand (the quantized values of a 16-token tile, used by up to 16 matrix instructions) is exchanged with the other half of the wave and widened to F16 where it is quantized (pack_act), 8 registers per K group instead of 2, and the softmax probabilities are exchanged once for both value tiles. The register cost makes the C = 96 kernels spill 30, so the variant applies to the C = 64 kernels only (the others stay as f16w); the C = 64 kernels take an occupancy floor of 7 waves (8 spills 8). Opt-in (WG_F16_A, variants.py "f16wa"); the default gfx11 build and every gfx12 object are unchanged. Verified: bit-identical to the default gfx11 build in the gfx11 emulator (TestVariantsGfx11); the identity goes red when one FP32 sum order is changed under WG_F16_A and green when restored. enc1 on the RX 7900 XT meets spec 09 as before; median of 50: 1.63 ms at 1706x960 and 3.48 ms at 2560x1440 (f16w 1.90 and 4.04, default 3.2 and 6.8). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
On top of f16wa: the activation operands come from the exact E4M3 rounding done as FP32 arithmetic and v_cvt_pk_rtz_f16_f32 (wg_q8_to_f16), not from code bytes widened again; the code bytes are made only for the stores. That takes the plain software conversion (about 3 900 vector instructions per window of swin_block_c64) and the widening of the activations out of the kernel: 6 060 to 3 620 vector instructions in swin_block_c64. The attention output o keeps the code form (with the F16 form the C = 64 kernels spill 75 registers). The F16 forms of the rounding (wg_q8m_round, wg_q8_to_f16*) move out of the WG_FAST_Q8 block of wg_platform.h so that they exist in the default gfx11 build (plain code conversion, -DWG_SLOW_Q8) and in the emulator; the code is the same. -DWG_SLOW_Q8=0 now selects the faster code conversion. Opt-in (WG_F16_Q, variants.py "f16wq"); the default gfx11 build and every gfx12 object are unchanged. Verified: bit-identical to the default gfx11 build in the gfx11 emulator (TestVariantsGfx11); the identity goes red when one FP32 sum order is changed under WG_F16_Q and green when restored. enc1 on the RX 7900 XT meets spec 09 as before; median of 50: 1.30 ms at 1706x960 and 2.79 ms at 2560x1440 (f16wa 1.63 and 3.48, default 3.2 and 6.8). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
runner/gfx11_decision.py measures stages enc1 and b0 (C = 128) for the default gfx11 build and each variant of variants.FAMILY_VARIANTS["gfx11"], and prints one markdown sheet: spec 09 numbers against the float64 reference, the verdict word of equiv/compare.py on the same pair (its use on a stage output against our own reference is marked INFERRED in the sheet), the one-code flip fraction against the gfx11 emulator, VGPRs, spills and waves per SIMD from the compiler, the median of 50 launches at 1706x960 and 2560x1440, and a per-frame Swin time extrapolated from them (INFERRED; the RX 9070 XT figure of l/SPEED.md is a different-card comparator only). It presents evidence and decides no output standard. Synthetic weights and inputs only. tests/test_gfx11_decision.py (CPU only): the verdict mapping on maps with a known share of one-step differences, the spec 09 limits, the per-frame arithmetic, the resource-report reading and the structure of the sheet. Checked red on purpose by reporting DIFFERENT as CLOSE, loosening the 2 % limit and counting b0 five times per frame; green again after restoring. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…r them (E1) The FP16 route of the C = 96 (enc2, dec2), C = 128 (b0 to b5) and `_add` (dec1, dec2) block kernels and the weight-resident `_r` kernels on the RX 7900 XT, each against the float64 reference with the acceptance of test_gpu_enc1 (provisional: the output standard is the maintainer's decision): spec 09 per case and output, fewer than 5 % of the values different per case, the "< 1 % over all cases" of the emulator tests. Worst case and output, default gfx11 build, over the cases of each test: stage more than 1 step off mean abs error (% RMS) values different over all cases enc2 0.977 % 0.217 4.54 % 0.638 % b0-b5 0.187 % 0.037 1.12 % 0.631 % dec2 0.181 % 0.033 0.94 % 0.839 % dec1 0.225 % 0.027 0.77 % 0.454 % No non-finite value, no limit missed, and the overall bound holds for every group (enc1's 1.008 % stays the one expected failure). Distance to the gfx11 emulator, reported: 4 500 to 6 900 one-code flips per million over all cases. The `_r` kernels give the plain kernels' bytes for 1, 2 and 5 workgroups (each wave then takes several windows), and the same reference numbers. Runner.resident = N launches them for enc1 and dec1 (the runtime's swin_resident); at 84 workgroups they are slower than the plain kernels on this card (enc1 4.0 against 3.2 ms at 1706x960, dec1 4.0 against 3.4 ms). Finding, pinned as an expected failure and left to slice E2: expand_c96 (107 spilled registers) writes NaN codes for valid tokens when a wave's second 16-token tile is only partly valid (60, 84, 120 low-resolution tokens; 36, 40, 100, 288 are fine). The emulator tests' case dec1 12x20 gives 5 120 non-finite values on the card for this reason, so the dec1 cases here use sizes without that shape; the block kernel `swin_block_c64_add` is not involved (dec1 36x32 and 16x20 are within the limits). gfx11_gates.py and gfx11_gates.json: compiler-reported VGPRs, spills and waves per SIMD of the 14 block kernels (`_nc` forms included) in the default gfx11 build and in f16w, f16wa and f16wq. A gate is no spill, at most the recorded registers, at least the recorded waves; a new table for the gfx11 builds, not clean/dx12/vgpr_limits.json. `python clean/runner/gfx11_gates.py --rocm <ROCm> --scratch <folder>` checks, `--write` records (clang 20). The merge, expand and fused kernels are not gated: they spill in these builds (E2). Verified: the new tests and test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages, test_kernels_emu_gfx11, test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass. Break on purpose, red then green: c128 heads and the dec skip add (bottleneck, dec1, dec2 red, enc2 green); c96 heads (enc2 red); the resident loop skipping a window (bit-identity red); the runner ignoring d_row and the resident option (row-table and launch tests red); one GELU coefficient changed (the overall bound red, spec 09 still green); the gate check's spill test, and a recorded wave count one too high (the compile-based gate test red). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…c128w7) The default gfx11 build leaves swin_block_c128 without an occupancy floor: the allocator takes 250 registers and 5 waves per SIMD. With a floor of 7 waves (WG_MINW_C128=7) it fits in 206 registers with no spill; a floor of 8 gives 192 and no spill too. Nothing else changes: no source edit, one entry in variants.py. On the RX 7900 XT, median of 50 launches, three runs (ms), at 1706x960 and 2560x1440: stage default c128w7 b0 1.089 1.117 1.171 (1.117) 0.911 0.913 0.910 (0.911) 2.139 2.130 2.154 (2.139) 1.887 1.744 1.743 (1.744) b1 1.092 1.085 1.094 (1.092) 0.831 0.827 0.830 (0.830) 2.086 2.090 2.073 (2.086) 1.616 1.618 1.615 (1.616) That is 18 to 24 % less per launch, six launches per frame. A floor of 8 timed the same at 1706x960 and 1.65 ms at 2560x1440. New on the card: TestVariantsGpu runs every gfx11 variant of the build against the default gfx11 build on seven cases of every block shape and asks for the same bytes. They are the same for f16w, f16wa, f16wq and c128w7 (the F16 operands are exact and the K order is the matrix instruction's own). The reference numbers of the stage tests for the c128w7 build equal the default's. gfx11_gates.json gets the row of c128w7 (206 registers, no spill, 7 waves). Verified: the new test red with a build whose c128w7 object has one GELU coefficient changed, green again with the real one; the gate table test red with the c128w7 row deleted; test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages, test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11 (c128w7 bit-identical to the default in the emulator), test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
Timing-only builds (output wrong on purpose) show where the C = 128 kernel's 1.1 ms goes on the card: without the W1 and W2 multiplies it takes 0.41 ms, without the QKV multiplies 0.75 ms, with no matrix work at all (gather and store) 0.035 ms. The matrix instructions themselves are a small part of that (INFERRED: about 770 of them a window); each call also widens its operands from codes to F16, and f16wq already removed that for the C = 64 kernels, whose activation operands are made once as F16 (C < 96). The wider kernels kept codes because the F16 operands cost registers (C = 96: 30 spilled, then 12 with the attention output left on codes). Three knobs name the smallest C they apply to (0 = none, default build unchanged): WG_F16_AWIDE makes the activation operands F16 operands in the kernels with C >= N, WG_F16_QWIDE makes them straight from the FP32 values there too, WG_SCHED_FRAGS puts a scheduler barrier after each fragment step of the QKV, Wp, W1 and W2 loops of those kernels. Variant f16wk is f16wq with AWIDE = QWIDE = SCHED = 128: only swin_block_c128 changes. 238 registers, 6 waves per SIMD, no spill (default: 250 registers, 5 waves). Without the barriers the register count is the same and the kernel is 20 % slower (0.85 against 0.70 ms at 1706x960; the cause was not looked into). F16 weights for C = 128 (WG_F16_W on this kernel) still spill 94 registers with or without the barriers, so the weights stay codes there. RX 7900 XT, median of 50 launches, three runs (ms), 1706x960 and 2560x1440: stage default f16wk default f16wk b0 1.089 1.117 1.171 (1.117) 0.705 0.715 0.703 (0.705) 2.139 2.130 2.154 (2.139) 1.276 1.290 1.284 (1.284) b1 1.092 1.085 1.094 (1.092) 0.628 0.626 0.634 (0.628) 2.086 2.090 2.073 (2.086) 1.193 1.193 1.186 (1.193) 37 to 43 % less per launch; c128w7 (previous commit) gave 18 to 24 %. Same card bytes as the default build on seven cases (TestVariantsGpu), same reference numbers as the default build (TestBottleneckGpu, TestEnc2Gpu, TestDec2Gpu with the f16wk build). Verified: bit-identical to the default gfx11 build in the gfx11 emulator for every case (TestVariantsGfx11); the identity goes red for f16wk, stage b1 only, when the C = 128 operand is scaled by 1.01 under WG_F16_QWIDE, green when restored; gate table re-recorded with the f16wk row (c64 212 / 7 waves, c96 248 / 5, c128 238 / 6, no spill); test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages, test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11, test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…16wm) The C = 96 block kernels (enc2, dec2) keep code activations under f16wq: F16 operands for the whole block spill 12 registers (30 with the operands made from FP32). Their MLP is the larger half of the matrix work (W1 and W2 against the norm output and the hidden values), so WG_F16_M makes only those two operands F16 operands, made once and used by every hidden tile, in the kernels with C >= 96 that WG_F16_A leaves on codes. Variant f16wm is f16wk with WG_F16_M = 1 and the scheduler barriers of WG_SCHED_FRAGS in the C = 96 kernels too (WG_SCHED_FRAGS = 96): without them the C = 96 kernels spill 23. The operands are made from codes, not straight from the FP32 values (that spills 133); the C = 64 kernels are f16wq's and the C = 128 kernel f16wk's. swin_block_c96[_add]: 252 registers, 5 waves per SIMD, no spill (f16wq: 248, 5 waves). RX 7900 XT, median of 50 launches, three runs (ms), 1706x960 and 2560x1440: stage f16wq f16wm f16wq f16wm enc2 1.523 1.509 1.505 (1.509) 1.328 1.347 1.322 (1.328) 3.002 3.016 2.995 (3.002) 2.648 2.697 2.688 (2.688) dec2 1.206 1.199 1.214 (1.206) 1.042 1.036 1.036 (1.036) 2.395 2.384 2.385 (2.385) 2.058 2.055 2.056 (2.056) 12 to 14 % less than f16wq, 44 % (enc2) and 49 % (dec2) less than the default gfx11 build (2.360 and 2.014 ms at 1706x960). Same card bytes as the default build on seven cases (TestVariantsGpu); the reference numbers of enc2, dec2, dec1 and the bottleneck with the f16wm build equal the default's (TestEnc2Gpu, TestDec2Gpu, TestDec1Gpu, TestBottleneckGpu). Verified: bit-identical to the default gfx11 build in the gfx11 emulator for every case (TestVariantsGfx11); the identity goes red for f16wm, stages enc2 and dec2 only, when the hidden values are scaled by 1.01 under WG_F16_M, green when restored; gate table re-recorded with the f16wm row (c64 212 / 7 waves, c96 252 / 5, c128 238 / 6, no spill); test_gpu_enc1, test_gpu_probes, test_gpu_q8, test_gpu_stages, test_gpu_swin_stages, test_gfx11_gates, test_kernels_emu_gfx11, test_gfx11_platform, test_gfx11_decision, test_build_targets, test_family, test_hip_env pass. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…x11 (E2) On gfx11 expand_c96 wrote NaN codes (or wrong values) for valid tokens when a 16-token tile of a wave was only partly valid (60, 84, 120 or 140 low-resolution tokens). The store was `if (token < ntok)`; the compiler moved the whole v_wmma chain of the result into that branch (its only user), so the matrix instruction ran with EXEC partly off. v_wmma needs every lane executing; the emulator runs all lanes, which is why it never showed. The same move made the kernels spill (expand_c96 107 registers, patch_merge_c96 101) and patch_merge_c64 and c96 carried the same latent hazard. Fix at the source, for the gfx11 family only (WG_TILE_STORES_UNGUARDED; the other targets keep their code, gfx1201 objects are byte-identical): a lane past the last token repeats the work of the last token and stores the same bytes to the same place, so there is no branch to move work into. Results are unchanged: the kernels are bit-identical in the gfx11 emulator. Spills now 0 in every gfx11 build: patch_merge_c64 185 VGPRs, patch_merge_c96 251, expand_c96 199, expand_c128 248; they join the gate table. On the card (1706x960): patch_merge_c96 0.639 -> 0.422 ms, expand_c96 0.688 -> 0.418 ms, patch_merge_c64 and expand_c128 unchanged. The expected-failure test becomes a passing test and a new test runs each kernel alone with partial tiles against the gfx11 emulator (red with the branch put back, green with the fix). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… gfx11 (E2) The default patch_merge widens the weight fragment and the token operand for every matrix instruction (the half-wave exchange is not merged), so each v_wmma pays for two widenings. In the variant the loops run K group outermost over 6 (C = 64) or 4 (C = 96) output tiles at a time: each token operand is widened once for all of them and each weight fragment once for both token groups. Every accumulator adds its K groups in the same order, so the result is bit-identical (gfx11 emulator, card). Opt-in (variants.FAMILY_VARIANTS): the default build and the gfx12 objects are unchanged. No spills in any kernel of the build (patch_merge_c64 250 VGPRs, patch_merge_c96 143, expand_c96 199, expand_c128 248); the table of resource gates has the mgx build. On the card (RX 7900 XT, median of 50, 1706x960): patch_merge_c64 0.402 -> 0.354 ms, patch_merge_c96 0.422 -> 0.311 ms (2560x1440: see the report); expand is unchanged. Tried and not adopted: 3 token groups per wave (merge c96 0.223 ms, expand_c128 0.342 ms but 9 spilled registers; splitting the expand into passes spills more or gains nothing), 128-lane workgroups, 1 token group per wave. Also: gfx11_gates.py --write no longer empties the table when a kernel spills (it built the table after opening the file); the partial-tile test has a third case size (36 tokens: a partial third tile with 3 groups per wave). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…rd (E3) swin_v2.hip with swin.hip's arithmetic (swin__bmegahg.co, the runtime's default for b0..b5 on gfx12) joins the default gfx11 build: swin_bneck6 (b0 to b5 in one launch) and swin_mid (enc2 with its merge, b0 to b5, dec2 with its expand in one launch). No kernel source changes: the file compiles for gfx1100 through the wrappers of wg_platform.h, and the sync-word protocol is the one of gfx12. build.py builds the object with the family's defines (variants.FAMILY_MEGA) and writes variants.json for it; the Runner takes it as a gfx11 variant with the grids of the family's card (FAMILY_MEGA_GROUPS: 126 workgroups for swin_bneck6, 84 for swin_mid; any grid is correct, these are the resident capacity). The E2 hazard, checked: a control-flow scan of the compiler's assembly (every basic block reachable under an s_and_saveexec before its join) finds no v_wmma and no v_permlanex16 under a lane-divergent exec in swin_bneck6, swin_mid or fused_merge. The kernels have no branch round a store for the compiler to sink a matrix chain into. Card (RX 7900 XT), against the plain launches of the default build: bit-identical for both parities, 1 / 3 workgroups and the build's grid, maps with partial windows and one or two window rows, clamping and clamp-free kernels; sync words zero after every launch. Block by block against the float64 reference (each block fed the card's own output of the one before): spec 09 and < 5 % different per block everywhere (worst: b1 12x16 1.27 % different, 0.31 % more than one step off). The six blocks chained against the reference chain compound the one-code flips of the F16 matrix instruction: 26.5 % more than one step off at 12x16 (plain launches the same bytes). A reference chain given flips of 0.2 % of the values per block shows the same amplification (34 % at the end), so this is the random synthetic network, not the kernel. swin_bneck6: 239 VGPRs, 0 spills, 6 waves per SIMD, 32 KB LDS (gate table, new build "bmegahg"; the other recorded builds are untouched). swin_mid: 256 VGPRs, 143 VGPRs and 28 SGPRs spilled, 5 waves: correct on the card but NOT gated. Its two C = 96 phases (enc2 with the merge, dec2 with the expand) spill about 70 registers each, the C = 128 phases none; plain C = 96 phases spill 79 too, so it is the C = 96 streamed body in the 256-lane queue, not the fusion. Two knobs tried without effect (WG_V2_PIPE=0: 131 spilled; no scheduling fences: 141). Open. Times, median of 50 launches, three runs (ms; 1706x960 | 2560x1440): bneck plain 6.165 | 12.354 megakernel 4.600 | 9.369 plain f16wm 3.449 | 6.808 mid plain 10.037 | 20.439 swin_mid 10.127 | 22.272 plain f16wm 5.407 | 11.075 The megakernel is 25 % faster than the default plain blocks and 33 % slower than the f16wm blocks; swin_mid is level with the plain launches at 1706x960 and 9 % slower at 2560x1440 (the spills), and 1.9 times f16wm. Grid sweep (swin_bneck6, 1706x960): 42 / 84 / 126 / 252 workgroups 12.0 / 6.2 / 4.5 / 4.5 ms. Header in LDS (bmegahg with WG_V2_MEGA_HDR_GLOBAL=0) 4.40 ms, 128 lanes 8.0 ms: no variant is added. Verified: TestBmegaGfx11 and TestMidGfx11 (the existing v2 emulator tests with a gfx11 mode) bit-identical to the plain gfx11 kernels; the gfx12 modes of those classes unchanged and green; clean.tests.test_gpu_swin_mega (new, card) and test_gpu_swin_stages, test_gpu_enc1 (known expected failure), test_gpu_probes, test_gpu_q8, test_gpu_stages, test_kernels_emu_gfx11, test_gfx11_platform, test_gfx11_decision, test_gfx11_gates (compiler check of all ten builds), test_build_targets, test_family, test_hip_env pass. Break on purpose, red then green: a wrong ping-pong buffer in swin_bneck6 plus a wrong expand slot in swin_mid (emulator tests: 14 failures; card tests red), a plain build with the attention switched off (card bit-identity and both reference tests red), the gate entry removed, the persistent build not built, a gfx12 build accepted as a gfx11 variant. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…t-build blocks on gfx11, and a scan that checks it The scan of the gfx11 code objects found every v_wmma and v_permlanex16 of swin_block_c64[_add][_nc] in swin__bmegahg.co under one lane-divergent mask (s_and_saveexec on vcc): the window loop's `if (win0 + i >= nwin) break;`. win0 is the wave's first window, a vector value to the compiler, so it could not see that the test is the same for the whole wave. v_wmma and v_permlanex16 need every lane executing. At run time the mask is all or nothing (win0 comes from the wave index), so the kernels passed the card cases; they carried the pattern. Fix for the gfx11 family only (WG_TILE_STORES_UNGUARDED, the switch of the E2 fix): win0 goes through wg_uniform (readfirstlane), the break becomes a scalar branch, and the whole loop runs with EXEC full. Other targets keep their code. Spills of the four kernels fall 48/46/108/90 -> 41/40/11/50 (still above zero: these kernels are not in the gate table, the runtime uses swin_bneck6). clean/runner/exec_scan.py: disassembles a code object and runs a forward data-flow over the control-flow graph of each function: EXEC is partial after s_and_saveexec, s_and/s_and_not1/s_xor into exec and v_cmpx, restored by s_or/s_mov exec from a register saved from EXEC (and after a divergent loop), joins are "partial if any path is". It lists every v_wmma / v_permlanex16 that may run under a partial mask. test_exec_scan: 15 hand-written functions (each rule, with a look-alike that must not be flagged) and, with a gfx11 build, a scan of every kernel the runner launches (0 hits in the fixed build, 4 kernels red in the build before this commit). On the card the old code cannot give a wrong result (the mask is uniform), so TestResidentBlockGpu (partly filled workgroups, 1 to 17 windows, both block kinds, clamping and clamp-free) is green before and after; it goes red with the window count off by one (checked). The scan test is the one that is red before. Not changed, reported by the scan: swin_mid and the fused kernels (_mg, _ex, fused_merge) still carry the pattern (swin_mid is parked for its owner); post.co (v_permlanex16 in the reconstruction kernels) and probe_xchg16_div (deliberate) are flagged too. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…under a lane mask (E4) kernels/encoder.hip builds for gfx1100 with the gfx11 wrappers unchanged (the packers and the weight blob are the ones of gfx12: only d_row differs). On a partly valid 16-token tile the first card build was wrong: the features store is `if (tok < ntok)`, the compiler moved the whole projection (the half-wave exchange of the pixel codes, v_permlanex16, and the v_wmma chain) into that branch, and both need every lane executing. Cases 3 and 13 of the encoder cases (22x18 padded to 26x20, 4x6: 130 and 12 tokens) gave 8 % and 23 % wrong features. Fix, gfx11 family only (WG_ENC_STORE_UNGUARDED, the E2 rule): the features are stored from every lane; a lane past the last token repeats the work of the last token (its loads are already clamped to it) and stores the same bytes to the same place. Other targets keep their code; gfx1201 objects are unchanged. Resources (gfx1100, clang 20.0.0.rocm): input_encoder 142 VGPRs, 0 spills, 10 waves per SIMD (gate recorded: new "enc0" entry in gfx11_gates.json; nothing else re-recorded). exec_scan.py: 0 of 113 matrix / exchange instructions under a lane mask (122 of 126 before the fix); the scan learns the if / else shape `s_and_not1_saveexec` that the encoder's code contains. Card (RX 7900 XT), the 14 cases of test_encoder_emu against the float64 reference: 0.000 to 0.003 % more than one step off, mean error at most 0.0015 % of RMS, 0.000 to 0.095 % of the features different per case (bound 5 %), 0.0575 % over all cases (467 840 values; the 1 % overall bound of the provisional acceptance); against the gfx11 emulator 0 to 946 per million one-code flips, the motion surfaces equal. The emulator test's own limits (0.01 % overall) are an expected failure on the card. Time, median of 50 launches, three runs: 1706x960 0.637 0.616 0.633 ms, 2560x1440 1.281 1.275 1.289 ms. Tests: test_gpu_e4.TestEncoderGpu (card; red with the encoder object built before this fix: cases 3 and 13 and the partly-valid-tile test; red with the store moved by 16 bytes: every case fails), test_e4_gfx11_emu (the same cases in the gfx11 emulator; red with the same mutation), test_exec_scan (encoder.co joins the kernels the card runs: red with the old object), test_gfx11_gates. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
kernels/decoder.hip needs no source change for gfx1100: its projection stays in registers, the pixel stores are scalar-condition stores with no matrix work behind them, and the scan finds 0 of its 68 matrix / exchange instructions under a lane mask. The change is the build list (decoder.hip joins the default gfx11 sources), the gate and the tests. Resources (gfx1100, clang 20.0.0.rocm): coef_decoder 103 VGPRs, 0 spills, 12 waves per SIMD, LDS 10 240 bytes (new "dec0" entry in gfx11_gates.json; nothing else re-recorded). Card (RX 7900 XT), the 5 cases of test_decoder_emu, the reset and the border case, against the float64 reference: A, B and latent in 8-bit codes, C in FP16 steps; at most 0.139 % of a case's values more than one step off, mean error at most 0.0052 % of RMS, at most 0.139 % different per case and output (bound 5 %), 0.0354 % over all cases (78 988 values; the 1 % overall bound of the provisional acceptance). Against the gfx11 emulator 0 to 1 389 per million flips. The emulator test's own limits (99.9 % identical) are an expected failure on the card. Time, median of 50 launches, three runs: 1706x960 0.431 0.411 0.413 ms, 2560x1440 0.728 0.729 0.728 ms. Tests: test_gpu_e4.TestDecoderGpu (card), test_e4_gfx11_emu.TestDecoderEmuGfx11 (gfx11 emulator, the emulator test's own limits), test_exec_scan (decoder.co joins the kernels the card runs), test_gfx11_gates. Red with the pixel-pair store condition `x0 + 1 < rw` changed to `x0 + 2 < rw`: the card cases and the emulator cases fail (5 cases, reset and border each); green restored. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
kernels/dec1_dec0.hip builds for gfx1100 unchanged and needs no source change: its epilogue (dec0's half-wave exchange and projection) runs after the swin block's work, in every lane, and the scan finds 0 of its 480 matrix / exchange instructions under a lane mask. The change is the build list (dec1_dec0.hip joins the default gfx11 sources, with -DWG_SLOW_Q8 like swin.hip, whose block it includes: 178 VGPRs against 214 with the faster conversion), the gate and the tests. Resources (gfx1100, clang 20.0.0.rocm): swin_block_c64_add_dec0 and _nc 178 VGPRs, 0 spills, 8 waves per SIMD, LDS 32 768 bytes (new "dec1dec0" entry in gfx11_gates.json; nothing else re-recorded). Card (RX 7900 XT): the fused kernel gives exactly the bytes of dec1 then dec0 (the kernel's contract; the 4 cases of test_dec1_dec0_emu, both clamping variants). Against the float64 reference, dec0's reference fed the card's own dec1 output: at most 0.104 % of a case's values more than one step off, mean error at most 0.003 % of RMS, at most 0.243 % different per case and output (bound 5 %), 0.0266 % over all cases (71 461 values; the 1 % overall bound of the provisional acceptance); dec1 itself 0.065 to 0.169 % more than one step off. Time, median of 50 launches, three runs: 1706x960 fused 3.412 3.446 3.434 ms, two launches 3.593 3.433 3.446 ms; 2560x1440 fused 7.406 7.404 7.439 ms, two launches 7.346 7.644 7.372 ms. The fusion is no faster than the two launches on this card (dec1's block dominates), so no speed variant was added. Tests: test_gpu_e4.TestDec1Dec0Gpu (card), test_e4_gfx11_emu.TestDec1Dec0EmuGfx11 (fused equals separate in the gfx11 emulator), test_exec_scan (dec1_dec0.co joins the kernels the card runs), test_gfx11_gates, test_build_targets (red with the define dropped). Red with dec0's pixel-pair store condition changed: the three reference cases of the card test fail; green restored. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…sts no longer skip) The tests of exec_scan ran with ROCM_PATH set to the link folder made for build.py (it has the compiler, not llvm/bin/llvm-objdump). The tool lookup took ROCM_PATH only and gave up, so TestExecScanBuild skipped every time in those runs, the acceptance runs of E3 and E4 included: the "no matrix instruction under a lane mask" test of the kernels the card runs was never evaluated there. find_objdump() tries WG_ROCM, ROCM_PATH, /usr/lib64/rocm and /opt/rocm in that order, then PATH. Checked by hand on the E4 head build (gfx1100, all kernels the card runs): the test now runs and passes, 0 findings in them. Tests: TestFindObjdump (first candidate with the tool wins; a folder without it is passed over). Red with the existence check dropped (the second case gets the missing path); green restored. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… post kernels on the gfx11 card The scan flags all 117 v_permlanex16 of post.co: 39 kernels (post_recon2 .. post_recon6, post_fused, post_fused3) with three each, the `nb[c] = wg_xor16(hsum[c])` of the reduced-history step, which the compiler put inside the store's branch `if (active && has1 && wl < 16 && x + 1 < dw)`. There is no v_wmma in post.co. Verdict: harmless, not a defect. In that branch only lanes 0 to 15 of a half-wave pair run, so the lanes the exchange reads (16 to 31) are off, but v_permlanex16 reads the register of a lane whose EXEC bit is off all the same (wg_platform.h says so; selftest probe_xchg16_div measures it on the card, mode 0 is this exact shape), and hsum was summed by every lane before the branch. The partner's value is the partner's real one. No source change, no change of any object: the gfx1100 post.co of this head and of the E4 head are the same bytes. The scan cannot prove this (it would need the reaching definitions of the source register), so it is pinned as a known, explained exception: test_exec_scan POST_EXCHANGE_KERNELS, exactly these 39 kernels with exactly three v_permlanex16 and no v_wmma each; a new site, a missing one, or a matrix instruction in a post kernel fails the test. Card (RX 7900 XT), new test_gpu_post: the emulator test classes of the post kernels (recon2 family, recon3 / 5 / 6, recon4, banded, output mode, fused, fused3) run on the card with their own cases (odd D, partial 32-column tiles, one D row or column, 3x ratio, NaN / inf inputs, reset): bit-identical to post_recon, which has no exchange, and the float64 limits where the emulator test has them. 21 tests, 0 failures; 38 of the 39 kernels are launched (a test checks the list). post_recon2_r_out_diet is not: the runtime's option post_diet=1 is its only caller and the stage runner has no launch plan for it; its body is post_recon2_r_out's. Not used by the Model M path on this card, for the record: the C++ runtime has no gfx11 build; its defaults would be post_recon2_r_band[_p12] with post_filter_c1_s4_band (no exchange), post_recon as the fallback. Red with the exchange removed (wg_xor16 returns its argument on gfx11, post.co rebuilt): 191 failures in test_gpu_post (test_identical_to_post_recon, test_identical_to_two_kernel_path, test_exact_variants, test_recon3, test_recon6); green with the source restored. Red with one kernel dropped from POST_EXCHANGE_KERNELS, and with a card class left out (the launched-kernels check); green restored. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…; the _mg / _ex blocks on the gfx11 card
The scan flags swin_block_c64_add_ex, swin_block_c96_add_ex (and their _nc forms)
and the two fused_merge functions the _mg blocks call, in swin.co, swin__mgx.co
and swin__f16wm.co: 216 / 300 / 32 (240 in f16wm for C = 64) matrix and exchange
instructions per function, all under ONE mask. Its origin is the entry test of
the loop over the output tiles, `for (mt = wave; mt < tiles; mt += 8)`: `wave` =
tid >> 5 is a vector value to the compiler, so it tests `tid < 32 * tiles` per
lane (v_cmpx with 0x1ff, 0x2ff, 0xc0, 0x100). The bound is a multiple of 32, a
wave32 holds 32 consecutive thread indices from a multiple of 32, so every lane of
a wave is on one side of it: the mask is full or empty at run time, and an empty
one skips the loop. The stores behind `if (tok[n] >= 0)` are after the matrix
chain, not around it. Verdict: harmless as launched (256 threads), not live.
No source change, on purpose. Making `wave` scalar (wg_uniform, as the persistent
blocks got) does clear the scan, but it adds spilled registers to kernels that
already spill: swin.co _ex_nc c64 / c96 37 -> 47, f16wm 4 -> 29, mgx 37 -> 47
(fused_merge shared with swin_mid kept its code with a template argument; checked
by building it). That is a speed cost for a mask that never happens, so the
pattern is pinned instead: test_exec_scan FUSED_GUARDED lists these functions with
their exact counts, one origin each, a v_cmpx with an immediate that is a multiple
of 32. swin_mid (swin__bmegahg.co, parked) is outside the pin and keeps its result.
Card (RX 7900 XT), new test_gpu_swin_fused (the first card runs of these kernels,
variants base, f16wm, mgx): 23 cases of enc1, enc2, dec1, dec2 with window offset
0 or 2, grids from 4x4 to 36x28 tokens (windows that stick out of the grid, a
workgroup of 8 windows only partly filled), fused kernels against the separate
ones: every output byte equal, no NaN left from the 0x7F fill; against the
float64 reference with the spec 09 limits of the stage tests (5 % different per
case where an output has 2000 values or more). 3 tests per variant, all pass.
Red with fused_merge's store changed to `tok[n] > 0` (swin.co rebuilt):
test_fused_kernels_give_the_bytes_of_the_separate_ones fails in 12 cases and the
reference test in 12; green with the source restored. FUSED_GUARDED red with a
count changed (216 -> 217) and red on the build that has the wg_uniform fix
("not flagged any more"); green restored. Not used by the Model M path on this card:
the C++ runtime has no gfx11 build (it would run the _ex kernels for dec1 and
dec2, WG_SWIN_FUSE_STAGES default); the stage runner's `fuse` geometry runs the
_mg ones.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ime accepts gfx1100
clean/runner/pipeline.py needs libwg_runtime.so, and the repository has none for
a gfx11 target, so this test skips (with the runtime's own message) at this head:
- build.py's gfx11 branch returns before the runtime step (its docs pass
--no-runtime); build.runtime_command compiles the unchanged sources fine;
- device.cpp:398 accepts gfx1200 / gfx1201 only ("no matching HIP device with
packaged gfx1200/gfx1201 kernels"), and load_module reads <arch>/<name> or
<arch>.<name> from the build folder.
Not changed here: that is the runtime's device policy, the maintainer's call.
Run in a scratch copy with exactly those three changes (nothing committed; synthetic
weights and frames only), RX 7900 XT, HIP, default kernels (swin.co, swin_bneck6 for
b0..b5, post_recon2_r_band + post_filter_c1_s4_band, separate dec1 / dec0):
- whole frame, median of 50, three runs: 64x40 -> 96x60 2.993 3.048 2.999 ms;
45x29 -> 67x43 3.008 2.999 2.995 ms; 1706x960 -> 2560x1440 17.761 18.018
17.891 ms. Output finite in all cases and frames. Per-stage times are not
exposed by Pipeline without the interop io (wg_rt_frame_io_timed); the stage
timings are those of the stage tests.
- stage by stage against the gfx11 lane emulator on the card's own stage inputs
(64x40 / 45x29 render): one-code flips 0.000 to 2.368 % per stage (enc1.merged
the highest), more than one step at most 0.651 %, b0..b5 chain 0.000 %.
Under the 5 % per case and 2 % bounds.
- output pixels against the exact (gfx12-arithmetic) emulator frame: 99.97 %
different (64x40, first frame); per-stage compounding on the random synthetic
network: enc2.merged 3.2 % of RMS, b5 8.0 %, output 28.3 %. Both the 5 % per
case and the 1 % overall bound are missed: recorded as expectedFailure.
Tests: test_gpu_pipeline (4 tests: 3 pass, 1 expected failure on the scratch
runtime; 4 skipped at this head). Red with the per-stage flip bound lowered to
0.001 % (9 failures); green restored.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ames of family.py device.cpp accepted gfx1200 and gfx1201 by two string comparisons, so a gfx11 card was refused at create. The gate now asks wgrt::target_family (clean/runtime/target_family.h), the exact-name map of clean/runner/family.py in C++: gfx1200 / gfx1201 -> gfx12, gfx1100 / gfx1101 -> gfx11, every other name refused (gfx1102, gfx1103, gfx115x, and gfx1170-1172, which are gfx11 names with the gfx12 matrix design, stay out). The refusal text names the four targets. Device gets arch(), the exact target whose code objects the device loads (empty for the emulator), for the runtime to derive the accumulator row table in the next change. Code objects are still looked up per exact target as <arch>.<name> or <arch>/<name> (never the unqualified file), so a build without gfx1100 objects fails closed with "missing gfx1100 code object". gfx12 devices take the same path as before. The D3D12 gates (rdna4.h, device_d3d12.cpp) are not touched. Whether players ever get a gfx11 build, and how that is gated, is the maintainer's decision; there is no option or environment variable for it here. Tests: - test_device_targets.cpp (Windows mock; run on Linux through a scratch windows.h shim and the mock built as libamdhip64.so, 27 checks): gfx1100 and gfx1101 now load their own gfx<arch>.<name> and report arch(); gfx1102, gfx1103, gfx1151, gfx1170, gfx1299, gfx11 and "unknown" are refused. The old expectation (gfx1100 refused) is flipped because this change is the policy change. - test_target_family.py (new, Linux and Windows): the C++ map equals family.py for every name, plus 16 refused names (decorated, partial, case, other families). Red on purpose, then restored: a gfx11 prefix match turned test_device_targets red (exit 7: gfx1102 accepted) and test_target_family red; dropping gfx1100 from the map turned both red (exit 3 and a map difference). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ce's target family The matrix instruction of gfx11 returns accumulator row 2 v + h in element v of lane half h, that of gfx12 row v + 8 h, and the packed weights follow the table (A3: packing.py D_ROW_LAYOUTS["gfx11"]). clean/runner/pipeline.py sets WgRtConfig.d_row from the build's target, but clean/runtime/interop never sets it, so a game's runtime always packed in gfx12 order and the gfx11 kernels would have read the wrong rows. The runtime now makes its device before it packs, and when the config's d_row is all zero it takes drow_gfx11() for a device whose arch() maps to gfx11 (target_family.h), else the gfx12 table as before. A d_row given in the config still wins. No device-less case changes: the emulator and D3D12 devices have no arch(), so they keep the gfx12 table, and on gfx12 the packed bytes are the ones of before. Creating the device is the only step that moved up; a device that cannot be made is still reported at the old place, after the model has loaded, with the same text. Tests: model_pack_dump takes an optional gfx11 argument, and test_model_pack_cpp packs all twelve stage blobs of two synthetic models with drow_gfx11 against packing.py's gfx11 table, byte for byte (the gfx11 and gfx12 blobs differ, so a wrong table cannot pass either test). Red on purpose, then restored: drow_gfx11 returning the gfx12 table fails it (seed 1, enc0: 4428 bytes differ). That the runtime picks the table from the device is checked on the card by the whole-frame test in a later change. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ch>.<name> next to the code objects build.py's gfx11 branch returned before the runtime step and refused --runtime-only, so no libwg_runtime.so existed for a gfx11 build. The runtime reads its code objects as <arch>.<name> (device.cpp, never the unqualified file), and a gfx12 build makes those copies as it embeds the files; a gfx11 build had neither. Unless --no-runtime or --sources is given (as before, neither builds a runtime), a gfx11 build now copies every code object to <arch>.<name> (link_arch_objects) and compiles the runtime with embed=False: no embedded table, since the gfx11 set is not the gfx12 list that write_embedded requires, and the runtime reads the files from the folder. --runtime-only does the same step alone for a gfx11 target; --motion-outres-only stays gfx12 only (new message: the gfx11 target has no motion code objects). gfx12 builds are untouched: runtime-only still embeds, and the guard's --no-runtime build is the same. The runtime's own sources are not changed here. clean/README.md follows. Not done: nothing embeds a gfx11 code-object set (which files a release would ship for gfx11, and whether it ships at all, is the maintainer's decision), and the Windows runtime_command (D3D12 interop sources) is not exercised for gfx11. Tests (test_build_targets, compiler replaced by a function that writes each output): both gfx11 names make the runtime with no -DWG_EMBEDDED_TABLE and no embedded call, one <arch>.<name> copy per code object with the same bytes and no other gfx* file; --no-runtime and --sources make neither; --runtime-only links the existing objects and does not embed for gfx11 but still embeds for gfx1201. Red on purpose, then restored: the runtime step switched off (2 failures), the copy removed (3 errors), gfx11 --runtime-only embedding (1), gfx12 --runtime-only no longer embedding (1). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… built for gfx1100 With the runtime's gate, the packing rows and the build of the previous changes, test_gpu_pipeline no longer skips on a gfx11 build that has its runtime: its 5 tests run on the RX 7900 XT through HIP, default kernels (swin.co, swin_bneck6 for b0..b5, post_recon2_r_band + post_filter_c1_s4_band, separate dec1 / dec0), synthetic weights and frames only. The docstring no longer says the repository lacks the runtime. New: test_the_runtime_picks_the_rows_of_the_cards_family. A game's interop never sets WgRtConfig.d_row, so the runtime must choose the gfx11 table itself. Pipeline takes d_row="runtime" (the config's table stays 0, the reference table of check_packing is the build's family), and the test checks that every stage blob equals packing.py's on the gfx11 table, that enc1's blob is not the gfx12 one, that the stage blobs equal those of a pipeline given the table, and that a frame of two new pipelines is the same bits (new ones: the cached pipelines have run frames and the frame counter is part of the result). Red on purpose, then restored: a runtime that ignores the device's family fails it with "runtime packing of enc0 differs from pack_weights". Result on the card (OK, expected failures=1): output finite in all cases and frames; stage by stage against the gfx11 lane emulator on the card's own inputs, one-code flips 0.000 to 2.368 % per stage (enc1.merged 45x29 the highest), more than one step at most 0.651 %, b0..b5 chain 0.000 %; the output pixels against the exact (gfx12-arithmetic) emulator frame are 99.97 % different at 64x40 (the expected failure, unchanged, synthetic network). Whole frame, median of 50, three runs: 64x40 -> 96x60 3.120 3.066 3.044 ms; 45x29 -> 67x43 3.073 3.182 3.169 ms; 1706x960 -> 2560x1440 18.664 18.822 19.757 ms (the card is shared with another project's jobs; the scratch run of the previous change gave 17.8 to 18.0 ms). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…11 (S1) wg_platform.hlsli takes WG_GFX (12 by default, unchanged; 11 for RDNA 3). Under 11: no FP8 matrix op, no FP8 convert and no _ags pack (wg_ags.hlsli hides the FP8 codes); wg_wmma_f16_k8 is one raw F16 instruction on fragments replicated across the half-waves, wg_wmma_f16_k16 chains two of them (a K=16 step), wg_wmma_f16 keeps the gfx12-shaped operands of HIP's helper on top of them, wg_d_row is the 2i+h row table, SW_UNPACK_AGS is 0, and the clamp form of the GELU LUT converts with f32tof16 (RADV rounds a (half) cast to nearest even). build_dx12.py --gfx 11 defines WG_GFX=11, writes <out>/dx12-gfx11/ with a "gfx": 11 manifest key, builds only declarations with gfx=11 (default gfx=12) and skips RGA. Tests: test_build_gfx.py (8, no tools needed) and test_gfx11_helpers.py: headless on the card under Wine (vkd3d-proton) against numpy and a HIP gfx1100 kernel, bit for bit for the raw op, the K=16 chain and wg_wmma_f16; convert is RNE; xor16; row table; rounding modes. They skip without the tools named in the module doc. Red on purpose then restored: row table i+8h, K=16 chain without its second op, swapped K halves, xor16 skipping a lane, swapped convert dwords, FP8 code visible on gfx11, build without the WG_GFX define. Guard: the 46 gfx12 DXIL files are identical to the baseline (manifest too). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…sts, PSO sweep (S3) wine_headless_run.sh runs a Windows program under Wine with a null-graphics prefix, a private runtime folder and the GPU lock, and refuses anything that could open a window (a prefix with a graphics driver, /tmp or /run/user scratch). build_kernel_replay.py cross-builds kernel_replay.exe, pso_sweep.exe and gfx11_probe_host.exe with llvm-mingw. pso_sweep.exe creates each kernel's pipeline with the runtime's buffer or direct root signature (the runtime itself is not used: its device refuses a card that is not RDNA 4); pso_sweep.py runs it over a built folder, records a driver crash per kernel and classifies each DXIL by FP8 channel codes (dxc -dumpbin): a PSO with an FP8 op must fail on RDNA 3, any other must be created. Result on the RX 7900 XT (vkd3d-proton, extended emulation): the 46 gfx12 DXIL files give 23 created (all without FP8), 17 failed with 0x80070057 and 5 crashed the driver (the swin c96 add_* blocks: a page fault inside the driver, reproducible alone), 1 tex kernel not tried; none unexpected. The 8 gfx11 helper kernels of test_gfx11_helpers all create. kernel_replay.exe builds and starts but its device refuses the card until the family gate opens. test_wine_headless_run.py: 4 tests; red on purpose: scratch check removed, null-graphics check removed. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…out in the CPU model (S2) native_driver_probe_f16 (gfx=11 only) runs D = A x B + C with FP16 operands as a K=16 step of two chained instructions on the replicated fragments of the gfx11 layout. The gfx12 kernel native_driver_probe is untouched (its DXIL is byte for byte the same, and it is not built for gfx11: it has an FP8 instruction). wmma_probe.h gets namespace gfx11: buffer layout, fill, expected results (the same matrices, rows 2v+h) and a compare with a tolerance of 1/4096, because the card's F16 accumulation differs from the exact sum in the last bit in 57 of 256 results (measured; equal to HIP's instruction). test_wmma_probe.cpp: 20 new checks (the right layout passes, six wrong hardware variants fail, replicated-half variants pass, all lane-pair swaps of A, B and C fail exactly when they change what the instruction reads, all element-pair swaps fail, the tolerance accepts last-bit differences and reports 2e-3 and NaN); the gfx12 checks are unchanged. Red on purpose: fragment 1 given the K of fragment 0 (6 failures), d_entry rows v+8h (1). test_gfx11_probe.py (3 tests): builds the kernel with build_dx12.py --gfx 11 and runs it on the card under Wine with wmma_probe_dump's operands: results within the tolerance; the gfx12 buffer fails. Red on purpose: the kernel with one instruction instead of two. Guard: the 46 gfx12 DXIL files are identical to the baseline. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…y normal build (S4) A compile-time macro, default off and never defined by clean/runner/build.py, lets the native D3D12 device, the executor choice and the Wine log line take an RX 7900 series board (PCI 1002:744C, Navi 31, gfx1100; the id read on the development card, shared by the XT, XTX and GRE; no other RDNA 3 id, 7448 included). The mechanism is one of the options listed for the maintainer in the RDNA 3 plan (a macro and a local build); the maintainer may prefer another (environment variable, ini key, a PCI id rule in the family function, or the gate left closed in main), and this change does not choose among them. It exists so that the D3D12 kernels for gfx11 can be built and tested before that decision. Without the macro nothing a player runs changes: rdna4_identity is untouched (the Ray Reconstruction runtime still asks only it), dev_gfx11_identity is constant false, dx12_folder answers "dx12", the device reports no target, the probe is the gfx12 one, and the order of the manifest read and the device is as before. With the macro on: - rdna4.h: kDevGfx11, gfx11_device_id, dev_gfx11_identity (same rule as rdna4_identity: the driver's ids, else the Wine fallback's), dx12_folder(arch), dev_gfx11_arch(); the Wine line and describe_identity know a taken board. - executor_select.h: the default rule takes the board (D3D12 executor). - device_d3d12.cpp: both identity gates take it; the device reports target gfx1100, reads <build>/dx12-gfx11/manifest.json after the device (the family comes from the adapter) and runs native_driver_probe_f16 with wmma_probe::gfx11 (F16 only, compared within 1/4096). - runtime.cpp: the device is made before any manifest is read, the manifest of a build is read through dx12_manifest(), the accumulator row table comes from the device's target as for HIP (already so), and the native-direct check accepts the gfx11 table. The tolerance: the card's F16 accumulation is not exact (57 of 256 probe results differ from the exact FP32 sum, by at most 2^-15 here, equal to HIP's instruction). 1/4096 is two ulps of the largest expected value (1331) and 1/2500 of the smallest gap between distinct expected values (0.607), so a wrong layout cannot pass. Tests: test_dev_gfx11.cpp (28 checks without the macro, 29 with; both builds pass) and tests/test_dev_gfx11.py (builds both, 6 source pins incl. that no build or release file mentions the macro). The existing test_executor_select.cpp: 234 pass without the macro; with it exactly the 12 cases that assert the RX 7900 XTX id (744C) is refused fail, as they must. On the card, headless under Wine with the system DXGI of DXVK in a scratch prefix (the Wine fallback ids, as a player's Proton gives): native_probe_test (new target of build_kernel_replay.py; the borrowed-device path) is refused without the macro and passes the F16 probe with it; WG_TEST_DRIVER_PROBE_FAIL=1 fails at float 7; kernel_replay runs native_driver_probe_f16 as a chain, 256 results within tolerance. Red on purpose, then restored: 11 host mutations and 6 Wine mutations (see the test files' pins). Guard: the 46 gfx12 DXIL files and the 42 HIP objects are unchanged (no kernel touched). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…fx12 kernels on RDNA 3, with a rounding check
Inside the WG_DEV_GFX11 opening a second developer-only choice lets a gfx1100 board run
the UNCHANGED gfx12 kernel set (dx12) through dxil-spirv's emulation of the FP8 matrix
operations in the raw-fragment path (DXIL_SPIRV_CONFIG=wmma_rdna3_raw_fp8 in a patched
vkd3d-proton). The mechanism is the environment variable WG_DEV_GFX11_SHORTCUT=1, read
only in a build made with the macro (rdna4.h: without it dev_gfx11_shortcut() is constant
false and the library never asks the environment). It is a placeholder: the maintainer
decides the mechanism later (ini key, option, a PCI id rule, or no shortcut at all), and
shipping a patched dxil-spirv is a product decision this change does not make.
Without the macro nothing changes for a player. With the macro and the variable unset
nothing changes either (the S4 behaviour). With both, the device reports no target (the
gfx12 folder dx12 and the gfx12 accumulator table, as on RDNA 4; the emulation keeps the
gfx12 accumulator layout), and:
- the borrowed-device probe is the gfx12 one (native_driver_probe, FP16 and emulated
FP8), compared within wmma_probe::gfx11::kTolerance: the emulation multiplies in the
FP16 instruction, whose accumulation is not exact on gfx1100 (same reasoning as S4).
- a rounding check runs at device creation, for owned and borrowed devices: the
kernels round to FP16 through an F32 -> F16 -> F32 pair of the matrix conversion, and
the driver's compiler folds such a pair away unless the conversions are exact (found
with the shortcut probe: a fold gives encoder features one FP8 step off). The kernel
native_round_probe (gfx=11 only, so the gfx12 set keeps its 46 files) converts 256
inputs that all have bits below FP16's precision, including exact ties of both
parities; the device compares bit for bit with the nearest-even result computed on the
CPU (wmma_probe.h, round16) and refuses with a log line otherwise. The kernel is read
from <build>/dx12-gfx11/manifest.json (read_manifest takes an optional kernel name).
Tests: test_shortcut_probe.cpp (27 checks, built with and without the macro) and
tests/test_dev_gfx11_shortcut.py (both builds plus 5 source pins). On the card, headless
under Wine: the shortcut passes the probe and the rounding check; with
DXIL_SPIRV_CONFIG unset the compiler folds the pair and the device is refused ("the
driver does not round F32 -> F16 -> F32"); WG_TEST_ROUNDING_FAIL=1 gives the same
refusal. Red on purpose, then restored: 8 mutations (see the test files' pins).
Guard: the 46 gfx12 DXIL files are unchanged.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…ess Wine, compared with the HIP whole frame (S12) frame_host.cpp is a small Windows host (cross-built with the other replay hosts by build_kernel_replay.py) that loads wronggpu_runtime.dll, creates the native D3D12 backend (backend 3) on a synthetic raw-tensor model, runs frames from input files, dumps the named buffers of each frame and times frames with the runtime's own timestamps. Nothing reads a model.wgcm. It exists because the whole-frame machinery (compare.py, time_frames.py) drives the runtime library from Python on Windows, and a headless run under Wine on Linux has no Python on the Windows side. frame_parity.py prepares the model, the inputs and the host's spec (the same synthetic frames as compare.py), runs the HIP or lane-emulator whole frame natively through pipeline.py, and compares every named buffer and the output image of two runs: bytes that differ per buffer (buffers that a fusion leaves unwritten, stale ones, and the decoder's other format are listed and not counted, as in compare.py), the output in FP16 steps and relative RMS, and the GPU times. It measures; it is not a gate, since the output standard of an RDNA 3 route is the maintainer's decision. Checked: a run against itself gives 0 differing bytes and byte-equal output; the D3D12 shortcut frames at 64x40, 853x481 and 1706x960 run end to end (see the S12 report). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… text, and a guard checks it The RDNA 3 changes in swin.hip, encoder.hip and wg_platform.h leave the gfx12 program the same, but not its text: the gfx11 operand types, fragment loads, store forms and wave id helper were written for every target, and the patch merge and expand stores gained braces. A different compiler can schedule different text differently: on the RDNA 3 line, ROCm 10.1 (clang 24) moved instructions in swin.co and swin__bmegahg.co for one restructured store, although the program was unchanged. "The same instructions under today's compiler" is not a promise RDNA 4 can rely on; the text is. Every gfx11 form now sits under WG_SWIN_G11 (gfx11 or the emulator) with main's earlier text in the #else. In wg_platform.h the v4h / v16h types and wg_xchg16_u exist only for gfx11 and the emulator; encoder.hip gets the same treatment for its gfx11 store. clean/tests/check_gfx12_source_unchanged.py preprocesses every shipped HIP compile for gfx1201 (each tree's own build.py with its compiler calls recorded, FG's three objects, every other clean/**/*.hip) and requires the text to match the base tree's token for token; its unit tests plant an edit to prove it can fail. dx12/tests/gfx11_hip_ref.hip, a host program of the gfx11 helper tests that no gfx12 build compiles, is named as unbuildable in the test and as a new form on the command line. Verified (ROCm 10.1, clang 24; only HIP kernel sources and the guard changed, no D3D12 or runtime file): - guard against the base on main (91436c4): 160 forms, 0 failed. The same guard with the unfixed branch head (0cd1a91) as the tree: 160 of 160 failed. - gfx1201 objects (M, K, L, FG: 42) of the fixed tree and of 91436c4: byte-identical 42 of 42. (The unfixed head 0cd1a91 also gave 42 of 42 identical objects under 10.1: the source differed, the code objects did not.) - gfx1100 objects (--variants f16wm,mgx: 11) before and after: byte-identical 11 of 11. - guard unit tests 14 OK; the emulator tests test_kernels_emu_gfx11, test_e4_gfx11_emu and test_kernels_emu: 36 OK. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
….hlsli and wg_ags.hlsli are main's again On main aa85515, model K's softmax self-check records kSourceDigest, a SHA-256 over the shader headers the softmax kernels include (wg_platform.hlsli and wg_ags.hlsli among them). S1's gfx11 edits of those two files changed the digest and build_dx12.py refused to build. Moved: everything gfx11 in wg_platform.hlsli (WG_GFX default and checks, wg_in_upper_half, wg_xor16_4, wg_wmma_f16_k8 / _k16, the gfx11 wg_wmma_f16, wg_d_row, the gfx12 wg_d_row) and in wg_ags.hlsli (the WG_GFX default and the #if hiding the FP8 codes) into the new hlsl/wg_gfx11.hlsli. A gfx11 kernel includes it instead of wg_platform.hlsli (it includes that one itself); gfx12 builds of it only include the platform layer. Under WG_GFX 11 it renames the platform layer's gfx12-shaped wg_wmma_f16 while including it, and hides the FP8 identifiers again with #undef / #define (a use does not compile, as before). native_game_helpers.hlsl, tests/gfx11_helpers.hlsl and the compile check include it. wg_gelu_lut.hlsli, swin_common.hlsli and build_dx12.py are not in the digest and stay. Verified: smx_selfcheck_digest.digest() equals the committed header again; the two gfx11 probe DXIL files are byte-identical to the ones built before the move; test_build_gfx (8) and TestGfx11Compile (4, including the FP8-is-hidden negative controls) pass; the card tests skip (no GPU in this slice). Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
… of compiling it for gfx1201
Since the gfx12 source of every HIP kernel is main's text again, the gfx11 exchange wg_xchg16_u exists only for gfx11
and the emulator. test_exchange_is_one_permlane_on_gfx11_and_bpermute_on_gfx12 still compiled it for gfx1201 and so
failed ("use of undeclared identifier 'wg_xchg16_u'"). Its gfx12 half now checks what the design promises: wg_xor16 on
gfx1201 is the one ds_bpermute exchange of main's text (no permlane), and wg_xchg16_u does not compile there. The gfx11
half is unchanged.
Verified (ROCm 10.1, clang 24): the test is red before (the old test fails on the parent commit) and green after; with
the wg_platform.h of the commit before the gfx12-inert fix (wg_xchg16_u for every target) the new test fails, restored
it passes. Without a ROCm clang the whole module skips, as on CI's Windows runners.
Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
…Wine runner test skips off Linux build_dx12.py's digest label used os.path.relpath, which raises ValueError when a source (a test's temp folder on C:) and the repository (D:) are on different drives. A source outside the repository is now labelled ext/<file name>. A new test in test_build_gfx.py simulates the cross-drive error (red before the fix, green after). test_wine_headless_run.py runs a bash script, so it is skipped unless on Linux. Signed-off-by: Jonathan Shin <85975340+j0nsh1n@users.noreply.github.com>
c3f6b75 to
0971e02
Compare
Draft, for visibility. Model M on RDNA 3 (RX 7900 XT, gfx1100): the HIP kernels on Linux, and a developer-only D3D12 route that runs in a game under Proton. Nothing here changes what a player gets: every RDNA 4 code object and DXIL file builds byte-identical, and the RDNA 3 openings are off in every normal build. Several open decisions are yours (listed at the end); I am not asking for a merge.
What this changes and why
RDNA 3 has no FP8 matrix instruction. These commits run Model M's arithmetic through the F16 one (
v_wmma_f32_16x16x16_f16): E4M3 operands are widened to F16 exactly, the FP32 -> E4M3 roundings stay exact in software, so each stage stays close to the RDNA 4 result (not bit-identical: the F16 instruction's FP32 sums round differently, measured in #28's analysis).In order (37 commits, each signed off):
12ef8dcA stand-alone fix:swin.hipbuilds for the host emulator again (the rawreadfirstlaneand an+sasm constraint broke it on main).gfx1100target with an exact-name family map (clean/runner/family.py, andclean/runtime/target_family.hkept equal to it by a test); a gfx11 base inwg_platform.h(half-wave exchange, a faster exact FP32 -> E4M3 conversion, the F16 matrix wrappers) and a gfx11 mode of the lane emulator; card probes.f16w*,mgx,c128w7).v_wmmachain into the branch of a per-lane store (if (token < ntok)), so on partly valid tiles the output had NaN or wrong values (the emulator cannot show it). Fixed by unguarded stores and a wave-uniform loop bound;clean/runner/exec_scan.pyscans the disassembly for matrix ops and half-wave exchanges under a lane mask (tests red on the old code).WG_GFX 11helpers andbuild_dx12.py --gfx 11; a headless Wine runner and Linux/mingw builds of the replay hosts; an F16-only driver probe; and, behind a compile-time macroWG_DEV_GFX11(off by default; a test pins that no build or release file defines it), an opening of the RX 7900 XT for the D3D12 device. Inside it, a "shortcut" mode (WG_DEV_GFX11_SHORTCUT=1, read only in such a build) runs the unchanged gfx12 kernel set on RDNA 3, with a start-up check that the driver rounds F32 -> F16 -> F32 as the encoder needs.What it needs outside this repo (not in this PR)
The shortcut runs because of a local dxil-spirv change on top of the raw-fragment support of dxil-spirv #362: under a new
DXIL_SPIRV_CONFIGvalue it emulates the FP8 raw-fragment matrix ops on RDNA 3 (F16 ops, the lane layout RDNA 3 needs, exact FP8 rounding), and vkd3d-proton built withenable_extended_emulation. Whether and how to offer that upstream is open; it overlaps #362 indxil_ags.cpp.How it was tested
Open decisions (yours; none chosen here)
WG_GFX/ family macros (as here) or separate*_gfx11files.Checklist
Signed-off-by:line (DCO 1.1).REUSE.toml.