Skip to content

Builds: the official HIP SDK only, never one from the environment - #239

Draft
phantoslayer21 wants to merge 4 commits into
mainfrom
kspeed/hip-toolchain
Draft

phantoslayer21 wants to merge 4 commits into
mainfrom
kspeed/hip-toolchain

Conversation

@phantoslayer21

@phantoslayer21 phantoslayer21 commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Milestone 0.6.6 (not merged before the 0.6.5 release). Draft. Every HIP build and every HIP runtime load of the Python tools uses an official AMD HIP SDK install, checked in one place. Nothing is taken from the environment or from PATH.

Out of scope: the native runtime's (clean/runtime/device.cpp) and the installer's (clean/installer/src/wg_core/checks.cpp) own HIP runtime lookup. Changing it changes product behaviour, so it has its own issue for 0.6.6, #242: System32 first, then an official SDK by full path, no bare-name search, HIP_PATH only when it resolves to an official install, and the installer probe under the same rules.

Before this change, build.py, k_build.py and about 30 other scripts and tests used ROCM_PATH as their default --rocm. find_clang and find_host_cxx fell back to PATH, and hip.py fell back to a bare runtime library name that the loader searches for.

The policy (clean/runner/sdk.py)

  • SDK: an official install, C:\Program Files\AMD\ROCm\<version> on Windows, /opt/rocm-<version> or /opt/rocm elsewhere, with a compatible 7.x version. Without --rocm, the newest one in numeric version order. /opt/rocm counts with the version it points to (so a newer /opt/rocm-<version> still wins); a standalone /opt/rocm counts with its .info/version, or ranks last without one. Never ROCM_PATH, HIP_PATH, LLVM_PATH, HIP_DEVICE_LIB_PATH, HIP_CLANG_PATH or PATH.
  • Compiler: the SDK's own. A bare --clang clang means the SDK's bin\clang.exe; an absolute path must lie inside the SDK.
  • Override: any other SDK folder or compiler needs --force-sdk, which approves exactly that folder or file for the process. Every script with --rocm has it (sdk.add_force_argument / sdk.apply: --rocm and --clang are resolved right after parsing), and passes it on to the child processes it gives --rocm. The LUT oracle builders that run the HIP compiler themselves (clean/dx12/gelu_lut/run.py, clean/dx12/f16lut/run.py, clean/dx12/f16lut/run_in.py) take --rocm / --force-sdk the same way, the SDK's clang from sdk.compiler, and call sdk.check before compiling (no fixed 7.2 path).
  • Host emulator: build.py --emu-only and k_build.py --emu-only build the CPU emulator with the host compiler alone; the HIP SDK is resolved only when a device build (or the runtime library) needs it.
  • One boundary: device_command calls sdk.check(), so every HIP compile passes it, whichever entry point started it (build.py, k_build.py, post_ablate.py, l_c32_check.py, k_port_check.py, fg_d3d12_check.py, ...). find_clang and find_host_cxx go through sdk.compiler().
  • Runtime: hip.py loads the HIP runtime library only from the checked SDK, by full path, with no bare-name fallback, and fails closed. Hip(rocm_dir) is under the same policy.
  • Logs: every toolchain prints one banner line naming the SDK, the compiler and the device-library folder. Official folders appear as they are; a custom one appears only as "custom (forced, id )": in the banner, in refusals, in the echoed commands, and in compiler output (build.run, run_post_w64 and its report file, post_ablate, post.build_wave64). Redaction matches the path as given (also relative), absolute, resolved, the 8.3 short form and the ~ form, with either separator and any case.
  • Every other script and test that defaulted to ROCM_PATH (or WG_K_TEST_ROCM) uses the official SDK. BUILDING.md states the rule.

Tests (clean/tests/test_sdk.py, CPU, fake SDK folders, no compiler started)

  • Discovery: the newest compatible version in numeric order (7.10 before 7.2); incompatible and incomplete installs are skipped; 7.2 missing; several versions; none installed.
  • Environment: the SDK variables are ignored.
  • Compilers: bare and absolute names are canonicalised to the SDK's file. Another compiler, or a bare name the SDK lacks (PATH would have one), is refused without --force-sdk and accepted with it.
  • Boundary: device_command and find_clang refuse an unapproved SDK or compiler. The entry points post_ablate.build_all, l_c32_check.compile_kernels, k_port_check.prepare and fg_d3d12_check.prepare refuse a custom SDK (also with an explicit compiler), and build.py / k_build.py refuse a custom --rocm or --clang without --force-sdk. No compiler is started in any of these.
  • Runtime: only the SDK's library, by full path; an official install without the library and an unapproved folder both fail with no load attempt.
  • Linux discovery: /opt/rocm pointing to 7.1 next to rocm-7.2 (7.2 wins; the link carries 7.1), pointing to the newest, a real symlink (skipped where the system has none), standalone with and without .info/version, standalone 6.x refused.
  • --force-sdk: post_ablate.py --rocm <custom> is refused without it and reaches the build with it. A scan of every tracked CLI with --rocm (61 files) checks that each one offers --force-sdk and resolves --rocm through the policy.
  • --emu-only: build.py and k_build.py build the emulator with no SDK installed (only the host compiler runs); without --emu-only they still refuse.
  • Direct compiles: every tracked file that runs the HIP compiler itself (--offload-arch) goes through sdk.resolve, sdk.compiler and sdk.check (build.py: sdk.check in device_command), with no fixed SDK path.
  • Each of these tests fails on the code before this change (checked by putting the old build.py, k_build.py and gelu_lut/run.py back).
  • Logs: a custom path appears in none of the refusal, the banner, the echoed command, run / run_post_w64 compiler output and report (failing and passing), or post_ablate's failure, in any of the spellings above; other names next to it stay untouched.
  • build.py: with every SDK variable pointing elsewhere, each device compile uses the SDK's clang, --rocm-path and --rocm-device-lib-path, every Model M source is compiled, and the build completes (no exception swallowed).
  • Scan: no Python file under clean/ or tools/ reads an SDK variable by os.environ[...], os.getenv, .get() or an alias. The scanner has negative-control fixtures; a failing git ls-files fails the test; fewer than 200 files scanned fails it.

test_build_cuid.py approves its dummy SDK for each test (plus a check that the boundary refuses it without that), and test_build_motion_define.py passes --force-sdk.

Whole CPU suite (python -m unittest discover -s clean/tests -t ., at this head, SDK variables removed from the environment): 279 tests run, 213 passed, 66 skipped, 0 failures or errors. The skips: 61 need Linux (lane emulator), 2 need clang++/g++, 2 need WG_NO_K_BUILD, 1 is the GPU smoke test. Every script with --force-sdk also starts and lists it in --help (61 files).

Provenance of the shipped HIP code objects (checked for this PR)

  • The build logs name the official clang and device libraries for every HIP compile behind the 0.6.5 candidate: Model M's pinned 0.5.5-rc1 objects (54 compiles, with the SDK variables stripped), and Model L's and Model K's objects built for the candidate. Ray Reconstruction ships no HIP code objects.
  • One shipped object per model was rebuilt from the candidate's source with the official SDK and compared byte for byte:
    • Model L l_post.co: identical.
    • Model K k_block.co: identical.
    • Model M swin.co: the code is identical. Only the compilation-unit id symbol (__hip_cuid_*) differs, because 0.5.5-rc1's build.py did not yet pass a fixed -cuid.

Not tested

  • A GPU run, and a Linux build. No kernel or runtime code changes.

- clean/runner/sdk.py: the SDK of every build is AMD's official HIP SDK folder; --rocm may name only that folder
  (another one needs --force-sdk); a banner names the SDK, compiler and device-library folders in every build log
- build.py / k_build.py: --rocm defaults to the official SDK instead of ROCM_PATH; find_clang never falls back to
  PATH, find_host_cxx not on Windows; children get --force-sdk when the parent had it; a release runtime build
  still reuses pinned Model L code objects unless --rocm is given
- every other script and test that defaulted to ROCM_PATH (or WG_K_TEST_ROCM) uses the official SDK; hip.py's
  default comes from sdk.py and its LLVM_PATH warning no longer prints the folder
- clean/tests/test_sdk.py: the default ignores ROCM_PATH / HIP_PATH / LLVM_PATH / HIP_DEVICE_LIB_PATH, another
  --rocm is refused without --force-sdk, clang is never taken from PATH, the banner, build.py's device compiles use
  the official device libraries with ROCM_PATH set elsewhere, and no Python file of clean/ reads ROCM_PATH as a default
- BUILDING.md: the SDK rule

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…d (review fixes)

- sdk.py: official installs discovered by version (C:\Program Files\AMD\ROCm\<version>, /opt/rocm-<version>, /opt/rocm),
  the newest compatible 7.x by default; the compiler canonicalised to the SDK's own (bare and absolute names), anything
  else approved only with --force-sdk; check() is the boundary; runtime_library() the only HIP runtime source; custom
  folders logged by id only (banner, refusal, command echo via redact)
- build.py: device_command calls sdk.check (every entry point: post_ablate, l_c32_check, k_port_check,
  fg_d3d12_check, ... pass it); find_clang / find_host_cxx through sdk.compiler; --clang validated; run() redacts
- k_build.py: the compiler through sdk.compiler
- hip.py: the runtime loaded only from the checked SDK by full path (no bare-name fallback); Hip(rocm_dir) under the
  same policy; fails closed
- test_sdk.py: discovery (newest by numeric version, 7.2 missing, none), compilers, the boundary and the four entry
  points, build.py / k_build.py refusals, the runtime loader failing closed, redaction, every build.py compile with
  the SDK's clang and flags (build completes), the SDK-variable scan with negative controls (indexed, getenv,
  aliases, other variables) and a failing git call failing the test

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…d compiler output (re-check fixes)

- every CLI with --rocm offers --force-sdk and resolves --rocm (and --clang) through the SDK policy right after
  parsing (sdk.add_force_argument / sdk.apply); children that get --rocm also get --force-sdk
- Linux: the newest /opt/rocm-<version> wins; /opt/rocm counts with the version it points to; a standalone /opt/rocm
  counts with its .info/version (ranked last without one)
- build.run / run_post_w64 / post_ablate / post.build_wave64: command echo, compiler output and the post_w64 report go
  through sdk.redact; redaction matches relative, resolved, 8.3 and home-folder spellings, either separator, any case
- test_build_cuid approves its dummy SDK; new tests for all of the above
- the native runtime's and installer's HIP runtime lookup stay out of scope (BUILDING.md, issue draft for 0.6.6)

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
…s no HIP SDK (re-check 2)

- clean/dx12/gelu_lut/run.py, f16lut/run.py, f16lut/run_in.py: --rocm / --force-sdk through sdk.resolve, the SDK's
  clang from sdk.compiler, sdk.check before the compile, logs redacted (no fixed 7.2 path)
- build.py and k_build.py resolve the HIP SDK only when a device build needs it: --emu-only builds the host emulator
  with the host compiler alone, also with no SDK installed
- tests: emu-only builds without an SDK (and the device build still refused); a scan that every file running the
  HIP compiler itself goes through the policy

Signed-off-by: phantoslayer21 <156617979+phantoslayer21@users.noreply.github.com>
@phantoslayer21 phantoslayer21 added this to the 0.6.6 milestone Nov 15, 2026
@j0nsh1n

j0nsh1n commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Some notes from building WrongGPU's HIP side on Linux, in case they help with the SDK policy here. This is about developers who build the HIP kernels or use the HIP diagnostic executor; since #225 players need no ROCm at all.

What Linux developers have today (checked 2026-11-17; corrections welcome):

Source ROCm Install location
Fedora 44 (and Nobara) packages 7.1.1 /usr/lib64/rocm, runtime in /usr/lib64
Arch extra (CachyOS builds on it) 7.2.4 /opt/rocm
Ubuntu 26.04 archive ships ROCm (version not checked) distro paths
AMD's own packages 7.2.x stable /opt/rocm-<ver>
AMD "TheRock" ROCm 10.1 release candidate only (10.1.0rc3 here) a self-contained tarball, anywhere (e.g. under the home folder)

So for now most Linux setups are on a distro 7.1-7.2 with distro paths, and 10.x is only available as AMD's relocatable tarball. With this PR as drafted (only /opt/rocm laid out AMD's way, anything else through --force-sdk), every Fedora developer and anyone on a relocated TheRock tarball would need --force-sdk, which may not be the intent. Accepting a ROCm root by its contents (clang under lib/llvm/bin, device libraries under amdgcn/bitcode, libamdhip64.so) rather than by path would cover all of them.

ROCm 10.1 (AMD clang 24) with current main: build.py --rocm <ROCm 10.1 root> works unchanged. The 52 gfx1201 code objects (M, K, L, FG) build deterministically (two builds, identical hashes), and the runtime's self-tests (memory round trip, FP8 and F16 WMMA, FP8 conversion) pass on an RX 9070 XT. The gfx1100 targets also still compile (the bundled math libraries in 10.1 are gfx120X-only, but WrongGPU uses its own kernels). Code objects differ from a clang 20 build, as expected, so byte-identity checks between builds must use the same toolchain on both sides.

Happy to help test a policy change on Fedora and with a relocated ROCm 10.1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants