ci(rocm): self-hosted gfx1151 runner that builds, links and smoke-tests ROCm changes #1811

Part of #1801. Phase 3. Depends on #1802 (closed, landed in PR #1818).

Status (2026-09-29)

Everything this issue can deliver without hardware has landed: the job, its path filter, the visible parked state (PR #1989), and a local gate that runs the same smoke (PR #2008). The one thing left is registering a self-hosted runner labelled rocm-gfx1151 and setting the repository variable ROCM_CI_ENABLED to true. That cannot happen now: there is no spare AMD machine to dedicate to it, and registration also needs org admin. Until then make verify-rocm on an AMD host is the ROCm gate, run by hand, and the three acceptance criteria that need a live job stay unchecked.

Context

No workflow has an AMD runner. (Outdated: PR #1989 added the rocm-build job, parked because no runner is registered.) Because the ROCm side of every MLX pin bump now lives in src/lib/mlx-cpp/patches-rocm/, a bump that breaks the ROCm build or runtime would go unnoticed without a job that links and runs it. The CUDA history shows the risk of green-but-hollow jobs: on the 81ba1c6a bump, CUDA checks were green while nothing linked (cargo check does not link, the sm_70 job skipped on a CUDA 13 runner, and the xla-link job had no path trigger for pin changes).

A Linux host with a Radeon 8060S (gfx1151) is available as a self-hosted runner. It also runs other GPU services, so jobs must account for shared GPU memory and load. (Update: no spare machine is available to register as that runner, so it has not been registered.)

Scope

A ROCm CI job on a self-hosted runner that builds with --features rocm, links, and runs a short generation, triggered by every change that can affect the ROCm build.

Implementation plan

1. Register the runner with a dedicated label (for example rocm-gfx1151) and document the host prerequisites (ROCm version, OpenBLAS/LAPACKE, Rust toolchain, disk for target/ and _deps/). Open, blocked on hardware. The label is fixed as rocm-gfx1151 (runs-on: [self-hosted, rocm-gfx1151], .github/workflows/ci.yml:1646). Host requirements live in the job's own comments next to it, as the GB10 runner's do (PR ci(rocm): build, link and generate on a self-hosted gfx1151 runner #1989): MLX_ROCM_ARCHITECTURES: gfx1151 set explicitly, a persistent CARGO_TARGET_DIR under $HOME/.cargo-target/mlxcel-rocm-ci, and a 180-minute timeout for a cold build.
2. Add a rocm job: cargo build --release --features rocm, then mlxcel generate -n 10 --temp 0 on the qwen3-0.6b-4bit CI fixture from the public release ci-fixtures/qwen3-0.6b-4bit-v1, run with LD_LIBRARY_PATH unset, asserting a non-empty, non-garbage output and a GPU device in the diagnostics. Done in PR ci(rocm): build, link and generate on a self-hosted gfx1151 runner #1989, body shared by PR test(rocm): add a local verify-rocm gate with a shared build-and-generate smoke #2008. The job is rocm-build ("ROCm build and generate", .github/workflows/ci.yml:1630-1714). It builds, downloads the same fixture, and runs scripts/ci/rocm_smoke.sh, which generates 32 tokens at --temp 0 with LD_LIBRARY_PATH unset (scripts/ci/rocm_smoke.sh:87-92) and asserts Runtime device: GPU, custom kernel backend: rocm, a HIP architecture gfx line, no CUDA compute capability, at least 16 generated tokens and at least 8 word-like tokens of generated text (:105-126).
3. Add a rocm path filter in .github/workflows/ci.yml (filters are defined around lines 104-160) covering src/lib/mlx-cpp/patches-rocm/**, src/lib/mlx-cpp/CMakeLists.txt, src/lib/mlxcel-core/build.rs, src/lib/mlxcel-core/build_support/**, the bridge cpp/ sources and Cargo.toml feature changes. The MLX pin line must trigger it. Done in PR ci(rocm): build, link and generate on a self-hosted gfx1151 runner #1989. The filter is at .github/workflows/ci.yml:189-203: all of src/lib/mlx-cpp/** minus the Metal and CUDA overlays (so patches-rocm/ and the pin in src/lib/mlx-cpp/CMakeLists.txt are both in), plus build.rs, build_support/**, src/lib/mlxcel-core/cpp/**, the root build.rs, **/Cargo.toml and Cargo.lock.
4. Fail the job (not skip) if the runner is offline for a pin-bump PR, or make the skip visible in the PR checks. Done in PR ci(rocm): build, link and generate on a self-hosted gfx1151 runner #1989, the "visible" variant. A job whose runner does not exist is queued for up to 24 hours rather than skipped, which would hold every overlay PR's run open, so rocm-build is gated on vars.ROCM_CI_ENABLED == 'true' (.github/workflows/ci.yml:1642-1645). The hosted rocm-ci-status job (:1727-1752) runs on every change that matches the filter and emits a "ROCm gate parked" warning naming the two activation steps. It reports the variable only: querying runner registration needs an administration scope that GITHUB_TOKEN cannot carry in a workflow file, so that was dropped before merge. It never fails a PR.
5. Keep the job's GPU footprint small and document how it coexists with other GPU tenants on the host. Done in PRs ci(rocm): build, link and generate on a self-hosted gfx1151 runner #1989 and test(rocm): add a local verify-rocm gate with a shared build-and-generate smoke #2008. One small checkpoint, one 32-token generation, a persistent target directory, and a memory preflight in scripts/ci/rocm_smoke.sh:55-72 that reports available memory and names the largest resident processes before building (co-tenant llama-server processes were measured holding 25.5 GiB of about 30 GiB).
6. (Added) Local substitute until a runner exists. Done in PR test(rocm): add a local verify-rocm gate with a shared build-and-generate smoke #2008. make verify-rocm (Makefile:804) mirrors make verify for ROCm: crate versions, kernel dtype keys, kernel port dispatch, llama-compat manifest, fmt, verify-clippy-rocm, then verify-rocm-smoke (Makefile:790, the same scripts/ci/rocm_smoke.sh the job runs, so the two cannot drift) and verify-test-rocm.

Acceptance criteria

• A PR touching patches-rocm/ or the MLX pin runs the ROCm job, which builds, links and generates. (Job and filter exist, PR ci(rocm): build, link and generate on a self-hosted gfx1151 runner #1989; blocked on a registered runner and ROCM_CI_ENABLED.)
• The job log shows the binary linked against the ROCm libraries and ran on the GPU. (The assertions exist in scripts/ci/rocm_smoke.sh; blocked on a runner. Locally, make verify-rocm-smoke showed this on gfx1151 in PR test(rocm): add a local verify-rocm gate with a shared build-and-generate smoke #2008.)
• A deliberately broken overlay file fails the job. (Blocked on a runner.)
• Unrelated PRs do not trigger it. (Path filter .github/workflows/ci.yml:189-203 plus the ROCM_CI_ENABLED gate, PR ci(rocm): build, link and generate on a self-hosted gfx1151 runner #1989.)

Activation, once hardware exists

1. Register a self-hosted runner with the labels self-hosted and rocm-gfx1151 (org admin).
2. Set the repository variable ROCM_CI_ENABLED to true.
3. Open a PR that touches src/lib/mlx-cpp/patches-rocm/ and confirm ROCm build and generate runs; then break one overlay file on a throwaway branch and confirm it fails. That closes the three open criteria.

References

• .github/workflows/ci.yml (rocm filter 189-203, rocm-build 1630-1714, rocm-ci-status 1727-1752)
• scripts/ci/rocm_smoke.sh, Makefile:790 (verify-rocm-smoke), Makefile:804 (verify-rocm)
• CI fixture: public release ci-fixtures/qwen3-0.6b-4bit-v1

Refresh log

2026-09-29

• Marked plan items 2 to 5 done with their landing PRs (ci(rocm): build, link and generate on a self-hosted gfx1151 runner #1989 for the job, filter and parked-state report; test(rocm): add a local verify-rocm gate with a shared build-and-generate smoke #2008 for the shared smoke script and memory preflight) and added item 6 for the local make verify-rocm gate.
• Checked off "unrelated PRs do not trigger it"; the other three criteria need a live runner and stay open.
• Stated plainly that the only remaining work is registering a rocm-gfx1151 runner and flipping ROCM_CI_ENABLED, which is blocked because no spare AMD machine exists.
• Replaced the stale filter location (104-160) with the verified .github/workflows/ci.yml:189-203, and noted that rocm-ci-status reports only the variable, not runner registration.