ci: add riscv64-linux build via cross-compilation on ubuntu-24.04#621
ci: add riscv64-linux build via cross-compilation on ubuntu-24.04#621alexcrichton merged 5 commits intoWebAssembly:mainfrom
Conversation
I can't comment much on the rest of this PR, but note that Wasmtime does publish releases for the |
|
You're right, thanks for the correction. Wasmtime does publish The actual blocker is that Once that lands, the |
|
Could you expand on why the cross-compilation approach didn't work? Personally I'd prefer to use the stock github actions runners rather than a custom one since we've had trouble with custom providers in the past, and I also think that cross-compilation should be possible to get working. |
|
Cross-compilation wasn't tried and then abandoned; I just didn't go that route. I followed the same pattern as Your concern about custom providers makes sense. The RISE runner is third-party infrastructure, unlike the GitHub-hosted ARM runner. Cross-compiling LLVM for a riscv64 host from x86_64 is technically possible but needs a two-stage build: first building If you have a preferred direction (CMake cross-compilation with a riscv64 toolchain, or Docker buildx with QEMU), I am happy to try it. Just let me know what the setup should look like. |
|
Ah I originally saw f678512 and got my presumptions from there I think... Regardless, however, I do think that cross-compilation is the way to go here. We got lucky with the arm64 runners where right as there was a desire to have them GitHub officially added them to its selection of runners meaning we didn't have to deal with the cross-compilation bit for arm64. If you're willing, however, I'd prefer to add CMake configuration/etc to cross-compile LLVM (doing the whole two-stage build thing and whatnot) for riscv64. I suspect cross-compilation will be much faster than native hardware, I agree it'll be way faster than QEMU, and it'll also mean we don't have to rely on third-party runners. |
Rework from native RISE runner to standard ubuntu-24.04 runner with CMake cross-compilation, following alexcrichton's preference. Changes: - ci/docker/Dockerfile.riscv64-linux: Ubuntu 24.04 with crossbuild-essential-riscv64; sets CC/CXX to riscv64-linux-gnu-gcc so CMake detects cross-compilation and LLVM builds a native tablegen before cross-compiling the rest of the toolchain. - ci/docker-build.sh: use artifact-specific Dockerfile if one exists (ci/docker/Dockerfile.<artifact>), fall back to default; make the wasmtime volume mount conditional on WASI_SDK_CI_SKIP_SYSROOT. - .github/workflows/main.yml: new riscv64-linux matrix entry on ubuntu-24.04 with cross_cmake_args (-DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=riscv64 -DWASI_SDK_LLDB=OFF) and WASI_SDK_CI_SKIP_SYSROOT=1; handle cross_cmake_args in the cmake flags step. WASI_SDK_CI_SKIP_SYSROOT=1: the cross-compiled clang runs on riscv64, not on the x86_64 build host, so the wasm sysroot step is skipped. WASI_SDK_LLDB=OFF: avoids cross-compiling libedit/libxml2 in this first iteration; can be re-enabled as a follow-up. Closes WebAssembly#607 Signed-off-by: Bruno Verachten <gounthar@gmail.com>
91887df to
0d34ab1
Compare
|
Reworked to use CMake cross-compilation on a standard The approach: a new
|
alexcrichton
left a comment
There was a problem hiding this comment.
Looks good! I suspect there'll be a number of stumbling blocks to overcome in CI, but once that's green happy to take another pass.
|
|
||
| - artifact: riscv64-linux | ||
| os: ubuntu-24.04 | ||
| rust_target: riscv64-unknown-linux-gnu |
There was a problem hiding this comment.
This'll want to be riscv64gc-unknown-linux-gnu
There was a problem hiding this comment.
Good catch on the triple -- pushed a fix for that.
There was also a second stumbling block: setting CC/CXX as Docker ENV vars meant LLVM's native tblgen sub-build inherited the riscv64 cross-compiler too, which gave an Exec format error when the build tried to run llvm-min-tblgen on the x86_64 host. I moved the compilers to -DCMAKE_C/CXX_COMPILER cmake flags and added WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS with CMAKE_SYSTEM_NAME/CMAKE_SYSTEM_PROCESSOR to trigger LLVM's native/target split -- hopefully that is the right approach, but happy to rework if not.
The correct Rust target triple for riscv64 Linux is riscv64gc-unknown-linux-gnu, not riscv64-unknown-linux-gnu. The gc suffix denotes the G (general-purpose) + C (compressed) ISA extensions that the standard Linux ABI requires. Also fix the matching CARGO_TARGET_ env var name in the Dockerfile so Cargo picks up the cross linker for the correct target. Signed-off-by: Bruno Verachten <gounthar@gmail.com>
|
The job was failing at the Two places had the wrong triple (
The |
Setting CC/CXX as Docker ENV vars caused LLVM's native tblgen sub-build to also use the riscv64 cross-compiler, producing an riscv64 binary that fails to run on the x86_64 host: Exec format error: llvm-min-tblgen Fix: pass the cross-compiler via CMAKE_C/CXX_COMPILER cmake flags instead (cmake cache vars are not inherited by subprocess cmake invocations, so LLVM's native build finds the host compiler). Also pass CMAKE_SYSTEM_NAME=Linux/CMAKE_SYSTEM_PROCESSOR=riscv64 to LLVM via WASI_SDK_CI_TOOLCHAIN_LLVM_CMAKE_ARGS so LLVM sets CMAKE_CROSSCOMPILING=TRUE and triggers the native/target build split, matching the pattern used by the macOS matrix entries. Signed-off-by: Bruno Verachten <gounthar@gmail.com>
|
The next failure is a classic LLVM cross-compilation issue. Setting When Two changes (following the same pattern as the macOS matrix entries):
|
Signed-off-by: Bruno Verachten <gounthar@gmail.com>
|
Thanks! |
|
Thanks to both of you. @alexcrichton, your suggestion to go with cross-compilation was the better direction -- I should have looked at it from the start rather than defaulting to the native runner approach. And @tschneidereit, glad you caught the wasmtime point early, I had that wrong. |
Rework of the riscv64-linux CI build to use CMake cross-compilation on a standard
ubuntu-24.04runner, rather than a native RISE runner.What changed
ci/docker/Dockerfile.riscv64-linux(new):crossbuild-essential-riscv64in its package reposCC=riscv64-linux-gnu-gcc/CXX=riscv64-linux-gnu-g++so CMake detects cross-compilation and causes LLVM to build a nativellvm-tblgenfirst, then cross-compile the rest of the toolchainCARGO_TARGET_RISCV64_UNKNOWN_LINUX_GNU_LINKERfor Rust cross-buildsXDG_CACHE_HOME=/tmp/cacheavoids write permission issues in the containerci/docker-build.sh:ci/docker/Dockerfile.<artifact>if it exists, fall back to the defaultci/docker/DockerfileWASI_SDK_CI_SKIP_SYSROOT != 1.github/workflows/main.yml:riscv64-linuxmatrix entry:os: ubuntu-24.04,rust_target: riscv64-unknown-linux-gnucross_cmake_args: -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=riscv64 -DWASI_SDK_LLDB=OFFWASI_SDK_CI_SKIP_SYSROOT: 1cross_cmake_argsin the cmake flags stepWhy WASI_SDK_CI_SKIP_SYSROOT
The cross-compiled clang runs on riscv64, not on the x86_64 build host, so the wasm sysroot step is skipped.
Why WASI_SDK_LLDB=OFF
Avoids cross-compiling libedit and libxml2 in this first iteration; can be re-enabled as a follow-up.
Closes #607