Merge upstream wasi-sdk (through wasi-sdk-33): C++ exceptions, libunwind, wasip2/p3#239
Draft
lewing wants to merge 166 commits intodotnet:dotnet/mainfrom
Draft
Merge upstream wasi-sdk (through wasi-sdk-33): C++ exceptions, libunwind, wasip2/p3#239lewing wants to merge 166 commits intodotnet:dotnet/mainfrom
lewing wants to merge 166 commits intodotnet:dotnet/mainfrom
Conversation
Previously, the release process for wasi-sdk was undocumented and manual. The `RELEASING.md` document in this commit describes the steps necessary to publish a new version of `wasi-sdk` as a GitHub release and provides helpful scripts to automate some of the steps. With this in place, a future change could add a workflow trigger that allows running the steps automatically in GitHub actions. Keeping the steps as scripts in the repository, however, allows developers to re-run steps themselves in the (likely) case that something goes awry.
) Exclude dist-ubuntu-latest to prefer dist-ubuntu-bionic, which is compatible with wider distributions. cf. WebAssembly#273 (comment) WebAssembly#303
This change switches the MacOS build to target 10.12, an older version than currently supported. This would close dotnet#127. @sbc100 mentions that this is the setting used by Emscripten and related projects.
This change brings in several helpful PRs (e.g., WebAssembly/wasi-libc#397, WebAssembly/wasi-libc#399, WebAssembly/wasi-libc#401) in anticipation of creating a release based on LLVM 16.
* llvm-project: update to 16.0.0 release This changes updates the `src/llvm-project` submodule to the `HEAD` of `release/16.x`, the same commit used to [release] the LLVM 16.0.0 binaries. [release]: https://github.com/llvm/llvm-project/releases/tag/llvmorg-16.0.0 * fix: use only Clang's major version in install prefix Due to [a change] in LLVM, Clang will expect to find the `libclang_rt.builtins-wasm32.a` file in a path that only contains the major version (`16`) instead of the entire version (`16.0.0`) as was previously the case. This change modifies the `CMAKE_INSTALL_PREFIX` to use Clang's major version only. [a change]: https://reviews.llvm.org/D125860 * review: only use `llvm_version_major.sh` Since the `Makefile` can get by with only knowing Clang's major version, this change removes `llvm_version.sh` and sets `CLANG_VERSION` to use only the major part.
…ssembly#315) Sometimes a C codebase might have the odd file written in Assembly. We should force the use of Clang as the assembly compiler for these cases (which it is perfectly capable of doing), and make sure we pass on the WASI compiler triple the same way we do for C and C++.
WebAssembly/wasi-libc#403 fixed an issue with `a_barrier` that should be included in the next release of wasi-sdk. This change updates wasi-libc to the latest `HEAD` of `main` to include it.
This commit passes through `EXTRA_CFLAGS` to the libcxx build, along with `EXTRA_CXXFLAGS`. This models after the pattern in `wasi-libc` for passing these flags to thread more flags through.
This change refactors the script logic that checks if a workflow in fact built the commit matching our release tag out into a separate script. This is mainly an improvement in clarity.
When testing out these scripts on a fork, I realized that it was more convenient to use a single environment variable (e.g., `GITHUB_API_URL=... ci/draft-release.sh`) to change some of the script's input parameters.
I found the artifacts created to have incorrect version numbers; clearing the Actions cache should resolve this.
This patch enables using latest version of wasmtime for testing. This should also make it possible to running tests for wasm32-wasi-threads in the future.
This is a recommendation from WebAssembly#304 to build less of LLVM in CI. Perhaps it can speed up build times.
It looks like some macOS-specific CMake options to LLVM end up forcing an x86_64 build of some blake3 intrinsics (or something like that) which causes the build to fail on AArch64 Linux, for example. I found, though, that when removing these options it was then possible to build a toolchain for AArch64 Linux.
This changes the front-page documentation to: - use out-of-line links in more places - mention the need for `libclang_rt.builtins-wasm32.a` when using standard Clang - mention how to use a `wasi-libc` sysroot - explain the experimental status of `wasm32-wasi-threads
In WebAssembly#321, some OSX-specific `Makefile` additions to `LLVM_CMAKE_FLAGS` were skipped unless `make` is run on a Darwin OS. This allowed building wasi-sdk for aarch64. But, as reported in WebAssembly#336, this also broke arm64/x86_64 universal binaries that are built during CI. The reason for this is that CI's `main.yml` overrides `LLVM_CMAKE_FLAGS` to add caching but `make` will not append to a variable set on the command line. This changes uses the `override` keyword to append to such a variable, as suggested [here]. [here]: https://www.gnu.org/software/make/manual/html_node/Override-Directive.html
I want to include WebAssembly/wasi-libc#409 which is rather easy to hit when people are experimenting wasi-threads.
FreeBSD (like MacOS) does not support the `-executable` argument for `find`. This commit checks if the OS is FreeBSD, and if so, uses the same workaround as already in place on MacOS.
Also add some basic testing for the cmake toolchain file. Fixes: dotnet#181
No need to pass `WASI_SDK_PREFIX` value from the command line. Use `CMAKE_CURRENT_LIST_DIR` to assemble the value of `WASI_SDK_PREFIX` now.
``` error: Server does not allow request for unadvertised object f992bcc08219edb283d2ab31dd3871a4a0e8220e fatal: Fetched in submodule path 'src/config', but it did not contain f992bcc08219edb283d2ab31dd3871a4a0e8220e. Direct fetching of that commit failed. ```
* Bump llvm version to 17.0.1 * docker/Dockerfile: bump LLVM_VERSION to 17
Keeping it up-to-date
Updating to handle WebAssembly/wasi-libc@688a685 from wasi-libc
This updates to include WebAssembly/wasi-libc#709 and then additionally configures wasi-libc with `-DWASI_SDK_VERSION=...` to ensure that wasi-sdk builds will now have `__wasi_sdk_major__` defined along with `__wasi_sdk_version__`. Closes WebAssembly/wasi-libc#688
During the build step, cargo is needed but this was not listed under requirements.
Removes the `_start` export for the `wasm32-wasip2` target, purging the use of the adapter.
> **Note**: this PR is me weaving together the work of many others. For example @yerzham's and @cpetig's work on WebAssembly#565 pretty much shaped this PR. Thank you! This commit adds initial configuration support for building wasi-sdk with support for C++ exceptions in WebAssembly. A small test is included which is exercised in CI at this time, but otherwise this does not update CI to actually ship sdk builds with C++ exceptions enabled. Instead the intention here is to make it easier to test builds with support for C++ exceptions and centralize planning/development around this. The goal here is to get things compiling to the point that applications can be compiled. I haven't thoroughly tested C++ exceptions and as evident in this PR it still requires changes in LLVM. Some small logic is added here to apply a `*.patch` file to LLVM to avoid needing a new submodule fork or waiting for upstream changes. The intention is that this `*.patch` is short-lived once the changes are officially merged into LLVM itself. The `*.patch` included here contains llvm/llvm-project#168449 as well as another minor edit I found was necessary to get things compiling locally. Given the discussion on that PR it looks like once LLVM is updated with that merged the extra part of the `*.patch` won't be necessary. This PR notably does not enable shared libraries when exceptions are enabled. I don't know enough about how things are supposed to work to be able to fully diagnose the compilation errors I'm seeing if shared libraries are enabled. This is something I'd hope would be fixed before actually shipping exceptions support. This PR then additionally folds in [this gist][gist] for various bits of build logic to this repository itself. Finally, this PR includes some documentation about the current status of exceptions and links to various tracking issues too. [gist]: https://gist.github.com/yerzham/302efcec6a2e82c1e8de4aed576ea29d
Use latest `wasm-component-ld` in CI Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
Keeping it up-to-date
Keeping LLVM up-to-date. Closes WebAssembly#595
Build `wasip3` target ~Requires WebAssembly#592~ There's a lot of failing tests currently, so we'll probably want to rebase this PR as we pull in more recent versions of `wasi-libc` and merge once tests pass --------- Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net> Co-authored-by: Alex Crichton <alex@alexcrichton.com>
This includes a new library, `libclang.so`, which is Clang's public C API. This is used by tooling such as Rust's `bindgen` library to generate bindings and can be useful when bindings are generated using a wasi-sdk distribution.
Notably pull in some fixes for the wasip2 target related to networking.
This is intended to provide a known-good build of LLDB with wasm plugin support which can exist alongside the rest of the LLVM distribution provided by wasi-sdk.
* Update for C++ Exceptions being supported. * Update the threading-related bits slightly. * Indicate that dynamic linking works but may not be as fully tested. * Networking works, just not on WASIp1.
No longer use `wasm32-wasi` since Clang issues a deprecation warning about that. This fixes the `clang` binary, by default, emitting a warning for example in the final build.
Once more before wasi-sdk-31
Fix an issue with setjmp/longjmp when combined with LLVM 22.
Handle warnings cropping up in CI
First pass at trying to fix this...
Pulls in llvm/llvm-project#186054 and adds a regression test which previously failed. Closes WebAssembly#610
Attempting to fix WebAssembly#612
I'm shamelessly copying what [AlmaLinux does](https://git.almalinux.org/rpms/libedit/src/commit/3f0893c4cd8e0cbb2f556d2fad48326c9c037a6c/SPECS/libedit.spec#L44-L48), so let's try that... Closes WebAssembly#613
Currently LLDB binaries as-is don't work, and this should fix them.
Keeping it up-to-date
…bAssembly#621) Rework of the riscv64-linux CI build to use CMake cross-compilation on a standard `ubuntu-24.04` runner, rather than a native RISE runner. ## What changed **`ci/docker/Dockerfile.riscv64-linux`** (new): - Ubuntu 24.04 base — has `crossbuild-essential-riscv64` in its package repos - Sets `CC=riscv64-linux-gnu-gcc` / `CXX=riscv64-linux-gnu-g++` so CMake detects cross-compilation and causes LLVM to build a native `llvm-tblgen` first, then cross-compile the rest of the toolchain - Sets `CARGO_TARGET_RISCV64_UNKNOWN_LINUX_GNU_LINKER` for Rust cross-builds - `XDG_CACHE_HOME=/tmp/cache` avoids write permission issues in the container **`ci/docker-build.sh`**: - Select `ci/docker/Dockerfile.<artifact>` if it exists, fall back to the default `ci/docker/Dockerfile` - Make the wasmtime volume mount conditional on `WASI_SDK_CI_SKIP_SYSROOT != 1` **`.github/workflows/main.yml`**: - New `riscv64-linux` matrix entry: `os: ubuntu-24.04`, `rust_target: riscv64-unknown-linux-gnu` - `cross_cmake_args: -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=riscv64 -DWASI_SDK_LLDB=OFF` - `WASI_SDK_CI_SKIP_SYSROOT: 1` - Handle `cross_cmake_args` in the cmake flags step ## Why WASI_SDK_CI_SKIP_SYSROOT The cross-compiled clang runs on riscv64, not on the x86_64 build host, so the wasm sysroot step is skipped. ## Why WASI_SDK_LLDB=OFF Avoids cross-compiling libedit and libxml2 in this first iteration; can be re-enabled as a follow-up. Closes WebAssembly#607 --------- Signed-off-by: Bruno Verachten <gounthar@gmail.com>
) This commit collects together some LLVM PRs, some changes in the build configuration here, and some thoughts from WebAssembly#565 and related issues. Specifically the changes here are: * The patch for llvm/llvm-project#168449 is updated to its upstream (unlanded) form. * Patches for the (landed) llvm/llvm-project#185770 and llvm/llvm-project#185775 are added. * The `WASI_SDK_EXCEPTIONS` configuration is now either `ON`, `OFF`, or `DUAL`. The default depends on the version of Clang in use, where 23.0.0+ (which isn't released officially yet) will be `DUAL` and otherwise it's `OFF`. CI for our custom-built patched toolchain defaults to `DUAL`. * In `DUAL` mode libcxx is built twice into two different directories, once with exceptions and once without. This is supported by LLVM patches and means that Clang will select the right set of libraries based on compiler flags. The end result here is that the produced toolchain from this repository, by default, supports C++ exceptions. Additionally if exceptions-related flags are not passed then the final binary will not use C++ exceptions nor require the wasm exception-handling proposal. There's still follow-up work from WebAssembly#565, such as: * Subjectively it feels wordy to pass `-fwasm-exceptions` vs `-fexceptions`. * Personally I think `-mllvm -wasm-use-legacy-eh=false` should become the default upstream. * Subjectively I don't think that `-lunwind` should be necessary and it should be injected automatically with `-fwasm-exceptions` (or `-fexceptions`). * Shared libraries for exceptions remain disabled due to build errors I do not personally know how to resolve. I'll file follow-up issues for these once this has landed since they're more minor compared to the main body of "anything works". Closes WebAssembly#334 Closes WebAssembly#565 --------- Co-authored-by: Joel Dice <joel.dice@akamai.com>
I tried to built it locally but failed, hopefully CI can do it.
# Conflicts: # .gitmodules # docker/Dockerfile # src/llvm-project
lewing
force-pushed
the
enable-wasm-exceptions
branch
from
3596e0d to
1370489
Compare
lewing
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR was created with assistance from GitHub Copilot.
Summary
Merge upstream WebAssembly/wasi-sdk through wasi-sdk-33, bringing in C++ exception handling support and other improvements.
Key upstream changes included
-fwasm-exceptionsand exnref supportWhy
The CoreCLR WASM interpreter on WASI uses C++ exceptions (
throw/catch) for managed exception handling dispatch. Without an exception-enabled sysroot (libc++, libc++abi, libunwind), the runtime cannot be linked.Conflict resolution
.gitmodules: Added upstream'ssrc/configsubmodule, kept dotnet'ssrc/llvm-projectpointing todotnet/llvm-projectdocker/Dockerfile: Kept upstream's updated versionsrc/llvm-project: Kept dotnet's submodule referenceTesting
The resulting sysroot has been validated with the CoreCLR WASM interpreter on both nesm and wasmtime 43, with managed exception handling (try/catch/throw) working end-to-end on both legacy and exnref EH models.