This commit collects together some LLVM PRs, some changes in the build
configuration here, and some thoughts from #565 and related issues.
Specifically the changes here are:
* The patch for llvm/llvm-project#168449 is updated to its upstream
(unlanded) form.
* Patches for the (landed) llvm/llvm-project#185770 and
llvm/llvm-project#185775 are added.
* The `WASI_SDK_EXCEPTIONS` configuration is now either `ON`, `OFF`, or
`DUAL`. The default depends on the version of Clang in use, where
23.0.0+ (which isn't released officially yet) will be `DUAL` and
otherwise it's `OFF`. CI for our custom-built patched toolchain defaults
to `DUAL`.
* In `DUAL` mode libcxx is built twice into two different directories,
once with exceptions and once without. This is supported by LLVM patches
and means that Clang will select the right set of libraries based on
compiler flags.
The end result here is that the produced toolchain from this repository,
by default, supports C++ exceptions. Additionally if exceptions-related
flags are not passed then the final binary will not use C++ exceptions
nor require the wasm exception-handling proposal.
There's still follow-up work from #565, such as:
* Subjectively it feels wordy to pass `-fwasm-exceptions` vs
`-fexceptions`.
* Personally I think `-mllvm -wasm-use-legacy-eh=false` should become
the default upstream.
* Subjectively I don't think that `-lunwind` should be necessary and it
should be injected automatically with `-fwasm-exceptions` (or
`-fexceptions`).
* Shared libraries for exceptions remain disabled due to build errors I
do not personally know how to resolve.
I'll file follow-up issues for these once this has landed since they're
more minor compared to the main body of "anything works".
Closes#334Closes#565
---------
Co-authored-by: Joel Dice <joel.dice@akamai.com>
Build `wasip3` target
~Requires #592~
There's a lot of failing tests currently, so we'll probably want to
rebase this PR as we pull in more recent versions of `wasi-libc` and
merge once tests pass
---------
Signed-off-by: Roman Volosatovs <rvolosatovs@riseup.net>
Co-authored-by: Alex Crichton <alex@alexcrichton.com>
> **Note**: this PR is me weaving together the work of many others. For
example @yerzham's and @cpetig's work on #565 pretty much shaped this
PR. Thank you!
This commit adds initial configuration support for building wasi-sdk
with support for C++ exceptions in WebAssembly. A small test is included
which is exercised in CI at this time, but otherwise this does not
update CI to actually ship sdk builds with C++ exceptions enabled.
Instead the intention here is to make it easier to test builds with
support for C++ exceptions and centralize planning/development around
this.
The goal here is to get things compiling to the point that applications
can be compiled. I haven't thoroughly tested C++ exceptions and as
evident in this PR it still requires changes in LLVM. Some small logic
is added here to apply a `*.patch` file to LLVM to avoid needing a new
submodule fork or waiting for upstream changes. The intention is that
this `*.patch` is short-lived once the changes are officially merged
into LLVM itself.
The `*.patch` included here contains llvm/llvm-project#168449 as well as
another minor edit I found was necessary to get things compiling
locally. Given the discussion on that PR it looks like once LLVM is
updated with that merged the extra part of the `*.patch` won't be
necessary.
This PR notably does not enable shared libraries when exceptions are
enabled. I don't know enough about how things are supposed to work to be
able to fully diagnose the compilation errors I'm seeing if shared
libraries are enabled. This is something I'd hope would be fixed before
actually shipping exceptions support.
This PR then additionally folds in [this gist][gist] for various bits of
build logic to this repository itself.
Finally, this PR includes some documentation about the current status of
exceptions and links to various tracking issues too.
[gist]: https://gist.github.com/yerzham/302efcec6a2e82c1e8de4aed576ea29d
This updates to include WebAssembly/wasi-libc#709 and then additionally
configures wasi-libc with `-DWASI_SDK_VERSION=...` to ensure that
wasi-sdk builds will now have `__wasi_sdk_major__` defined along with
`__wasi_sdk_version__`.
ClosesWebAssembly/wasi-libc#688
This commit updates the build of the `wasm32-wasip2` target to always
pass `-fPIC` to compilation of wasi-libc. This notably enables using
`libc.a` in dynamic linking situations instead of being forced to use
`libc.so`. While many applications likely want to use `libc.so` I've
found it more flexible if objects are compatible with as many builds as
possible. This will enable, for example, building shared libraries in
Rust by default since Rust only ships `libc.a`, not `libc.so`, in the
pre-built sysroot. This behavior additionally matches the Rust
`wasm32-wasip2` target where `-fPIC` is enabled by default there.
Add a cmake option WASI_SDK_CPU_CFLAGS to specify CFLAGS to control wasm
features to enable/disable.
Make the default `-mcpu=lime1`.
Comparing to the default "generic" target of the recent llvm, this
effectively
* disables reference-types and bulk-memory (enable their partial
counterparts instead)
* enables extended-const
Note: as of writing this, a few popular runtimes, including wasm3 and
wasm-micro-runtime,
don't support extented-const yet.
Note: regardless of this, wasi-libc enables necessary features for
certain files. (bulk-memory and exception-handling)
cf. https://github.com/WebAssembly/wasi-sdk/issues/525
References:
https://github.com/WebAssembly/tool-conventions/blob/main/Lime.mdhttps://github.com/bytecodealliance/wasm-micro-runtime/issues/4272
This commit enables thread support in libc++ for all thread models,
enabling C++ applications that use threading APIs like `<atomic>` but do
not spawn threads (e.g. Clang) to be built with minimal changes.
Fixes#546.
Depends on WebAssembly/wasi-libc#602.
This change allows us to have per-target libclang_rt.* libraries instead
of per-OS. This is particularly required for w/threads and wo/threads
targets, whose rt libraries could be incompatible even with the same OS
string.
This change starts building `compiler-rt` for w/threads and wo/threads
separately, so that we can enable thread-related features for p1-threads
target for building upcoming ASan clang_rt libraries. Also
`COMPILER_RT_OS_DIR` is not set anymore as it's unused when
`LLVM_ENABLE_PER_TARGET_RUNTIME_DIR` is enabled.
* Pin CI runner images to explicit versions
CI is run infrequently enough on this repository that there's a
nontrivial chance that GitHub updates the definition of `ubuntu-latest`
for example between two CI runs. This can be confusing for contributors
as PRs seemingly break CI when in reality they have nothing to do with
the breakage and it's due to the image changing.
This PR pins all images to the definition of `*-latest` at this time.
The one exception is that one builder was pinned to 22.04 and I'm going
to update it to 24.04 here and see if I can't fix CI issues that come up.
This'll require explicit PRs to update these images in the future, but
hopefully that's only once every few ~years so not too much of a burden.
* Try not custom-installing clang 18
* Try to fix CI issue
* Fix CI again (attempt)
* Further try to fix CI
* Another fix for CI
Surely if I keep making commits that say "another fix" eventually I'll
run out of fixes. Surely, right? I'd put this in limerick form to be
more amusing if someone comes to read this but I'm not clever enough to
do that, so instead I'll just abruptly
* Add back `VERSION` file to release tarballs
Make sure it makes its way into the sysroot tarball and then copy it
from the sysroot location to the top of the sdk tarball as well.
Closes#471
* Fix how `VERSION` is installed
This commit adds a test and matrix entry to CI which asserts that the
sysroot can be built to run tests using a stock Clang compiler found on
the system. This fixes a few minor issues as well in developing this.
This additionally refactors CI a bit to move shared steps amongst jobs
into separate composite actions in this repository to avoid duplication
across jobs.
* Require that the compiler is Clang, for example gcc and msvc cannot
compile to WebAssembly.
* Require that the Clang version is above the minimum threshold. Unsure
what the minimum threshold is at this time so I've set it to 18.0.0
* Add LTO build of wasi-libc
* Add LTO build of libc++
* Update wasi-libc (to include the LTO support)
* Add a comment about /llvm-lto/${llvm_version} convention
* Use separate targets for wasi-libc with and without lto
A downside: this makes four more copies of wasi-libc source tree.
* Make LTO build optional
* Retire clang_version
Note: this makes the "install" target install the compiler rt
into the compiler's runtime directory. IMO, it's what "install" is
supposed to do. If you want to avoid modifing the runtime directory
for some reasons, you can still do "dist" without "install".
* wasi-sdk-sysroot.cmake: Avoid modifying things out of CMAKE_INSTALL_PREFIX
* wasi-sdk-sysroot.cmake: control the resource-dir modification by a cmake option
* Add a CI check for testing toolchains as-distributed
One aspect of testing lost in the CMake-based migration is the ability
to test the toolchains as distributed in release artifacts. Tests use
`--sysroot` and `-resource-dir` (soon) to customize how the host
compiler runs but this means that it would be possible to regress the
default sysroot theoretically. To rectify this situation this commit
adds a new CI test which uses the release artifacts of previous steps to
build a `wasi-sdk-*.tar.gz` tarball which is then extracted and tested
as-is. A new flag was added to the cmake configuration to avoid
depending on fresh sysroot libraries for tests and instead test the host
toolchain.
* Fix version.py script running
* Fix artifact download
* Add ninja
* Update submodules in new test job
* Only add extra options for libcxx build
Otherwise the test directory seems like it inherits these options which
isn't desired when testing the host toolchain.
This commit updates the building of the wasi-sdk sysroot to leverage the
`-resource-dir` argument from Clang to avoid modifying the host
compiler's sysroot with compiler-rt things. This should help improve the
experience of building a standalone sysroot with whatever host Clang is
on the system.
Closes#444
* Update build of toolchain/sysroot to not touch installation root
This changes everything to ensure that only the `install` step actually
tries to install things. Everything is staged into temporary `./install`
folders inside of the build directory and then running the build
system's `install` target will actually copy out everything using CMake
builtins.
Closes#442
* Better integrate generating a version file
* Rewrite the build system with CMake
This commit is an attempt to provide a concrete path forward on
WebAssembly/wasi-sdk#425. I personally think it's pretty important to
get the ability to have more architectures here but at the same time I
also think it's important to to take this as an opportunity to refactor
and improve the build system of this repository. To that end this
represents my attempt to improve the status quo.
This removes the old `Makefile` and replaces it with a CMake-based
system to build all these projects. Overall this is intended to be a "no
functional change" intended sort of refactoring. Changing build systems
inevitably causes issues, however, so this change additionally has a
very high likelihood of needing follow-up fixes. At a high enough level
this commit introduces two major changes to how this repository is
built:
1. The `make`-based system (the root `Makefile`) is replaced with CMake.
This additionally updates tests to use CMake.
2. A single "build" is split into either building a toolchain or
building a sysroot. This enables builds to only build one or the
other as necessary.
The first change, using CMake, is due to the fact that using `make` on
Windows basically is not pleasant coupled with the fact that more
advanced logic, such as changing flags, compilers, etc, is much easier
with a CMake-based system. The second change is intended to cover the
use case of #425 in addition to refactoring the current build.
Throughout this change I have intentionally not tried to keep a 1:1
correspondance with behaviors in the old `Makefile` because much of this
PR is intended to address shortcomings in the old build system. A list
of changes, improvements, etc, made here are:
* CMake provides a much nicer portability story to Windows than `make`.
This is moving towards the direction of not needing `bash`, for
example, to build an SDK. Currently `wasi-libc` still requires this,
but that's now the only "hard" dependency.
* The set of targets built can now be configured for smaller builds
and/or debugging just a single target. All WASI targets are still
built by default but it's much easier to add/remove them.
* Different targets are now able to be built in parallel as opposed to
the unconditional serial-nature of the `Makefile`.
* Use of `ninja` is no longer required and separate build systems can be
used if desired.
* The sysroot and the toolchain can now be built with different CMake
build profiles. For example the `Makefile` hardcoded `MinSizeRel` and
`RelWithDebInfo` and this can now be much more easily customized by
the SDK builder.
* Tarballs are now more consistently produced and named. For a tarball
of the name `foo.tar.gz` it's guaranteed that there's a single folder
`foo` created when unpacking the tarball.
* The macOS binaries are no longer hybrid x64/arm64 binaries which
greatly inflates the size of the SDK. There's now a separate build for
each architecture.
* CI now produces arm64-linux binaries. The sysroot is not built on the
arm64-linux builder and the sysroot from the x86_64-linux builder is
used instead.
* Tests are almost ready to execute on Windows, there's just a few minor
issues related to exit statuses and probably line endings which need
to be worked out. Will require someone with a Windows checkout, however.
* Tests are now integrated into CMake. This means that the wasm binaries
are able to be built in parallel and the tests are additionally
executed in parallel with `ctest`. It is possible to build/run a
single test. Tests no longer place all of their output in the source
tree.
* Out-of-tree builds are now possible and the build/installation
directories can both be customized.
* CI configuration of Windows/macOS/Linux is much more uniform by having
everything in one build matrix instead of separate matrices.
* Linux builds are exclusively done in docker containers in CI now. CI
no longer produces two Linux builds only for one to be discarded when
artifacts are published.
* Windows 32-bit builds are no longer produced in CI since it's expected
that everyone actually wants the 64-bit ones instead.
* Use of `ccache` is now automatically enabled if it's detected on the
system.
* Many preexisting shell scripts are now translated to CMake one way or
another.
* There's no longer a separate build script for how to build wasi-sdk in
docker and outside of docker which needs to be kept in sync,
everything funnels through the same script.
* The `docker/Dockerfile` build of wasi-sdk now uses the actual
toolchain built from CI and additionally doesn't duplicate various
CMake-based configuration files.
Overall one thing I want to additionally point out is that I'm not CMake
expert. I suspect there's lots of little stylistic and such improvements
that can be made.
* Fix skipping tests on windows
* Fetch a full depth in the finalize workflow too
* Fix multi-arch docker build
* Fix assembling of sysroot
* Fix script syntax
* Clean up the merge script slightly
* Remove Pat's email
* Move configuration of CMAKE_EXECUTABLE_SUFFIX
* Remove redundant sysroot option
* Fix comment in testcase.sh
* Update new p2 cmake files
* Remove now-duplicate wasi-sdk-p2.cmake
Alex Crichton
Roman Volosatovs
YAMAMOTO Takashi
Catherine
Dan Gohman
Yuta Saito