* Add back `VERSION` file to release tarballs
Make sure it makes its way into the sysroot tarball and then copy it
from the sysroot location to the top of the sdk tarball as well.
Closes#471
* Fix how `VERSION` is installed
This commit adds a test and matrix entry to CI which asserts that the
sysroot can be built to run tests using a stock Clang compiler found on
the system. This fixes a few minor issues as well in developing this.
This additionally refactors CI a bit to move shared steps amongst jobs
into separate composite actions in this repository to avoid duplication
across jobs.
* Require that the compiler is Clang, for example gcc and msvc cannot
compile to WebAssembly.
* Require that the Clang version is above the minimum threshold. Unsure
what the minimum threshold is at this time so I've set it to 18.0.0
This commit fixes a bug from the cmake migration where tests were not
actually testing the correct target. Object files were compiled with the
right options but the link step was missing both `--target` and
`-pthread` which caused everything to accidentally be tested as
`wasm32-wasi`. When fixing this one test was needed to have its stderr
updated because the component output of `wasm32-wasip2` is slightly
different. A timeout was additionally added because without `-pthread`
at the link step some tests infinitely ran which made debugging
difficult.
This is an attempt to fix CI errors where the `config` submodule is now
located on a commit further than 32 commits behind the main branch
(presumably).
To include https://github.com/WebAssembly/wasi-libc/pull/517,
which is necessary to avoid modifying host compiler's resource
directory.
At least on my environment (macOS if it matters), without the above
mentioned fix, gmake tries to create the directory for wasi compiler-rt
during wasi-libc build. I'm not sure why it didn't happen for others.
cf. https://github.com/WebAssembly/wasi-sdk/pull/445
* Add LTO build of wasi-libc
* Add LTO build of libc++
* Update wasi-libc (to include the LTO support)
* Add a comment about /llvm-lto/${llvm_version} convention
* Use separate targets for wasi-libc with and without lto
A downside: this makes four more copies of wasi-libc source tree.
* Make LTO build optional
* Retire clang_version
Note: this makes the "install" target install the compiler rt
into the compiler's runtime directory. IMO, it's what "install" is
supposed to do. If you want to avoid modifing the runtime directory
for some reasons, you can still do "dist" without "install".
* wasi-sdk-sysroot.cmake: Avoid modifying things out of CMAKE_INSTALL_PREFIX
* wasi-sdk-sysroot.cmake: control the resource-dir modification by a cmake option
* Use MSVC for the Windows toolchain instead of MinGW
Explicitly use MSVC to avoid the runtime dependencies that the default
toolchain CMake is using is bringing in.
Closes#454
* Update .github/workflows/main.yml
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
* Specify MinSizeRel in `build.sh`
Looks like MSVC defaults to "Debug" instead of an empty string to so the
default logic which works for other compilers isn't kicking in here.
* Don't persist ccache stats across runs
* Move zeroing
---------
Co-authored-by: Andrew Brown <andrew.brown@intel.com>
This command is generally not included in toolchain files. It is used in
project configuration files that will consume the toolchain file. Also,
such a low value triggers warnings, support for <3.5.0 is being removed
from CMake.
I noticed in a [recent build] that the cache size for Windows is quite
large at 500M. That might be related to switching to MSVC, I'm not sure,
but something else I've realized is that as-configured wasi-sdk will
continuously grow the cache over time and it won't ever get trimmed
until we hit github actions limits. This is because the cache is
restored from an older version, then appended to with the current
build, then saved again. That theoretically means that each builder
could make up to a 5G cache which is a bit too large.
This commit adds an extra step that removes all objects older than 1d to
ensure that older builds eventually get cleaned out of the cache. GitHub
Actions should then still delete older caches pretty regularly but each
individual cache should be bounded still.
[recent build]: https://github.com/WebAssembly/wasi-sdk/actions/runs/10045872592/job/27764084758?pr=456
* Refactor handling env vars in CI build
This commit updates the handling of various environment variables to
centralize them in one location in `main.yml` as part of CI
configuration. This removes a few custom-named variables in favor of
using explicitly-listed env vars. Additionally this moves some env var
management from the CI container files to the github actions config as
well to keep everything in one place ideally.
* Add some more comments about where configuration comes in
* Add a CI check for testing toolchains as-distributed
One aspect of testing lost in the CMake-based migration is the ability
to test the toolchains as distributed in release artifacts. Tests use
`--sysroot` and `-resource-dir` (soon) to customize how the host
compiler runs but this means that it would be possible to regress the
default sysroot theoretically. To rectify this situation this commit
adds a new CI test which uses the release artifacts of previous steps to
build a `wasi-sdk-*.tar.gz` tarball which is then extracted and tested
as-is. A new flag was added to the cmake configuration to avoid
depending on fresh sysroot libraries for tests and instead test the host
toolchain.
* Fix version.py script running
* Fix artifact download
* Add ninja
* Update submodules in new test job
* Only add extra options for libcxx build
Otherwise the test directory seems like it inherits these options which
isn't desired when testing the host toolchain.
This commit updates the building of the wasi-sdk sysroot to leverage the
`-resource-dir` argument from Clang to avoid modifying the host
compiler's sysroot with compiler-rt things. This should help improve the
experience of building a standalone sysroot with whatever host Clang is
on the system.
Closes#444
This includes support for a new `--component-type` option, which allows passing
one or more component types as WIT files rather than (or in addition to) object
files, which can be preferable when such files must be stored in source control.
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
* Update build of toolchain/sysroot to not touch installation root
This changes everything to ensure that only the `install` step actually
tries to install things. Everything is staged into temporary `./install`
folders inside of the build directory and then running the build
system's `install` target will actually copy out everything using CMake
builtins.
Closes#442
* Better integrate generating a version file
This commit updates the release process of `wasi-sdk` to remove most
of the manual interaction and steps done. Instead now draft releases are
automatically created for tags made. This means that there's only two
steps necessary: (1) pushing a tag and (2) hitting publish on the
generated release. This commit also removes a number of the CI scripts
previously used to manage releases.
This is a revival of #397 but only applies the logic to non-Windows
platforms. Windows builds still seem to not work so Windows is not
improved as a result of this change.
* Rewrite the build system with CMake
This commit is an attempt to provide a concrete path forward on
WebAssembly/wasi-sdk#425. I personally think it's pretty important to
get the ability to have more architectures here but at the same time I
also think it's important to to take this as an opportunity to refactor
and improve the build system of this repository. To that end this
represents my attempt to improve the status quo.
This removes the old `Makefile` and replaces it with a CMake-based
system to build all these projects. Overall this is intended to be a "no
functional change" intended sort of refactoring. Changing build systems
inevitably causes issues, however, so this change additionally has a
very high likelihood of needing follow-up fixes. At a high enough level
this commit introduces two major changes to how this repository is
built:
1. The `make`-based system (the root `Makefile`) is replaced with CMake.
This additionally updates tests to use CMake.
2. A single "build" is split into either building a toolchain or
building a sysroot. This enables builds to only build one or the
other as necessary.
The first change, using CMake, is due to the fact that using `make` on
Windows basically is not pleasant coupled with the fact that more
advanced logic, such as changing flags, compilers, etc, is much easier
with a CMake-based system. The second change is intended to cover the
use case of #425 in addition to refactoring the current build.
Throughout this change I have intentionally not tried to keep a 1:1
correspondance with behaviors in the old `Makefile` because much of this
PR is intended to address shortcomings in the old build system. A list
of changes, improvements, etc, made here are:
* CMake provides a much nicer portability story to Windows than `make`.
This is moving towards the direction of not needing `bash`, for
example, to build an SDK. Currently `wasi-libc` still requires this,
but that's now the only "hard" dependency.
* The set of targets built can now be configured for smaller builds
and/or debugging just a single target. All WASI targets are still
built by default but it's much easier to add/remove them.
* Different targets are now able to be built in parallel as opposed to
the unconditional serial-nature of the `Makefile`.
* Use of `ninja` is no longer required and separate build systems can be
used if desired.
* The sysroot and the toolchain can now be built with different CMake
build profiles. For example the `Makefile` hardcoded `MinSizeRel` and
`RelWithDebInfo` and this can now be much more easily customized by
the SDK builder.
* Tarballs are now more consistently produced and named. For a tarball
of the name `foo.tar.gz` it's guaranteed that there's a single folder
`foo` created when unpacking the tarball.
* The macOS binaries are no longer hybrid x64/arm64 binaries which
greatly inflates the size of the SDK. There's now a separate build for
each architecture.
* CI now produces arm64-linux binaries. The sysroot is not built on the
arm64-linux builder and the sysroot from the x86_64-linux builder is
used instead.
* Tests are almost ready to execute on Windows, there's just a few minor
issues related to exit statuses and probably line endings which need
to be worked out. Will require someone with a Windows checkout, however.
* Tests are now integrated into CMake. This means that the wasm binaries
are able to be built in parallel and the tests are additionally
executed in parallel with `ctest`. It is possible to build/run a
single test. Tests no longer place all of their output in the source
tree.
* Out-of-tree builds are now possible and the build/installation
directories can both be customized.
* CI configuration of Windows/macOS/Linux is much more uniform by having
everything in one build matrix instead of separate matrices.
* Linux builds are exclusively done in docker containers in CI now. CI
no longer produces two Linux builds only for one to be discarded when
artifacts are published.
* Windows 32-bit builds are no longer produced in CI since it's expected
that everyone actually wants the 64-bit ones instead.
* Use of `ccache` is now automatically enabled if it's detected on the
system.
* Many preexisting shell scripts are now translated to CMake one way or
another.
* There's no longer a separate build script for how to build wasi-sdk in
docker and outside of docker which needs to be kept in sync,
everything funnels through the same script.
* The `docker/Dockerfile` build of wasi-sdk now uses the actual
toolchain built from CI and additionally doesn't duplicate various
CMake-based configuration files.
Overall one thing I want to additionally point out is that I'm not CMake
expert. I suspect there's lots of little stylistic and such improvements
that can be made.
* Fix skipping tests on windows
* Fetch a full depth in the finalize workflow too
* Fix multi-arch docker build
* Fix assembling of sysroot
* Fix script syntax
* Clean up the merge script slightly
* Remove Pat's email
* Move configuration of CMAKE_EXECUTABLE_SUFFIX
* Remove redundant sysroot option
* Fix comment in testcase.sh
* Update new p2 cmake files
* Remove now-duplicate wasi-sdk-p2.cmake
I forgot in #422 that by specifying `EXTRA_CFLAGS` to the wasi-libc
build that it would override the defaults of wasi-libc which is to pass
these two flags in. This passes them back in to ensure the default build
still has the same flags.
* Add DWARF debugging information to all artifacts by default
This commit adds DWARF debugging information with the `-g` compiler flag
to all WASI artifacts for wasi-sdk. The LLVM build itself does not have
debugging information, only the sysroot artifacts. This is intended to
assist with debugging. The main downside to this is binary size of
generated artifacts will, by default, be larger. Stripping debug
information from an artifact though involves removing custom sections
which is generally pretty easy to do through wasm tooling.
* Pass extra cflags to wasi-libc
* Fix tests from previous commit
* Update some expected error messages and remove some files with
duplicate error messages that are no longer needed.
* Remove undefined behavior in `stat.c` where padding bytes were being
compared.
This helps ensure the identically-named 32-bit tarfile doesn't overwrite the
64-bit one when we extract everything into a single directory.
I'll follow this up by making an updated wasi-sdk-22.0.m-mingw.tar.gz and adding
it to the wasi-sdk-22 release.
Fixes#326
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
Recent changes have added various flags for customizing the
component-creation process in addition to supporting more flags being
forwarded to `wasm-ld`.
This commit fixes a minor mistake from #392 where the previous perl
script had a special case that if the "minor" version was zero then the
git hash information wouldn't be printed.
My release attempt in #408 did not go well because the release artifacts
were not named correctly. This fixes an issue described by
actions/checkout#290 where checkouts of annotated tags overwrite the
annotation which breaks `git describe`. That means that version
detection is broken in CI during releases which causes artifacts to have
the wrong information.
This applies the workaround described in that issue to `git fetch --tags
--force` after the checkout step to undo the overwrite done in the
checkout step.
* Update LLVM to 18.1.2 and add wasm-component-ld
This commit has two intertwined changes within it. First the LLVM
submodule is updated to the 18.1.2 release branch. This alone would
cause CI and tests to fail due to differing behavior for the
`wasm32-wasip2` target. To fix these test failures the
`wasm-component-ld` tool is added to the build. This tool is a
Rust-written tool and installed via `cargo install` as part of the build
at a pinned version written in the `Makefile`. This linker, used for
components, is then used for the `wasm32-wasip2` target.
Tests and CI are then updated to skip the need to have `wasm-tools` or
the adapter for WASI when making components since that's now the job of
`wasm-component-ld`. This then necessitated some changes to tests too.
* Add back accidentally deleted CI
* Inherit tools on PATH on Windows
* Add Rust to docker build
* Put rust in a different location in docker
* Try to fix permissions
* Review comments
* Revert changes to test outputs
This updates the `wasi-libc` submodule to point to `main` and also:
- update tests/testcase.sh to use new wasm32-wasip2 target name
- update various tests/**/*.expected files to match Wasmtime output
Signed-off-by: Joel Dice <joel.dice@fermyon.com>
In #399, @glandium points out that newer versions of LLVM will place
their version information at a different path. This change adapts #399
to the new Python version script.
When running Git commands inside this Docker container (i.e., commands
that the `version.py` script needs for determining version information),
the Docker build would run into issues like:
```
fatal: detected dubious ownership in repository at '/workspace'
To add an exception for this directory, call:
git config --global --add safe.directory /workspace
```
This is due to an extra Git check that detects that the Docker user is
not the same one who owns the `.git` directory of this project. After
looking into this, the best solution the internet has to offer is to
thread the current user's UID and GID through the Docker image (i.e.,
the new `builder` user) and then `docker run --user ...`. This both
avoids the Git check but also seems to be considered a best practice in
some circles (?).
As noted in #372, some users may need to identify which version of
wasi-sdk they are using after it has been downloaded. There are many
ways to solve this, but the one I chose here is to dump the wasi-sdk
version into a `VERSION` file and follow that up with several dependency
versions that may be helpful to note when troubleshooting. Ideally
someone could paste the contents of that file when filing a bug. If we
adopt this approach, this fixes#372.
In the past, I've had trouble keeping the current set of version scripts
to correctly output the version, especially in light of how we append
Git suffixes for a non-tagged commit and dirty state. This change
replaces those scripts with a single Python one which, though much more
wordy than the previous one, may be easier for contributors to read and
modify. The original scripts relied on Perl; this one relies on Python,
which seems like a fair exchange. Having a single script also makes it
easier to solve #372.
This test looks to be asserting that `getrandom` never returns 256
consecutive zeros, but the way it's asserting that is summing up the
bytes and asserting the sum is nonzero. Due to this being a signed
addition, however, it's possible for the bytes to be nonzero and still
trigger the assert. Locally running this test in a loop took 30 or so
seconds before it triggered a failure.
I've updated the test to instead hunt for any entry which is not equal
to zero and then assert that something is not zero.
* Explicitly set LIBCXXABI_USE_LLVM_UNWINDER off
Clang trunk has recently turned it on, and also now fails the build when
libunwind is not built at the same time.
* Bump LLVM to 18.1.0
---------
Co-authored-by: Mike Hommey <mh@glandium.org>
Currently the `bin` directory installed by wasi-sdk is not currently
suitable for putting in `$PATH` in all cases because it can shadow a
system-installed `clang` executable which is intended for native
binaries. For Linux distributions with gcc-based cross-compilers I've
often seen the pattern where they're installed as `$target-gcc` and so
I've taken a leaf out of their books to do that here as well.
This commit adds, currently alongside the preexisting `clang`
executable, target-prefixed executables such as `wasm32-wasi-clang` and
`wasm32-wasi-clang++`. These executables are symlinks to the `clang`
executable itself in the same manner that `clang++` is a symlink to
`clang` itself.
I'll also note that this doesn't fix the problem of "add the wasi-sdk
bin dir to PATH" because `clang` and other prominent executables are
still there. My hope though is that this opens up a future refactoring
for doing so.
Alex Crichton
SingleAccretion
Andrew Brown
YAMAMOTO Takashi
Matt McCormick
Joel Dice
Catherine
Dan Gohman
Anuraag (Rag) Agrawal
Philip Rying
Mike Hommey