wasm-micro-runtime/doc/build_wasm_app.md

Build WASM applications

Prepare WASM building environments

For C and C++, WASI-SDK version 19.0+ is the major tool supported by WAMR to build WASM applications. Also, we can use Emscripten SDK (EMSDK), but it is not recommended. And there are some other compilers such as the standard clang compiler, which might also work here.

To install WASI SDK, please download the wasi-sdk release and extract the archive to default path /opt/wasi-sdk.

The official wasi-sdk release doesn't fully support latest 128-bit SIMD spec yet. WAMR provides a script in build-wasi-sdk to generate another wasi-sdk with llvm-15 from source code and installs it at ../test-tools/wasi-sdk. If you plan to build WASM applications with latest 128-bit SIMD, please use it instead of the official release.

And sample workloads are using the self-compiled wasi-sdk.

For AssemblyScript, please refer to AssemblyScript quick start and AssemblyScript compiler for how to install asc compiler and build WASM applications.

For Rust, please refer to Install Rust and Cargo to install cargo, rustc and rustup. By default they are under ~/.cargo/bin.

And then run such a command to install wasm32-wasi target.

$ rustup target add wasm32-wasi

To build WASM applications, run

$ cargo build --target wasm32-wasi

The output files are under target/wasm32-wasi.

To build a release version

$ cargo build --release --target wasm32-wasi

Build WASM applications with wasi-sdk

You can write a simple test.c as the first sample.

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
    char *buf;

    printf("Hello world!\n");

    buf = malloc(1024);
    if (!buf) {
        printf("malloc buf failed\n");
        return -1;
    }

    printf("buf ptr: %p\n", buf);

    sprintf(buf, "%s", "1234\n");
    printf("buf: %s", buf);

    free(buf);
    return 0;
}

To build the source file to WASM bytecode, we can input the following command:

/opt/wasi-sdk/bin/clang -O3 -o test.wasm test.c

1. wasi-sdk options

There are some useful options that are used to compile C/C++ to Wasm (for a full introduction, please refer to clang command line argument reference and wasm-ld command line argument manual):

• -nostdlib Do not use the standard system startup files or libraries when linking. In this mode, the libc-builtin library of WAMR must be built to run the wasm app, otherwise, the libc-wasi library must be built. You can specify -DWAMR_BUILD_LIBC_BUILTIN=1 or -DWAMR_BUILD_LIBC_WASI=1 for CMake to build WAMR with libc-builtin support or libc-wasi support.

• -Wl,--no-entry Do not output any entry point

• -Wl,--export=<value> Force a symbol to be exported, e.g. -Wl,--export=foo to export foo function

• -Wl,--export-all Export all symbols (normally combined with --no-gc-sections)

• -Wl,--initial-memory=<value> Initial size of the linear memory, which must be a multiple of 65536

• -Wl,--max-memory=<value> Maximum size of the linear memory, which must be a multiple of 65536

• -z stack-size=<vlaue> The auxiliary stack size, which is an area of linear memory, must be smaller than the initial memory size.

• -Wl,--strip-all Strip all symbols

• -Wl,--shared-memory Use shared linear memory

• -Wl,--allow-undefined Allow undefined symbols in linked binary

• -Wl,--allow-undefined-file=<value> Allow symbols listed in <file> to be undefined in linked binary

• -pthread Support POSIX threads in generated code

For example, we can build the wasm app with the command:

/opt/wasi-sdk/bin/clang -O3 -nostdlib \
    -z stack-size=8192 -Wl,--initial-memory=65536 \
    -o test.wasm test.c \
    -Wl,--export=main -Wl,--export=__main_argc_argv \
    -Wl,--export=__heap_base -Wl,--export=__data_end \
    -Wl,--no-entry -Wl,--strip-all -Wl,--allow-undefined

to generate a wasm binary with nostdlib mode, the auxiliary stack size is 8192 bytes, initial memory size is 64 KB, main function, heap base global and data end global are exported, no entry function is generated (no _start function is exported), and all symbols are stripped. Note that it is nostdlib mode, so libc-builtin should be enabled by runtime embedder or iwasm (with cmake -DWAMR_BUILD_LIBC_BUILT=1, enabled by iwasm in Linux by default).

If we want to build the wasm app with wasi mode, we may build the wasm app with the command:

/opt/wasi-sdk/bin/clang -O3 \
    -z stack-size=8192 -Wl,--initial-memory=65536 \
    -o test.wasm test.c \
    -Wl,--export=__heap_base -Wl,--export=__data_end \
    -Wl,--strip-all

to generate a wasm binary with wasi mode, the auxiliary stack size is 8192 bytes, initial memory size is 64 KB, heap base global and data end global are exported, wasi entry function exported (_start function), and all symbols are stripped. Note that it is wasi mode, so libc-wasi should be enabled by runtime embedder or iwasm (with cmake -DWAMR_BUILD_LIBC_WASI=1, enabled by iwasm in Linux by default), and normally no need to export main function, by default _start function is executed by iwasm.

Note: for the Rust project, we can set these flags by setting the rustflags in the Cargo configuration file, e.g. <project_dir>/.cargo/config.toml or $CARGO_HOME/config.toml, for example:

[build]
rustflags = [
  "-C", "link-arg=--initial-memory=65536",
  "-C", "link-arg=-zstack-size=8192",
  "-C", "link-arg=--export=__heap_base",
  "-C", "link-arg=--export=__data_end",
  "-C", "link-arg=--strip-all",
]

2. How to reduce the footprint?

Firstly if libc-builtin (-nostdlib) mode meets the requirements, e.g. there are no file io operations in wasm app, we should build the wasm app with -nostdlib option as possible as we can, since the compiler doesn't build the libc source code into wasm bytecodes, which greatly reduces the binary size.

(1) Methods to reduce the libc-builtin (-nostdlib) mode footprint

• export __heap_base global and __data_end global

  -Wl,--export=__heap_base -Wl,--export=__data_end
  

  If the two globals are exported, and there are no memory.grow and memory.size opcodes (normally nostdlib mode doesn't introduce these opcodes since the libc malloc function isn't built into wasm bytecode), WAMR runtime will truncate the linear memory at the place of __heap_base and append app heap to the end, so we don't need to allocate the memory specified by -Wl,--initial-memory=n which must be at least 64 KB. This is helpful for some embedded devices whose memory resource might be limited.

For the Rust project, please set the flags in the Cargo configuration file, for example:

[build]
rustflags = [
  "-C", "link-arg=--export=__heap_base",
  "-C", "link-arg=--export=__data_end",
  "-C", "link-arg=--initial-memory=65536",
]

• reduce auxiliary stack size

  The auxiliary stack is an area of linear memory, normally the size is 64 KB by default which might be a little large for embedded devices and partly used, we can use -z stack-size=n to set its size.

For the Rust project, please set the flag in the Cargo configuration file, for example:

[build]
rustflags = [
  "-C", "link-arg=-zstack-size=8192"
]

• use -O3 and -Wl,--strip-all

For the Rust project, please set the flag in the Cargo configuration file, for example:

[build]
rustflags = [
 "-C", "link-arg=--strip-all"
]

• reduce app heap size when running iwasm

  We can pass --heap-size=n option to set the maximum app heap size for iwasm, by default it is 16 KB. For the runtime embedder, we can set the uint32_t heap_size argument when calling API wasm_runtime_instantiate.

• reduce wasm operand stack size when running iwasm

  WebAssembly is a binary instruction format for a stack-based virtual machine, which requires a stack to execute the bytecodes. We can pass --stack-size=n option to set the maximum stack size for iwasm, by default it is 16 KB. For the runtime embedder, we can set the uint32_t stack_size argument when calling API wasm_runtime_instantiate and wasm_runtime_create_exec_env.

• decrease block_addr_cache size for classic interpreter

  The block_addr_cache is a hash cache to store the else/end addresses for WebAssembly blocks (BLOCK/IF/LOOP) to speed up address lookup. This is only available in the classic interpreter. We can set it by defineing macro -DBLOCK_ADDR_CACHE_SIZE=n, e.g. add add_defintion (-DBLOCK_ADDR_CACHE_SIZE=n) in CMakeLists.txt, by default it is 64, and the total block_addr_cache size is 3072 bytes in 64-bit platform and 1536 bytes in 32-bit platform.

(2) Methods to reduce the libc-wasi (without -nostdlib) mode footprint

Most of the above methods are also available for libc-wasi mode, besides them, we can export malloc and free functions with -Wl,--export=malloc -Wl,--export=free option, so WAMR runtime will disable its app heap and call the malloc/free function exported to allocate/free the memory from/to the heap space managed by libc.

Note: wasm-ld from LLVM 13 and later automatically inserts ctor/dtor calls for all exported functions for a command. (vs reactor) It breaks the malloc/free exports mentioned above.

3. Build wasm app with pthread support

Please ref to pthread library for more details.

4. Build wasm app with SIMD support

The official wasi-sdk release doesn't fully support latest 128-bit SIMD spec yet. WARM provides a script in build-wasi-sdk to generate another wasi-sdk with llvm-13 from source code and installs it at ../test-tools/wasi-sdk. If you plan to build WASM applications with latest 128-bit SIMD, please use it instead of the official release.

And also you can install emsdk and use its SSE header files, please ref to workload samples, e.g. bwa CMakeLists.txt and wasm-av1 CMakeLists.txt for more details.

For both wasi-sdk and emsdk, please add the option -msimd128 for clang or emcc to generate WASM application with SIMD bytecodes.

Build WASM applications with emsdk

1. Install emsdk

Assuming you are using Linux, you may install emcc and em++ from Emscripten EMSDK following the steps below:

git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
./emsdk install latest
./emsdk activate latest
# And then source the emsdk_env.sh script before build wasm app
source emsdk_env.sh    (or add it to ~/.bashrc if you don't want to run it each time)

The Emscripten website provides other installation methods beyond Linux.

2. emsdk options

To build the wasm C source code into wasm binary, we can use the following command:

EMCC_ONLY_FORCED_STDLIBS=1 emcc -O3 -s STANDALONE_WASM=1 \
    -o test.wasm test.c \
    -s TOTAL_STACK=4096 -s TOTAL_MEMORY=65536 \
    -s "EXPORTED_FUNCTIONS=['_main']" \
    -s ERROR_ON_UNDEFINED_SYMBOLS=0

There are some useful options:

• EMCC_ONLY_FORCED_STDLIBS=1 whether to link libc library into the output binary or not, similar to -nostdlib option of wasi-sdk clang. If specified, then no libc library is linked and the libc-builtin library of WAMR must be built to run the wasm app, otherwise, the libc-wasi library must be built. You can specify -DWAMR_BUILD_LIBC_BUILTIN=1 or -DWAMR_BUILD_LIBC_WASI=1 for CMake to build WAMR with libc-builtin support or libc-wasi support.

  The emsdk's wasi implementation is incomplete, e.g. open a file might just return fail, so it is strongly not recommended to use this mode, especially when there are file io operations in wasm app, please use wasi-sdk instead.

• -s STANDALONE_WASM=1 build wasm app in standalone mode (non-web mode), if the output file has suffix ".wasm", then only wasm file is generated (without html file and JavaScript file).

• -s TOTAL_STACK=<value> the auxiliary stack size, same as -z stack-size=\<value\> of wasi-sdk

• -s TOTAL_MEMORY=<value> or -s INITIAL_MEORY=<value> the initial linear memory size

• -s MAXIMUM_MEMORY=<value> the maximum linear memory size, only take effect if -s ALLOW_MEMORY_GROWTH=1 is set

• -s ALLOW_MEMORY_GROWTH=1/0 whether the linear memory is allowed to grow or not

• -s "EXPORTED_FUNCTIONS=['func name1', 'func name2']" to export functions

• -s ERROR_ON_UNDEFINED_SYMBOLS=0 disable the errors when there are undefined symbols

For more options, please ref to <EMSDK_DIR>/upstream/emscripten/src/settings.js, or Emscripten document.

Build a project with cmake

If you have a complex WASM application project which contains dozens of source files, you can consider using cmake for project building.

You can cross compile your project by using the toolchain provided by WAMR.

Assume the original CMakeLists.txt for test.c likes below:

cmake_minimum_required (VERSION 3.5)
project(hello_world)
add_executable(hello_world test.c)

It is easy to use wasi-sdk in CMake by setting CMAKE_TOOLCHAIN_FILE without any modification on the original CMakeLists.txt.

$ cmake -DWASI_SDK_PREFIX=${WASI_SDK_INSTALLTION_DIR}
        -DCMAKE_TOOLCHAIN_FILE=${WASI_SDK_INSTALLTION_DIR}/share/cmake/wasi-sdk.cmake
        -DCMAKE_SYSROOT=<a sysroot directory>
        ..

WASI_SDK_INSTALLTION_DIR is the directory in where you install the wasi-sdk. like /opt/wasi-sdk

If you prefer WASI, set CMAKE_SYSROOT to wasi-sysroot

$ cmake <same as above>
        -DCMAKE_SYSROOT=${WASI_SDK_INSTALLTION_DIR}/share/wasi-sysroot
        ..

If you prefer WAMR builtin libc, set CMAKE_SYSROOT to libc-builtin-sysroot

Note: If you have already built a SDK profile

$ cmake <same as above>
        -DCMAKE_SYSROOT=${WAMR_SOURCE_ROOT}/wamr-sdk/app/libc-builtin-sysroot
        ..

You will get hello_world which is the WASM app binary.

Compile WASM to AOT module

Please ensure the wamrc was already generated and available in your shell PATH. Then we can use wamrc to compile WASM app binary to WAMR AOT binary.

wamrc -o test.aot test.wasm

wamrc supports a number of compilation options through the command line arguments:

wamrc --help
Usage: wamrc [options] -o output_file wasm_file
  --target=<arch-name>      Set the target arch, which has the general format: <arch><sub>
                            <arch> = x86_64, i386, aarch64, arm, thumb, xtensa, mips,
                                     riscv64, riscv32.
                              Default is host arch, e.g. x86_64
                            <sub> = for ex. on arm or thumb: v5, v6m, v7a, v7m, etc.
                            Use --target=help to list supported targets
  --target-abi=<abi>        Set the target ABI, e.g. gnu, eabi, gnueabihf, msvc, etc.
                              Default is gnu if target isn't riscv64 or riscv32
                              For target riscv64 and riscv32, default is lp64d and ilp32d
                            Use --target-abi=help to list all the ABI supported
  --cpu=<cpu>               Set the target CPU (default: host CPU, e.g. skylake)
                            Use --cpu=help to list all the CPU supported
  --cpu-features=<features> Enable or disable the CPU features
                            Use +feature to enable a feature, or -feature to disable it
                            For example, --cpu-features=+feature1,-feature2
                            Use --cpu-features=+help to list all the features supported
  --opt-level=n             Set the optimization level (0 to 3, default is 3)
  --size-level=n            Set the code size level (0 to 3, default is 3)
  -sgx                      Generate code for SGX platform (Intel Software Guard Extention)
  --bounds-checks=1/0       Enable or disable the bounds checks for memory access:
                              by default it is disabled in all 64-bit platforms except SGX and
                              in these platforms runtime does bounds checks with hardware trap,
                              and by default it is enabled in all 32-bit platforms
  --format=<format>         Specifies the format of the output file
                            The format supported:
                              aot (default)  AoT file
                              object         Native object file
                              llvmir-unopt   Unoptimized LLVM IR
                              llvmir-opt     Optimized LLVM IR
  --disable-bulk-memory     Disable the MVP bulk memory feature
  --enable-multi-thread     Enable multi-thread feature, the dependent features bulk-memory and
                            thread-mgr will be enabled automatically
  --enable-tail-call        Enable the post-MVP tail call feature
  --disable-simd            Disable the post-MVP 128-bit SIMD feature:
                              currently 128-bit SIMD is only supported for x86-64 target,
                              and by default it is enabled in x86-64 target and disabled
                              in other targets
  --disable-ref-types       Disable the MVP reference types feature
  --disable-aux-stack-check Disable auxiliary stack overflow/underflow check
  --enable-dump-call-stack  Enable stack trace feature
  --enable-perf-profiling   Enable function performance profiling
  -v=n                      Set log verbose level (0 to 5, default is 2), larger with more log
Examples: wamrc -o test.aot test.wasm
          wamrc --target=i386 -o test.aot test.wasm
          wamrc --target=i386 --format=object -o test.o test.wasm

AoT-compiled module compatibility among WAMR versions

When making major ABI changes for AoT-compiled modules, we bump AOT_CURRENT_VERSION constant in core/config.h header. The runtime rejects to load a module AoT-compiled with wamrc with a non-compatibleAOT_CURRENT_VERSION.

We try our best to maintain our runtime ABI for AoT-compiled modules compatible among WAMR versions with compatible AOT_CURRENT_VERSION so that combinations of older wamrc and newer runtime usually work. However, there might be minor incompatibilities time to time. For productions, we recommend to use compatible versions of wamrc and the runtime.

WAMR version	AOT_CURRENT_VERSION	Compatible AOT version
1.x	3	3
2.0.0	3	3
2.1.x	3	3
2.2.0	3	3
next	4	3,4

AoT compilation with 3rd-party toolchains

wamrc uses LLVM to compile wasm bytecode to AoT file, this works for most of the architectures, but there may be circumstances where you want to use 3rd-party toolchains to take over some steps of the compilation pipeline, e.g.

1. The upstream LLVM doesn't support generating object file for your CPU architecture (such as ARC), then we may need some other assembler to do such things.
2. You may get some other LLVM-based toolchains which may have better optimizations for the specific target, then you may want your toolchain to take over all optimization steps.

wamrc provides two environment variables to achieve these:

• WAMRC_LLC_COMPILER

  When specified, wamrc will emit the optimized LLVM-IR (.bc) to a file, and invoke $WAMRC_LLC_COMPILER with -c -O3 to generate the object file.

  Optionally, you can use environment variable WAMRC_LLC_FLAGS to overwrite the default flags.

• WAMRC_ASM_COMPILER

  When specified, wamrc will emit the text based assembly file (.s), and invoke $WAMRC_ASM_COMPILER with -c -O3 to generate the object file.

  Optionally, you can use environment variable WAMRC_ASM_FLAGS to overwrite the default flags.

Usage example

WAMRC_LLC_COMPILER=/usr/local/opt/llvm@14/bin/clang WAMRC_LLC_FLAGS="--target=x86_64-pc-linux-gnu -mcmodel=medium -c -O3" ./wamrc -o test.aot test.wasm

Note: wamrc will verify whether the specified file exists and executable. If verification failed, wamrc will report a warning and fallback to normal pipeline. Since the verification is based on file, you must specify the absolute path to the binary even if it's in $PATH

Note: WAMRC_LLC_COMPILER has higher priority than WAMRC_ASM_COMPILER, if WAMRC_LLC_COMPILER is set and verified, then WAMRC_ASM_COMPILER will be ignored.

Note: the LLC and ASM in the env name just means this compiler will be used to compile the LLVM IR file/assembly file to object file, usually passing the compiler driver is the simplest way. (e.g. for LLVM toolchain, you don't need to pass /usr/bin/llc, using /usr/bin/clang is OK)

Note: You might need to set WAMRC_LLC_FLAGS/WAMRC_ASM_FLAGS to match whatever the wamrc command would automatically do. In the above example, -mcmodel=medium corresponds to wamrc --size-level=1, which is the default of wamrc on macOS.

Run WASM app in WAMR mini product build

Run the test.wasm or test.aot with WAMR mini product build:

./iwasm test.wasm   or
./iwasm test.aot

You will get the following output:

Hello world!
buf ptr: 0xffffc2c8
buf: 1234

If you would like to run the test app on Zephyr, we have embedded a test sample into its OS image. You will need to execute:

ninja run