diff --git a/ATTRIBUTIONS.md b/ATTRIBUTIONS.md index 59239b72a..04b5fa25b 100644 --- a/ATTRIBUTIONS.md +++ b/ATTRIBUTIONS.md @@ -8,6 +8,8 @@ WAMR project reused some components from other open source project: - **freebsd libm**: used in core/shared/platform/alios/bh_math.c - **littlevgl**: for the gui samples and wrapped the wasm graphic layer. +The WAMR fast interpreter is a clean room development. We would acknowledge the inspirations by [WASM3](https://github.com/wasm3/wasm3) open source project for the approach of pre-calculated oprand stack location. + ## Licenses ### wasmtime diff --git a/README.md b/README.md index d16353ea8..5b7fbcbe0 100644 --- a/README.md +++ b/README.md @@ -20,12 +20,13 @@ iwasm VM core ### key features -- [Embeddable with the supporting C API's](./doc/embed_wamr.md) +- 100% compliant to the W3C WASM MVP - Small runtime binary size (85K for interpreter and 50K for AoT) and low memory usage -- Near to native speed by AoT -- Unique AoT support for embedded systems which have no system loaders +- Near to native speed by AoT +- Self-implemented module loader enables AoT working cross Linux, SGX and MCU systems - Choices of WASM application libc support: the built-in libc subset for the embedded environment or [WASI](https://github.com/WebAssembly/WASI) for standard libc -- [The mechanism for exporting native API's to WASM applications](./doc/export_native_api.md) +- [Embeddable with the supporting C API's](./doc/embed_wamr.md) +- [The mechanism for exporting native API's to WASM applications](./doc/export_native_api.md) ### Supported architectures and platforms @@ -36,11 +37,11 @@ The iwasm supports the following architectures: - MIPS - XTENSA -Following platforms are supported: +Following platforms are supported. Refer to [WAMR porting guide](./doc/port_wamr.md) for how to port WAMR to a new platform. - [Linux](./doc/build_wamr.md#linux), [Zephyr](./doc/build_wamr.md#zephyr), [MacOS](./doc/build_wamr.md#macos), [VxWorks](./doc/build_wamr.md#vxworks), [AliOS-Things](./doc/build_wamr.md#alios-things), [Intel Software Guard Extention (Linux)](./doc/build_wamr.md#linux-sgx-intel-software-guard-extention), [Android](./doc/build_wamr.md#android) -Refer to [WAMR porting guide](./doc/port_wamr.md) for how to port WAMR to a new platform. + ### Build wamrc AoT compiler @@ -76,27 +77,26 @@ Browse the folder [core/app-framework](./core/app-framework) for how to extend # Remote application management -The WAMR application manager supports remote application management from the host environment or the cloud through any physical communications such as TCP, UPD, UART, BLE, etc. Its modular design makes it able to support application management for different managed runtimes. +The WAMR application manager supports [remote application management](./core/app-mgr) from the host environment or the cloud through any physical communications such as TCP, UPD, UART, BLE, etc. Its modular design makes it able to support application management for different managed runtimes. The tool [host_tool](./test-tools/host-tool) communicates to the WAMR app manager for installing/uninstalling the WASM applications on companion chip from the host system. And the [IoT App Store Demo](./test-tools/IoT-APP-Store-Demo/) shows the conception of remotely managing the device applications from the cloud. -Browse the folder [core/app-mgr](./core/app-mgr) for the details. WAMR SDK ========== Usually there are two tasks for integrating the WAMR into a particular project: -- Select what WAMR components (vmcore, libc, app-mgr, app-framework components) to be integrated, and get the associated source files added into the project building configuration +- Select what WAMR components (vmcore, libc, app-mgr, app-framework components) to be integrated, and get the associated source files added into the project building configuration - Generate the APP SDK for developing the WASM apps on the selected libc and framework components -The **[WAMR SDK](./wamr-sdk)** tools is helpful to finish the two tasks quickly. It supports menu configuration for selecting WAMR components and builds the WAMR to a SDK package that includes **runtime SDK** and **APP SDK**. The runtime SDK is used for building the native application and the APP SDK should be shipped to WASM application developers. +The **[WAMR SDK](./wamr-sdk)** tools is helpful to finish the two tasks quickly. It supports menu configuration for selecting WAMR components and builds the WAMR to a SDK package that includes **runtime SDK** and **APP SDK**. The runtime SDK is used for building the native application and the APP SDK should be shipped to WASM application developers. Samples ================= -The WAMR samples integrate the iwasm VM core, application manager and selected application framework components. The samples are located in folder [samples](./samples): +The WAMR [samples](./samples) integrate the iwasm VM core, application manager and selected application framework components. - **[Simple](./samples/simple/README.md)**: The runtime is integrated with most of the WAMR APP libraries, and a few WASM applications are provided for testing the WAMR APP API set. It uses **built-in libc** and executes apps in **interpreter** mode by default. - **[littlevgl](./samples/littlevgl/README.md)**: Demonstrating the graphic user interface application usage on WAMR. The whole [LittlevGL](https://github.com/littlevgl/) 2D user graphic library and the UI application is built into WASM application. It uses **WASI libc** and executes apps in **AoT mode** by default. - **[gui](./samples/gui/README.md)**: Moved the [LittlevGL](https://github.com/littlevgl/) library into the runtime and defined a WASM application interface by wrapping the littlevgl API. It uses **WASI libc** and executes apps in **interpreter** mode by default. diff --git a/build-scripts/config_common.cmake b/build-scripts/config_common.cmake index 6d4c02851..8cb6ed141 100644 --- a/build-scripts/config_common.cmake +++ b/build-scripts/config_common.cmake @@ -127,4 +127,11 @@ if (WAMR_BUILD_LIBC_WASI EQUAL 1) else () message (" Libc WASI disbled") endif () +if (WAMR_BUILD_FAST_INTERP EQUAL 1) + add_definitions (-DWASM_ENABLE_FAST_INTERP=1) + message (" Fast interpreter enabled") +else () + add_definitions (-DWASM_ENABLE_FAST_INTERP=0) + message (" Fast interpreter disbled") +endif () diff --git a/core/app-framework/README.md b/core/app-framework/README.md index 0deb4af86..34f0e27eb 100644 --- a/core/app-framework/README.md +++ b/core/app-framework/README.md @@ -7,7 +7,7 @@ Application framework This folder "app-native-shared" is for the source files shared by both WASM APP and native runtime -- The c files in this directory are compiled into both the WASM APP and runtime. +- The c files in this directory are compiled into both the WASM APP and runtime. - The header files for distributing to SDK are placed in the "bi-inc" folder. @@ -16,7 +16,7 @@ This folder "template" contains a pre-defined directory structure for a framewor -Every other subfolder is framework component. Each component contains two library parts: **app and native**. +Every other subfolder is framework component. Each component contains two library parts: **app and native**. - The "base" component provide timer API and inter-app communication support. It must be enabled if other components are selected. - Under the "app" folder of a component, the subfolder "wa_inc" holds all header files that should be included by the WASM applications @@ -85,13 +85,13 @@ Generally you should follow following steps to create a new component: ```c //use right signature for your functions - EXPORT_WASM_API_WITH_SIG(wasm_my_component_api_1, "(i*~)i"), + EXPORT_WASM_API_WITH_SIG(wasm_my_component_api_1, "(i*~)i"), EXPORT_WASM_API_WITH_SIG(wasm_my_component_api_2, "(i)i"), ``` - Ensure "wasm_lib.cmake" is provided as it will be included by the WAMR SDK building script - - Add a definition in "wasm_lib.cmake" for your component, e.g. + - Add a definition in "wasm_lib.cmake" for your component, e.g. ```cmake add_definitions (-DAPP_FRAMEWORK_MY_COMPONENT) @@ -101,12 +101,12 @@ Generally you should follow following steps to create a new component: ``` #include "lib_export.h" - + ... #ifdef APP_FRAMEWORK_MY_COMPONENT // this definition is created in wasm_lib.cmake #include "my_component_native_api.h" #endif - + static NativeSymbol extended_native_symbol_defs[] = { ... #ifdef APP_FRAMEWORK_MY_COMPONENT @@ -115,7 +115,7 @@ Generally you should follow following steps to create a new component: }; ``` - + ## Sensor component working flow diff --git a/core/app-framework/base/native/base_lib_export.c b/core/app-framework/base/native/base_lib_export.c index ee90f1da9..0b716fc5e 100644 --- a/core/app-framework/base/native/base_lib_export.c +++ b/core/app-framework/base/native/base_lib_export.c @@ -17,7 +17,7 @@ static NativeSymbol extended_native_symbol_defs[] = { #include "base_lib.inl" }; -int get_base_lib_export_apis(NativeSymbol **p_base_lib_apis) +uint32 get_base_lib_export_apis(NativeSymbol **p_base_lib_apis) { *p_base_lib_apis = extended_native_symbol_defs; return sizeof(extended_native_symbol_defs) / sizeof(NativeSymbol); diff --git a/core/iwasm/aot/aot_loader.c b/core/iwasm/aot/aot_loader.c index 211ba61f9..84018e88d 100644 --- a/core/iwasm/aot/aot_loader.c +++ b/core/iwasm/aot/aot_loader.c @@ -1703,7 +1703,7 @@ create_sections(const uint8 *buf, uint32 size, memset(section, 0, sizeof(AOTSection)); section->section_type = (int32)section_type; - section->section_body = p; + section->section_body = (uint8*)p; section->section_body_size = section_size; if (section_type == AOT_SECTION_TYPE_TEXT) { diff --git a/core/iwasm/common/wasm_native.h b/core/iwasm/common/wasm_native.h index dcacd5826..039987848 100644 --- a/core/iwasm/common/wasm_native.h +++ b/core/iwasm/common/wasm_native.h @@ -7,18 +7,13 @@ #define _WASM_NATIVE_H #include "bh_common.h" +#include "../include/wasm_export.h" #include "../interpreter/wasm.h" #ifdef __cplusplus extern "C" { #endif -typedef struct NativeSymbol { - const char *symbol; - void *func_ptr; - const char *signature; -} NativeSymbol; - typedef struct NativeSymbolsNode { struct NativeSymbolsNode *next; const char *module_name; diff --git a/core/iwasm/common/wasm_runtime_common.c b/core/iwasm/common/wasm_runtime_common.c index 8aaa97674..bdee993f5 100644 --- a/core/iwasm/common/wasm_runtime_common.c +++ b/core/iwasm/common/wasm_runtime_common.c @@ -50,6 +50,49 @@ wasm_runtime_destroy() vm_thread_sys_destroy(); } +bool +wasm_runtime_full_init(RuntimeInitArgs *init_args) +{ + if (init_args->mem_alloc_type == Alloc_With_Pool) { + void *heap_buf = init_args->mem_alloc.pool.heap_buf; + uint32 heap_size = init_args->mem_alloc.pool.heap_size; + if (bh_memory_init_with_pool(heap_buf, heap_size) != 0) + return false; + } + else if (init_args->mem_alloc_type == Alloc_With_Allocator) { + void *malloc_func = init_args->mem_alloc.allocator.malloc_func; + void *free_func = init_args->mem_alloc.allocator.free_func; + if (bh_memory_init_with_allocator(malloc_func, free_func) != 0) + return false; + } + else + return false; + + if (!wasm_runtime_init()) + goto fail1; + + if (init_args->n_native_symbols > 0 + && !wasm_runtime_register_natives(init_args->native_module_name, + init_args->native_symbols, + init_args->n_native_symbols)) + goto fail2; + + return true; + +fail2: + wasm_runtime_destroy(); +fail1: + bh_memory_destroy(); + return false; +} + +void +wasm_runtime_full_destroy() +{ + wasm_runtime_destroy(); + bh_memory_destroy(); +} + PackageType get_package_type(const uint8 *buf, uint32 size) { @@ -202,7 +245,7 @@ wasm_runtime_get_module_inst(WASMExecEnv *exec_env) } WASMFunctionInstanceCommon * -wasm_runtime_lookup_function(const WASMModuleInstanceCommon *module_inst, +wasm_runtime_lookup_function(WASMModuleInstanceCommon * const module_inst, const char *name, const char *signature) { @@ -339,7 +382,7 @@ wasm_runtime_get_custom_data(WASMModuleInstanceCommon *module_inst) #if WASM_ENABLE_INTERP != 0 if (module_inst->module_type == Wasm_Module_Bytecode) return ((WASMModuleInstance*)module_inst)->custom_data; -#endif +#endif #if WASM_ENABLE_AOT != 0 if (module_inst->module_type == Wasm_Module_AoT) return ((AOTModuleInstance*)module_inst)->custom_data.ptr; @@ -619,7 +662,7 @@ wasm_runtime_set_wasi_args(WASMModuleCommon *module, const char *dir_list[], uint32 dir_count, const char *map_dir_list[], uint32 map_dir_count, const char *env_list[], uint32 env_count, - const char *argv[], uint32 argc) + char *argv[], int argc) { WASIArguments *wasi_args = NULL; @@ -649,7 +692,7 @@ wasm_runtime_init_wasi(WASMModuleInstanceCommon *module_inst, const char *dir_list[], uint32 dir_count, const char *map_dir_list[], uint32 map_dir_count, const char *env[], uint32 env_count, - const char *argv[], uint32 argc, + char *argv[], uint32 argc, char *error_buf, uint32 error_buf_size) { WASIContext *wasi_ctx; diff --git a/core/iwasm/common/wasm_runtime_common.h b/core/iwasm/common/wasm_runtime_common.h index 11fc5c797..28b279c40 100644 --- a/core/iwasm/common/wasm_runtime_common.h +++ b/core/iwasm/common/wasm_runtime_common.h @@ -11,6 +11,7 @@ #include "bh_thread.h" #include "wasm_exec_env.h" #include "wasm_native.h" +#include "../include/wasm_export.h" #include "../interpreter/wasm.h" #if WASM_ENABLE_LIBC_WASI != 0 #include "wasmtime_ssp.h" @@ -24,13 +25,6 @@ extern "C" { #define wasm_malloc bh_malloc #define wasm_free bh_free -/* Package Type */ -typedef enum { - Wasm_Module_Bytecode = 0, - Wasm_Module_AoT, - Package_Type_Unknown = 0xFFFF -} PackageType; - typedef struct WASMModuleCommon { /* Module type, for module loaded from WASM bytecode binary, this field is Wasm_Module_Bytecode, and this structure should @@ -53,19 +47,6 @@ typedef struct WASMModuleInstanceCommon { uint8 module_inst_data[1]; } WASMModuleInstanceCommon; -typedef void WASMFunctionInstanceCommon; - -/* WASM section */ -typedef struct WASMSection { - struct WASMSection *next; - /* section type */ - int section_type; - /* section body, not include type and size */ - const uint8 *section_body; - /* section body size */ - uint32 section_body_size; -} WASMSection, AOTSection; - #if WASM_ENABLE_LIBC_WASI != 0 typedef struct WASIContext { struct fd_table *curfds; @@ -74,6 +55,9 @@ typedef struct WASIContext { } WASIContext; #endif +typedef package_type_t PackageType; +typedef wasm_section_t WASMSection, AOTSection; + /* See wasm_export.h for description */ bool wasm_runtime_init(); @@ -82,6 +66,14 @@ wasm_runtime_init(); void wasm_runtime_destroy(); +/* See wasm_export.h for description */ +bool +wasm_runtime_full_init(RuntimeInitArgs *init_args); + +/* See wasm_export.h for description */ +void +wasm_runtime_full_destroy(); + /* See wasm_export.h for description */ PackageType get_package_type(const uint8 *buf, uint32 size); @@ -112,7 +104,7 @@ wasm_runtime_deinstantiate(WASMModuleInstanceCommon *module_inst); /* See wasm_export.h for description */ WASMFunctionInstanceCommon * -wasm_runtime_lookup_function(const WASMModuleInstanceCommon *module_inst, +wasm_runtime_lookup_function(WASMModuleInstanceCommon * const module_inst, const char *name, const char *signature); /* See wasm_export.h for description */ @@ -245,7 +237,7 @@ wasm_runtime_set_wasi_args(WASMModuleCommon *module, const char *dir_list[], uint32 dir_count, const char *map_dir_list[], uint32 map_dir_count, const char *env_list[], uint32 env_count, - const char *argv[], uint32 argc); + char *argv[], int argc); /* See wasm_export.h for description */ bool @@ -260,7 +252,7 @@ wasm_runtime_init_wasi(WASMModuleInstanceCommon *module_inst, const char *dir_list[], uint32 dir_count, const char *map_dir_list[], uint32 map_dir_count, const char *env[], uint32 env_count, - const char *argv[], uint32 argc, + char *argv[], uint32 argc, char *error_buf, uint32 error_buf_size); void diff --git a/core/iwasm/include/lib_export.h b/core/iwasm/include/lib_export.h index 97e043e11..8489afc49 100644 --- a/core/iwasm/include/lib_export.h +++ b/core/iwasm/include/lib_export.h @@ -6,6 +6,8 @@ #ifndef _LIB_EXPORT_H_ #define _LIB_EXPORT_H_ +#include + #ifdef __cplusplus extern "C" { #endif @@ -31,25 +33,9 @@ typedef struct NativeSymbol { * * @return the number of the exported API */ -int +uint32_t get_base_lib_export_apis(NativeSymbol **p_base_lib_apis); -/** - * Get the exported APIs of extended lib, this API isn't provided by WASM VM, - * it must be provided by developer to register the extended native APIs, - * for example, developer can register his native APIs to extended_native_symbol_defs, - * array, and include file ext_lib_export.h which implements this API. - * And if developer hasn't any native API to register, he can define an empty - * extended_native_symbol_defs array, and then include file ext_lib_export.h to - * implements this API. - * - * @param p_base_lib_apis return the exported API array of extend lib - * - * @return the number of the exported API - */ -int -get_extend_lib_export_apis(NativeSymbol **p_base_lib_apis); - #ifdef __cplusplus } #endif diff --git a/core/iwasm/include/wasm_export.h b/core/iwasm/include/wasm_export.h index a0654c496..635da7ff1 100644 --- a/core/iwasm/include/wasm_export.h +++ b/core/iwasm/include/wasm_export.h @@ -9,6 +9,7 @@ #include #include #include "lib_export.h" +#include "bh_memory.h" #ifdef __cplusplus @@ -25,21 +26,19 @@ struct WASMModuleInstanceCommon; typedef struct WASMModuleInstanceCommon *wasm_module_inst_t; /* Function instance */ -struct WASMFunctionInstanceCommon; -typedef struct WASMFunctionInstanceCommon *wasm_function_inst_t; +typedef void WASMFunctionInstanceCommon; +typedef WASMFunctionInstanceCommon *wasm_function_inst_t; /* WASM section */ -typedef struct wasm_section { - struct wasm_section *next; +typedef struct wasm_section_t { + struct wasm_section_t *next; /* section type */ int section_type; /* section body, not include type and size */ uint8_t *section_body; /* section body size */ uint32_t section_body_size; -} wasm_section_t, *wasm_section_list_t; - -typedef wasm_section_t aot_section_t, *aot_section_list_t; +} wasm_section_t, aot_section_t, *wasm_section_list_t, *aot_section_list_t; /* Execution environment, e.g. stack info */ struct WASMExecEnv; @@ -52,6 +51,31 @@ typedef enum { Package_Type_Unknown = 0xFFFF } package_type_t; +/* Memory allocator type */ +typedef enum { + Alloc_With_Pool = 0, + Alloc_With_Allocator +} mem_alloc_type_t; + +/* WASM runtime initialize arguments */ +typedef struct RuntimeInitArgs { + mem_alloc_type_t mem_alloc_type; + union { + struct { + void *heap_buf; + uint32_t heap_size; + } pool; + struct { + void *malloc_func; + void *free_func; + } allocator; + } mem_alloc; + + const char *native_module_name; + NativeSymbol *native_symbols; + uint32_t n_native_symbols; +} RuntimeInitArgs; + /** * Initialize the WASM runtime environment. * @@ -66,6 +90,25 @@ wasm_runtime_init(); void wasm_runtime_destroy(); +/** + * Initialize the WASM runtime environment, and also initialize + * the memory allocator and register native symbols, which are specified + * with init arguments + * + * @param init_args specifies the init arguments + * + * @return return true if success, false otherwise + */ +bool +wasm_runtime_full_init(RuntimeInitArgs *init_args); + +/** + * Destroy the wasm runtime environment, and also destroy + * the memory allocator and registered native symbols + */ +void +wasm_runtime_full_destroy(); + /** * Get the package type of a buffer. * @@ -167,7 +210,7 @@ wasm_runtime_lookup_wasi_start_function(wasm_module_inst_t module_inst); * @return the function instance found */ wasm_function_inst_t -wasm_runtime_lookup_function(const wasm_module_inst_t module_inst, +wasm_runtime_lookup_function(wasm_module_inst_t const module_inst, const char *name, const char *signature); /** diff --git a/core/iwasm/interpreter/iwasm_interp.cmake b/core/iwasm/interpreter/iwasm_interp.cmake index 9ac035b1c..c5d72b233 100644 --- a/core/iwasm/interpreter/iwasm_interp.cmake +++ b/core/iwasm/interpreter/iwasm_interp.cmake @@ -7,7 +7,17 @@ add_definitions (-DWASM_ENABLE_INTERP=1) include_directories(${IWASM_INTERP_DIR}) -file (GLOB_RECURSE source_all ${IWASM_INTERP_DIR}/*.c) +if (WAMR_BUILD_FAST_INTERP EQUAL 1) + set (INTERPRETER "wasm_interp_fast.c") +else () + set (INTERPRETER "wasm_interp.c") +endif () + +file (GLOB_RECURSE source_all + ${IWASM_INTERP_DIR}/wasm_loader.c + ${IWASM_INTERP_DIR}/wasm_runtime.c + ${IWASM_INTERP_DIR}/${INTERPRETER} +) set (IWASM_INTERP_SOURCE ${source_all}) diff --git a/core/iwasm/interpreter/wasm.h b/core/iwasm/interpreter/wasm.h index f730d7a11..d1c6b661e 100644 --- a/core/iwasm/interpreter/wasm.h +++ b/core/iwasm/interpreter/wasm.h @@ -189,6 +189,12 @@ typedef struct WASMFunction { bool has_op_func_call; uint32 code_size; uint8 *code; +#if WASM_ENABLE_FAST_INTERP != 0 + uint32 code_compiled_size; + uint8 *code_compiled; + uint8 *consts; + uint32 const_cell_num; +#endif } WASMFunction; typedef struct WASMGlobal { @@ -231,7 +237,7 @@ typedef struct WASIArguments { uint32 map_dir_count; const char **env; uint32 env_count; - const char **argv; + char **argv; uint32 argc; } WASIArguments; #endif diff --git a/core/iwasm/interpreter/wasm_interp.c b/core/iwasm/interpreter/wasm_interp.c index bcba0d516..6141f1c2e 100644 --- a/core/iwasm/interpreter/wasm_interp.c +++ b/core/iwasm/interpreter/wasm_interp.c @@ -225,27 +225,24 @@ LOAD_I16(void *addr) #endif /* WASM_CPU_SUPPORTS_UNALIGNED_64BIT_ACCESS != 0 */ #define CHECK_MEMORY_OVERFLOW() do { \ - uint32 offset1 = offset + addr; \ + uint64 offset1 = offset + addr; \ /* if (flags != 2) \ LOG_VERBOSE("unaligned load/store in wasm interp, flag: %d.\n", flags); */\ /* The WASM spec doesn't require that the dynamic address operand must be \ unsigned, so we don't check whether integer overflow or not here. */ \ /* if (offset1 < offset) \ goto out_of_bounds; */ \ - if (offset1 < memory_data_size) { \ + if (offset1 + LOAD_SIZE[opcode - WASM_OP_I32_LOAD] <= memory_data_size) { \ /* If offset1 is in valid range, maddr must also be in valid range, \ no need to check it again. */ \ maddr = memory->memory_data + offset1; \ - if (maddr + LOAD_SIZE[opcode - WASM_OP_I32_LOAD] > memory->end_addr) \ - goto out_of_bounds; \ } \ - else if (offset1 > heap_base_offset \ - && offset1 < heap_base_offset + heap_data_size) { \ + else if (offset1 > DEFAULT_APP_HEAP_BASE_OFFSET \ + && (offset1 + LOAD_SIZE[opcode - WASM_OP_I32_LOAD] <= \ + DEFAULT_APP_HEAP_BASE_OFFSET + heap_data_size)) { \ /* If offset1 is in valid range, maddr must also be in valid range, \ no need to check it again. */ \ maddr = memory->heap_data + offset1 - memory->heap_base_offset; \ - if (maddr + LOAD_SIZE[opcode - WASM_OP_I32_LOAD] > memory->heap_data_end) \ - goto out_of_bounds; \ } \ else \ goto out_of_bounds; \ @@ -795,7 +792,6 @@ wasm_interp_call_func_bytecode(WASMModuleInstance *module, WASMMemoryInstance *memory = module->default_memory; uint32 num_bytes_per_page = memory ? memory->num_bytes_per_page : 0; uint32 memory_data_size = memory ? num_bytes_per_page * memory->cur_page_count : 0; - uint32 heap_base_offset = memory ? (uint32)memory->heap_base_offset : 0; uint32 heap_data_size = memory ? (uint32)(memory->heap_data_end - memory->heap_data) : 0; uint8 *global_data = memory ? memory->global_data : NULL; WASMTableInstance *table = module->default_table; @@ -1080,7 +1076,7 @@ wasm_interp_call_func_bytecode(WASMModuleInstance *module, HANDLE_OP_END (); } - HANDLE_OP (WASM_OP_GET_LOCAL_FAST): + HANDLE_OP (EXT_OP_GET_LOCAL_FAST): { local_offset = *frame_ip++; if (local_offset & 0x80) @@ -1111,7 +1107,7 @@ wasm_interp_call_func_bytecode(WASMModuleInstance *module, HANDLE_OP_END (); } - HANDLE_OP (WASM_OP_SET_LOCAL_FAST): + HANDLE_OP (EXT_OP_SET_LOCAL_FAST): { local_offset = *frame_ip++; if (local_offset & 0x80) @@ -1143,7 +1139,7 @@ wasm_interp_call_func_bytecode(WASMModuleInstance *module, HANDLE_OP_END (); } - HANDLE_OP (WASM_OP_TEE_LOCAL_FAST): + HANDLE_OP (EXT_OP_TEE_LOCAL_FAST): { local_offset = *frame_ip++; if (local_offset & 0x80) @@ -2225,6 +2221,11 @@ wasm_interp_call_func_bytecode(WASMModuleInstance *module, HANDLE_OP (WASM_OP_UNUSED_0x25): HANDLE_OP (WASM_OP_UNUSED_0x26): HANDLE_OP (WASM_OP_UNUSED_0x27): + /* Used by fast interpreter */ + HANDLE_OP (EXT_OP_SET_LOCAL_FAST_I64): + HANDLE_OP (EXT_OP_TEE_LOCAL_FAST_I64): + HANDLE_OP (EXT_OP_COPY_STACK_TOP): + HANDLE_OP (EXT_OP_COPY_STACK_TOP_I64): { wasm_set_exception(module, "WASM interp failed: unsupported opcode."); goto got_exception; diff --git a/core/iwasm/interpreter/wasm_interp.h b/core/iwasm/interpreter/wasm_interp.h index 3d7844065..9883d2654 100644 --- a/core/iwasm/interpreter/wasm_interp.h +++ b/core/iwasm/interpreter/wasm_interp.h @@ -26,6 +26,13 @@ typedef struct WASMInterpFrame { /* Instruction pointer of the bytecode array. */ uint8 *ip; +#if WASM_ENABLE_FAST_INTERP != 0 + /* return offset of current frame. + the callee will put return value here */ + uint32 ret_offset; + uint32 *lp; + uint32 operand[1]; +#else /* Operand stack top pointer of the current frame. The bottom of the stack is the next cell after the last local variable. */ uint32 *sp_bottom; @@ -43,6 +50,7 @@ typedef struct WASMInterpFrame { ref to frame end: data types of local vairables and stack data */ uint32 lp[1]; +#endif } WASMInterpFrame; /** diff --git a/core/iwasm/interpreter/wasm_interp_fast.c b/core/iwasm/interpreter/wasm_interp_fast.c new file mode 100644 index 000000000..9e9d33810 --- /dev/null +++ b/core/iwasm/interpreter/wasm_interp_fast.c @@ -0,0 +1,2276 @@ +/* + * Copyright (C) 2019 Intel Corporation. All rights reserved. + * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + */ + +#include "wasm_interp.h" +#include "bh_memory.h" +#include "bh_log.h" +#include "wasm_runtime.h" +#include "wasm_opcode.h" +#include "wasm_loader.h" +#include "../common/wasm_exec_env.h" + +typedef int32 CellType_I32; +typedef int64 CellType_I64; +typedef float32 CellType_F32; +typedef float64 CellType_F64; + +#define BR_TABLE_TMP_BUF_LEN 32 + +/* 64-bit Memory accessors. */ +#if WASM_CPU_SUPPORTS_UNALIGNED_64BIT_ACCESS != 0 +#define PUT_I64_TO_ADDR(addr, value) do { \ + *(int64*)(addr) = (int64)(value); \ + } while (0) +#define PUT_F64_TO_ADDR(addr, value) do { \ + *(float64*)(addr) = (float64)(value); \ + } while (0) + +#define GET_I64_FROM_ADDR(addr) (*(int64*)(addr)) +#define GET_F64_FROM_ADDR(addr) (*(float64*)(addr)) + +/* For STORE opcodes */ +#define STORE_I64 PUT_I64_TO_ADDR +#define STORE_U32(addr, value) do { \ + *(uint32*)(addr) = (uint32)(value); \ + } while (0) +#define STORE_U16(addr, value) do { \ + *(uint16*)(addr) = (uint16)(value); \ + } while (0) + +/* For LOAD opcodes */ +#define LOAD_I64(addr) (*(int64*)(addr)) +#define LOAD_F64(addr) (*(float64*)(addr)) +#define LOAD_F32(addr) (*(float32*)(addr)) +#define LOAD_I32(addr) (*(int32*)(addr)) +#define LOAD_U32(addr) (*(uint32*)(addr)) +#define LOAD_I16(addr) (*(int16*)(addr)) +#define LOAD_U16(addr) (*(uint16*)(addr)) + +#else /* WASM_CPU_SUPPORTS_UNALIGNED_64BIT_ACCESS != 0 */ +#define PUT_I64_TO_ADDR(addr, value) do { \ + union { int64 val; uint32 parts[2]; } u; \ + u.val = (int64)(value); \ + (addr)[0] = u.parts[0]; \ + (addr)[1] = u.parts[1]; \ + } while (0) +#define PUT_F64_TO_ADDR(addr, value) do { \ + union { float64 val; uint32 parts[2]; } u; \ + u.val = (value); \ + (addr)[0] = u.parts[0]; \ + (addr)[1] = u.parts[1]; \ + } while (0) + +static inline int64 +GET_I64_FROM_ADDR(uint32 *addr) +{ + union { int64 val; uint32 parts[2]; } u; + u.parts[0] = addr[0]; + u.parts[1] = addr[1]; + return u.val; +} + +static inline float64 +GET_F64_FROM_ADDR (uint32 *addr) +{ + union { float64 val; uint32 parts[2]; } u; + u.parts[0] = addr[0]; + u.parts[1] = addr[1]; + return u.val; +} + +/* For STORE opcodes */ +#define STORE_I64(addr, value) do { \ + uintptr_t addr1 = (uintptr_t)(addr); \ + union { int64 val; uint32 u32[2]; \ + uint16 u16[4]; uint8 u8[8]; } u; \ + if ((addr1 & (uintptr_t)7) == 0) \ + *(int64*)(addr) = (int64)(value); \ + else { \ + u.val = (int64)(value); \ + if ((addr1 & (uintptr_t)3) == 0) { \ + ((uint32*)(addr))[0] = u.u32[0]; \ + ((uint32*)(addr))[1] = u.u32[1]; \ + } \ + else if ((addr1 & (uintptr_t)1) == 0) { \ + ((uint16*)(addr))[0] = u.u16[0]; \ + ((uint16*)(addr))[1] = u.u16[1]; \ + ((uint16*)(addr))[2] = u.u16[2]; \ + ((uint16*)(addr))[3] = u.u16[3]; \ + } \ + else { \ + int32 t; \ + for (t = 0; t < 8; t++) \ + ((uint8*)(addr))[t] = u.u8[t]; \ + } \ + } \ + } while (0) + +#define STORE_U32(addr, value) do { \ + uintptr_t addr1 = (uintptr_t)(addr); \ + union { uint32 val; \ + uint16 u16[2]; uint8 u8[4]; } u; \ + if ((addr1 & (uintptr_t)3) == 0) \ + *(uint32*)(addr) = (uint32)(value); \ + else { \ + u.val = (uint32)(value); \ + if ((addr1 & (uintptr_t)1) == 0) { \ + ((uint16*)(addr))[0] = u.u16[0]; \ + ((uint16*)(addr))[1] = u.u16[1]; \ + } \ + else { \ + ((uint8*)(addr))[0] = u.u8[0]; \ + ((uint8*)(addr))[1] = u.u8[1]; \ + ((uint8*)(addr))[2] = u.u8[2]; \ + ((uint8*)(addr))[3] = u.u8[3]; \ + } \ + } \ + } while (0) + +#define STORE_U16(addr, value) do { \ + union { uint16 val; uint8 u8[2]; } u; \ + u.val = (uint16)(value); \ + ((uint8*)(addr))[0] = u.u8[0]; \ + ((uint8*)(addr))[1] = u.u8[1]; \ + } while (0) + +/* For LOAD opcodes */ +static inline int64 +LOAD_I64(void *addr) +{ + uintptr_t addr1 = (uintptr_t)addr; + union { int64 val; uint32 u32[2]; + uint16 u16[4]; uint8 u8[8]; } u; + if ((addr1 & (uintptr_t)7) == 0) + return *(int64*)addr; + + if ((addr1 & (uintptr_t)3) == 0) { + u.u32[0] = ((uint32*)addr)[0]; + u.u32[1] = ((uint32*)addr)[1]; + } + else if ((addr1 & (uintptr_t)1) == 0) { + u.u16[0] = ((uint16*)addr)[0]; + u.u16[1] = ((uint16*)addr)[1]; + u.u16[2] = ((uint16*)addr)[2]; + u.u16[3] = ((uint16*)addr)[3]; + } + else { + int32 t; + for (t = 0; t < 8; t++) + u.u8[t] = ((uint8*)addr)[t]; + } + return u.val; +} + +static inline float64 +LOAD_F64(void *addr) +{ + uintptr_t addr1 = (uintptr_t)addr; + union { float64 val; uint32 u32[2]; + uint16 u16[4]; uint8 u8[8]; } u; + if ((addr1 & (uintptr_t)7) == 0) + return *(float64*)addr; + + if ((addr1 & (uintptr_t)3) == 0) { + u.u32[0] = ((uint32*)addr)[0]; + u.u32[1] = ((uint32*)addr)[1]; + } + else if ((addr1 & (uintptr_t)1) == 0) { + u.u16[0] = ((uint16*)addr)[0]; + u.u16[1] = ((uint16*)addr)[1]; + u.u16[2] = ((uint16*)addr)[2]; + u.u16[3] = ((uint16*)addr)[3]; + } + else { + int32 t; + for (t = 0; t < 8; t++) + u.u8[t] = ((uint8*)addr)[t]; + } + return u.val; +} + +static inline int32 +LOAD_I32(void *addr) +{ + uintptr_t addr1 = (uintptr_t)addr; + union { int32 val; uint16 u16[2]; uint8 u8[4]; } u; + if ((addr1 & (uintptr_t)3) == 0) + return *(int32*)addr; + + if ((addr1 & (uintptr_t)1) == 0) { + u.u16[0] = ((uint16*)addr)[0]; + u.u16[1] = ((uint16*)addr)[1]; + } + else { + u.u8[0] = ((uint8*)addr)[0]; + u.u8[1] = ((uint8*)addr)[1]; + u.u8[2] = ((uint8*)addr)[2]; + u.u8[3] = ((uint8*)addr)[3]; + } + return u.val; +} + +static inline int16 +LOAD_I16(void *addr) +{ + union { int16 val; uint8 u8[2]; } u; + u.u8[0] = ((uint8*)addr)[0]; + u.u8[1] = ((uint8*)addr)[1]; + return u.val; +} + +#define LOAD_U32(addr) ((uint32)LOAD_I32(addr)) +#define LOAD_U16(addr) ((uint16)LOAD_I16(addr)) +#define LOAD_F32(addr) ((float32)LOAD_I32(addr)) + +#endif /* WASM_CPU_SUPPORTS_UNALIGNED_64BIT_ACCESS != 0 */ + +#define CHECK_MEMORY_OVERFLOW() do { \ + uint64 offset1 = offset + addr; \ + /* if (flags != 2) \ + LOG_VERBOSE("unaligned load/store in wasm interp, flag: %d.\n", flags); */\ + /* The WASM spec doesn't require that the dynamic address operand must be \ + unsigned, so we don't check whether integer overflow or not here. */ \ + /* if (offset1 < offset) \ + goto out_of_bounds; */ \ + if (offset1 + LOAD_SIZE[opcode - WASM_OP_I32_LOAD] <= memory_data_size) { \ + /* If offset1 is in valid range, maddr must also be in valid range, \ + no need to check it again. */ \ + maddr = memory->memory_data + offset1; \ + } \ + else if (offset1 > DEFAULT_APP_HEAP_BASE_OFFSET \ + && (offset1 + LOAD_SIZE[opcode - WASM_OP_I32_LOAD] <= \ + DEFAULT_APP_HEAP_BASE_OFFSET + heap_data_size)) { \ + /* If offset1 is in valid range, maddr must also be in valid range, \ + no need to check it again. */ \ + maddr = memory->heap_data + offset1 - DEFAULT_APP_HEAP_BASE_OFFSET; \ + } \ + else \ + goto out_of_bounds; \ + } while (0) + +#define CHECK_MEMORY_OVERFLOW_FAST(bytes) do { \ + uint64 offset1 = offset + addr; \ + /* if (flags != 2) \ + LOG_VERBOSE("unaligned load/store in wasm interp, flag: %d.\n", flags); */\ + /* The WASM spec doesn't require that the dynamic address operand must be \ + unsigned, so we don't check whether integer overflow or not here. */ \ + /* if (offset1 < offset) \ + goto out_of_bounds; */ \ + if (offset1 + bytes <= memory_data_size) { \ + /* If offset1 is in valid range, maddr must also be in valid range, \ + no need to check it again. */ \ + maddr = memory->memory_data + offset1; \ + } \ + else if (offset1 > DEFAULT_APP_HEAP_BASE_OFFSET \ + && (offset1 + bytes <= \ + DEFAULT_APP_HEAP_BASE_OFFSET + heap_data_size)) { \ + /* If offset1 is in valid range, maddr must also be in valid range, \ + no need to check it again. */ \ + maddr = memory->heap_data + offset1 - DEFAULT_APP_HEAP_BASE_OFFSET; \ + } \ + else \ + goto out_of_bounds; \ + } while (0) + +static inline uint32 +rotl32(uint32 n, uint32 c) +{ + const uint32 mask = (31); + c = c % 32; + c &= mask; + return (n<>( (-c)&mask )); +} + +static inline uint32 +rotr32(uint32 n, uint32 c) +{ + const uint32 mask = (31); + c = c % 32; + c &= mask; + return (n>>c) | (n<<( (-c)&mask )); +} + +static inline uint64 +rotl64(uint64 n, uint64 c) +{ + const uint64 mask = (63); + c = c % 64; + c &= mask; + return (n<>( (-c)&mask )); +} + +static inline uint64 +rotr64(uint64 n, uint64 c) +{ + const uint64 mask = (63); + c = c % 64; + c &= mask; + return (n>>c) | (n<<( (-c)&mask )); +} + +static inline double +wa_fmax(double a, double b) +{ + double c = fmax(a, b); + if (c==0 && a==b) + return signbit(a) ? b : a; + return c; +} + +static inline double +wa_fmin(double a, double b) +{ + double c = fmin(a, b); + if (c==0 && a==b) + return signbit(a) ? a : b; + return c; +} + +static inline uint32 +clz32(uint32 type) +{ + uint32 num = 0; + if (type == 0) + return 32; + while (!(type & 0x80000000)) { + num++; + type <<= 1; + } + return num; +} + +static inline uint32 +clz64(uint64 type) +{ + uint32 num = 0; + if (type == 0) + return 64; + while (!(type & 0x8000000000000000LL)) { + num++; + type <<= 1; + } + return num; +} + +static inline uint32 +ctz32(uint32 type) +{ + uint32 num = 0; + if (type == 0) + return 32; + while (!(type & 1)) { + num++; + type >>= 1; + } + return num; +} + +static inline uint32 +ctz64(uint64 type) +{ + uint32 num = 0; + if (type == 0) + return 64; + while (!(type & 1)) { + num++; + type >>= 1; + } + return num; +} + +static inline uint32 +popcount32(uint32 u) +{ + uint32 ret = 0; + while (u) { + u = (u & (u - 1)); + ret++; + } + return ret; +} + +static inline uint32 +popcount64(uint64 u) +{ + uint32 ret = 0; + while (u) { + u = (u & (u - 1)); + ret++; + } + return ret; +} + +static uint64 +read_leb(const uint8 *buf, uint32 *p_offset, uint32 maxbits, bool sign) +{ + uint64 result = 0; + uint32 shift = 0; + uint32 bcnt = 0; + uint64 byte; + + while (true) { + byte = buf[*p_offset]; + *p_offset += 1; + result |= ((byte & 0x7f) << shift); + shift += 7; + if ((byte & 0x80) == 0) { + break; + } + bcnt += 1; + } + if (sign && (shift < maxbits) && (byte & 0x40)) { + /* Sign extend */ + result |= - ((uint64)1 << shift); + } + return result; +} + +#define read_leb_uint32(p, p_end, res) do { \ + uint8 _val = *p; \ + if (!(_val & 0x80)) { \ + res = _val; \ + p++; \ + break; \ + } \ + uint32 _off = 0; \ + res = (uint32)read_leb(p, &_off, 32, false); \ + p += _off; \ +} while (0) + +#define GET_LOCAL_INDEX_TYPE_AND_OFFSET() do { \ + uint32 param_count = cur_func->param_count; \ + read_leb_uint32(frame_ip, frame_ip_end, local_idx); \ + bh_assert(local_idx < param_count + cur_func->local_count); \ + local_offset = cur_func->local_offsets[local_idx]; \ + if (local_idx < param_count) \ + local_type = cur_func->param_types[local_idx]; \ + else \ + local_type = cur_func->local_types[local_idx - param_count]; \ + } while (0) + +#define GET_OFFSET() (frame_ip += 2, *(int16 *)(frame_ip - 2)) + +#define SET_OPERAND(type, off, value) \ + (*(type*)(frame_lp + *(int16*)(frame_ip + off))) = value + +#define GET_OPERAND(type, off) (*(type*)(frame_lp + *(int16*)(frame_ip + off))) + +#define PUSH_I32(value) do { \ + *(int32*)(frame_lp + GET_OFFSET()) = value; \ + } while (0) + +#define PUSH_F32(value) do { \ + *(float32*)(frame_lp + GET_OFFSET()) = value; \ + } while (0) + +#define PUSH_I64(value) do { \ + *(int64*)(frame_lp + GET_OFFSET()) = value; \ + } while (0) + +#define PUSH_F64(value) do { \ + *(float64*)(frame_lp + GET_OFFSET()) = value; \ + } while (0) + +#define POP_I32() (*(int32*)(frame_lp + GET_OFFSET())) + +#define POP_F32() (*(float32*)(frame_lp + GET_OFFSET())) + +#define POP_I64() (*(int64*)(frame_lp + GET_OFFSET())) + +#define POP_F64() (*(float64*)(frame_lp + GET_OFFSET())) + +#define SYNC_ALL_TO_FRAME() do { \ + frame->ip = frame_ip; \ + } while (0) + +#define UPDATE_ALL_FROM_FRAME() do { \ + frame_ip = frame->ip; \ + } while (0) + +#define RECOVER_CONTEXT(new_frame) do { \ + frame = (new_frame); \ + cur_func = frame->function; \ + prev_frame = frame->prev_frame; \ + frame_ip = frame->ip; \ + frame_lp = frame->lp; \ + } while (0) + +#if WASM_ENABLE_LABELS_AS_VALUES != 0 +#define GET_OPCODE() opcode = *(frame_ip++); +#else +#define GET_OPCODE() (void)0 +#endif + +#define DEF_OP_EQZ(ctype, src_op_type) do { \ + SET_OPERAND(int32, 2, (GET_OPERAND(ctype, 0) == 0)); \ + frame_ip += 4; \ + } while (0) + +#define DEF_OP_CMP(src_type, src_op_type, cond) do { \ + SET_OPERAND(uint32, 4, GET_OPERAND(src_type, 2) cond \ + GET_OPERAND(src_type, 0)); \ + frame_ip += 6; \ + } while (0) + +#define DEF_OP_BIT_COUNT(src_type, src_op_type, operation) do { \ + SET_OPERAND(src_type, 2, \ + (src_type)operation(GET_OPERAND(src_type, 0))); \ + frame_ip += 4; \ + } while (0) + +#define DEF_OP_NUMERIC(src_type1, src_type2, src_op_type, operation) do { \ + SET_OPERAND(src_type1, 4, (GET_OPERAND(src_type1, 2) \ + operation GET_OPERAND(src_type1, 0))); \ + frame_ip += 6; \ + } while (0) + +#define DEF_OP_REINTERPRET(src_type) do { \ + SET_OPERAND(src_type, 2, GET_OPERAND(src_type, 0)); \ + frame_ip += 4; \ + } while (0) + +#if WASM_CPU_SUPPORTS_UNALIGNED_64BIT_ACCESS != 0 +#define DEF_OP_NUMERIC_64 DEF_OP_NUMERIC +#else +#define DEF_OP_NUMERIC_64(src_type1, src_type2, src_op_type, operation) do {\ + src_type1 val1; \ + src_type2 val2; \ + val1 = (src_type1)GET_##src_op_type##_FROM_ADDR(frame_ip + 2); \ + val2 = (src_type2)GET_##src_op_type##_FROM_ADDR(frame_ip); \ + val1 operation##= val2; \ + PUT_##src_op_type##_TO_ADDR(frame_ip + 4, val1); \ + } while (0) +#endif + +#define DEF_OP_NUMERIC2(src_type1, src_type2, src_op_type, operation) do { \ + SET_OPERAND(src_type1, 4, (GET_OPERAND(src_type1, 2) \ + operation (GET_OPERAND(src_type1, 0) % 32))); \ + frame_ip += 6; \ + } while (0) + +#define DEF_OP_NUMERIC2_64(src_type1, src_type2, src_op_type, operation) do { \ + SET_OPERAND(src_type1, 4, (GET_OPERAND(src_type1, 2) \ + operation (GET_OPERAND(src_type1, 0) % 64))); \ + frame_ip += 6; \ + } while (0) + +#define DEF_OP_MATH(src_type, src_op_type, method) do { \ + SET_OPERAND(src_type, 2, method(GET_OPERAND(src_type, 0))); \ + frame_ip += 4; \ + } while (0) + +#define DEF_OP_TRUNC(dst_type, dst_op_type, src_type, src_op_type, \ + min_cond, max_cond) do { \ + src_type value = GET_OPERAND(src_type, 0); \ + if (isnan(value)) { \ + wasm_set_exception(module, "invalid conversion to integer"); \ + goto got_exception; \ + } \ + else if (value min_cond || value max_cond) { \ + wasm_set_exception(module, "integer overflow"); \ + goto got_exception; \ + } \ + SET_OPERAND(dst_type, 2, value); \ + frame_ip += 4; \ + } while (0) + +#define DEF_OP_CONVERT(dst_type, dst_op_type, \ + src_type, src_op_type) do { \ + dst_type value = (dst_type)(src_type)POP_##src_op_type(); \ + PUSH_##dst_op_type(value); \ + } while (0) + +#define RECOVER_BR_INFO() do { \ + uint16 stack_index, ret_cell_num; \ + stack_index = *(uint16*)frame_ip; \ + frame_ip += sizeof(uint16); \ + ret_cell_num = *(uint8*)frame_ip; \ + frame_ip += sizeof(uint8); \ + if (ret_cell_num == 1) \ + frame_lp[stack_index] = \ + frame_lp[*(int16*)frame_ip]; \ + else if (ret_cell_num == 2) { \ + *(int64*)(frame_lp + stack_index) = \ + *(int64*)(frame_lp + *(int16*)frame_ip);\ + } \ + frame_ip += sizeof(int16); \ + frame_ip = *(uint8**)frame_ip; \ + } while (0) + +static inline int32 +sign_ext_8_32(int8 val) +{ + if (val & 0x80) + return (int32)val | (int32)0xffffff00; + return val; +} + +static inline int32 +sign_ext_16_32(int16 val) +{ + if (val & 0x8000) + return (int32)val | (int32)0xffff0000; + return val; +} + +static inline int64 +sign_ext_8_64(int8 val) +{ + if (val & 0x80) + return (int64)val | (int64)0xffffffffffffff00; + return val; +} + +static inline int64 +sign_ext_16_64(int16 val) +{ + if (val & 0x8000) + return (int64)val | (int64)0xffffffffffff0000; + return val; +} + +static inline int64 +sign_ext_32_64(int32 val) +{ + if (val & (int32)0x80000000) + return (int64)val | (int64)0xffffffff00000000; + return val; +} + +static inline void +word_copy(uint32 *dest, uint32 *src, unsigned num) +{ + for (; num > 0; num--) + *dest++ = *src++; +} + +static inline WASMInterpFrame* +ALLOC_FRAME(WASMExecEnv *exec_env, uint32 size, WASMInterpFrame *prev_frame) +{ + WASMInterpFrame *frame = wasm_exec_env_alloc_wasm_frame(exec_env, size); + + if (frame) + frame->prev_frame = prev_frame; + else { + wasm_set_exception((WASMModuleInstance*)exec_env->module_inst, + "WASM interp failed: stack overflow."); + } + + return frame; +} + +static inline void +FREE_FRAME(WASMExecEnv *exec_env, WASMInterpFrame *frame) +{ + wasm_exec_env_free_wasm_frame(exec_env, frame); +} + +static void +wasm_interp_call_func_native(WASMModuleInstance *module_inst, + WASMExecEnv *exec_env, + WASMFunctionInstance *cur_func, + WASMInterpFrame *prev_frame) +{ + unsigned local_cell_num = 2; + WASMInterpFrame *frame; + uint32 argv_ret[2]; + bool ret; + + if (!(frame = ALLOC_FRAME(exec_env, + wasm_interp_interp_frame_size(local_cell_num), + prev_frame))) + return; + + frame->function = cur_func; + frame->ip = NULL; + frame->lp = frame->operand; + + wasm_exec_env_set_cur_frame(exec_env, frame); + + if (!cur_func->u.func_import->func_ptr_linked) { + char buf[128]; + snprintf(buf, + sizeof(buf), "fail to call unlinked import function (%s, %s)", + cur_func->u.func_import->module_name, + cur_func->u.func_import->field_name); + wasm_set_exception((WASMModuleInstance*)module_inst, buf); + return; + } + + ret = wasm_runtime_invoke_native(exec_env, cur_func->u.func_import->func_ptr_linked, + cur_func->u.func_import->func_type, + cur_func->u.func_import->signature, + frame->lp, cur_func->param_cell_num, argv_ret); + + if (!ret) + return; + + if (cur_func->ret_cell_num == 1) { + prev_frame->lp[prev_frame->ret_offset] = argv_ret[0]; + } + else if (cur_func->ret_cell_num == 2) { + *(int64*)(prev_frame->lp + prev_frame->ret_offset) = *(int64*)argv_ret; + } + + FREE_FRAME(exec_env, frame); + wasm_exec_env_set_cur_frame(exec_env, prev_frame); +} + +#if WASM_ENABLE_LABELS_AS_VALUES != 0 + +//#define HANDLE_OP(opcode) HANDLE_##opcode:printf(#opcode"\n");h_##opcode +#define HANDLE_OP(opcode) HANDLE_##opcode +#if WASM_ENABLE_FAST_INTERP == 0 +#define FETCH_OPCODE_AND_DISPATCH() goto *handle_table[*frame_ip++] +#else +#if WASM_ENABLE_ABS_LABEL_ADDR != 0 +#define FETCH_OPCODE_AND_DISPATCH() do { \ + const void *p_label_addr = *(void**)frame_ip; \ + frame_ip += sizeof(void*); \ + goto *p_label_addr; \ + } while (0) +#else +#define FETCH_OPCODE_AND_DISPATCH() do { \ + const void *p_label_addr = label_base \ + + *(int16*)frame_ip; \ + frame_ip += sizeof(int16); \ + goto *p_label_addr; \ + } while (0) +#endif +#endif +#define HANDLE_OP_END() FETCH_OPCODE_AND_DISPATCH() + +#else /* else of WASM_ENABLE_LABELS_AS_VALUES */ + +#define HANDLE_OP(opcode) case opcode +#define HANDLE_OP_END() continue + +#endif /* end of WASM_ENABLE_LABELS_AS_VALUES */ + +#if WASM_ENABLE_FAST_INTERP != 0 +static void *global_handle_table[WASM_INSTRUCTION_NUM] = { 0 }; +#endif + +static void +wasm_interp_call_func_bytecode(WASMModuleInstance *module, + WASMExecEnv *exec_env, + WASMFunctionInstance *cur_func, + WASMInterpFrame *prev_frame) +{ + WASMMemoryInstance *memory = module->default_memory; + uint32 num_bytes_per_page = memory ? memory->num_bytes_per_page : 0; + uint32 memory_data_size = memory ? num_bytes_per_page * memory->cur_page_count : 0; + uint32 heap_data_size = memory ? (uint32)(memory->heap_data_end - memory->heap_data) : 0; + uint8 *global_data = memory ? memory->global_data : NULL; + WASMTableInstance *table = module->default_table; + WASMGlobalInstance *globals = module->globals; + uint8 opcode_IMPDEP = WASM_OP_IMPDEP; + WASMInterpFrame *frame = NULL; + /* Points to this special opcode so as to jump to the call_method_from_entry. */ + register uint8 *frame_ip = &opcode_IMPDEP; /* cache of frame->ip */ + register uint32 *frame_lp = NULL; /* cache of frame->lp */ +#if WASM_ENABLE_ABS_LABEL_ADDR == 0 + register uint8 *label_base = &&HANDLE_WASM_OP_UNREACHABLE; /* cache of label base addr */ +#endif + WASMGlobalInstance *global; + uint8 *frame_ip_end; + uint8 opcode; + uint32 cond, count, fidx, tidx, frame_size = 0; + uint64 all_cell_num = 0; + int16 addr1, addr2, addr_ret; + int32 didx, val; + uint8 *maddr = NULL; + uint32 local_idx, local_offset, global_idx; + uint8 local_type, *global_addr; + +#if WASM_ENABLE_LABELS_AS_VALUES != 0 + #define HANDLE_OPCODE(op) &&HANDLE_##op + DEFINE_GOTO_TABLE (handle_table); + #undef HANDLE_OPCODE +#if WASM_ENABLE_FAST_INTERP != 0 + if (exec_env == NULL) { + bh_memcpy_s(global_handle_table, sizeof(void*) * WASM_INSTRUCTION_NUM, + handle_table, sizeof(void*) * WASM_INSTRUCTION_NUM); + return; + } +#endif +#endif + + /* Size of memory load. + This starts with the first memory load operator at opcode 0x28 */ + uint32 LOAD_SIZE[] = { + 4, 8, 4, 8, 1, 1, 2, 2, 1, 1, 2, 2, 4, 4, /* loads */ + 4, 8, 4, 8, 1, 2, 1, 2, 4 }; /* stores */ + +#if WASM_ENABLE_LABELS_AS_VALUES == 0 + while (frame_ip < frame_ip_end) { + opcode = *frame_ip++; + switch (opcode) { +#else + goto *handle_table[WASM_OP_IMPDEP]; +#endif + /* control instructions */ + HANDLE_OP (WASM_OP_UNREACHABLE): + wasm_set_exception(module, "unreachable"); + goto got_exception; + + HANDLE_OP (WASM_OP_IF): + cond = (uint32)POP_I32(); + + if (cond == 0) { + if (*(uint8**)frame_ip == NULL) { + frame_ip = *(uint8**)(frame_ip + sizeof(uint8*)); + } + else { + frame_ip = *(uint8**)(frame_ip); + } + } + else { + frame_ip += sizeof(uint8*) * 2; + } + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_ELSE): + frame_ip = *(uint8**)(frame_ip); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_BR): + RECOVER_BR_INFO(); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_BR_IF): + cond = frame_lp[GET_OFFSET()]; + + if (cond) + RECOVER_BR_INFO(); + else { + frame_ip += (2 + 1 + 2 + sizeof(uint8*)); + } + + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_BR_TABLE): + count = GET_OPERAND(uint32, 0); + didx = GET_OPERAND(uint32, 2); + frame_ip += 4; + + if (!(didx >= 0 && (uint32)didx < count)) + didx = count; + + frame_ip += (didx * ((2 + 1 + 2 + sizeof(uint8*)))); + RECOVER_BR_INFO(); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_RETURN): + if (cur_func->ret_cell_num == 2) { + *((uint64 *)(prev_frame->lp + prev_frame->ret_offset)) = + GET_OPERAND(uint64, 0); + } else if (cur_func->ret_cell_num == 1) { + prev_frame->lp[prev_frame->ret_offset] = GET_OPERAND(int32, 0);; + } + goto return_func; + + HANDLE_OP (WASM_OP_CALL): + fidx = frame_lp[GET_OFFSET()]; + bh_assert(fidx < module->function_count); + cur_func = module->functions + fidx; + goto call_func_from_interp; + + HANDLE_OP (WASM_OP_CALL_INDIRECT): + { + WASMType *cur_type, *cur_func_type; + + tidx = GET_OPERAND(int32, 0); + val = GET_OPERAND(int32, 2); + frame_ip += 4; + + if (tidx >= module->module->type_count) { + wasm_set_exception(module, "type index is overflow"); + goto got_exception; + } + cur_type = module->module->types[tidx]; + + if (val < 0 || val >= (int32)table->cur_size) { + wasm_set_exception(module, "undefined element"); + goto got_exception; + } + + fidx = ((uint32*)table->base_addr)[val]; + if (fidx == (uint32)-1) { + wasm_set_exception(module, "uninitialized element"); + goto got_exception; + } + + cur_func = module->functions + fidx; + + if (cur_func->is_import_func) + cur_func_type = cur_func->u.func_import->func_type; + else + cur_func_type = cur_func->u.func->func_type; + if (!wasm_type_equal(cur_type, cur_func_type)) { + wasm_set_exception(module, "indirect call type mismatch"); + goto got_exception; + } + goto call_func_from_interp; + } + + /* parametric instructions */ + HANDLE_OP (WASM_OP_SELECT): + { + cond = frame_lp[GET_OFFSET()]; + addr1 = GET_OFFSET(); + addr2 = GET_OFFSET(); + addr_ret = GET_OFFSET(); + + if (!cond) + frame_lp[addr_ret] = frame_lp[addr1]; + else + frame_lp[addr_ret] = frame_lp[addr2]; + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_SELECT_64): + { + cond = frame_lp[GET_OFFSET()]; + addr1 = GET_OFFSET(); + addr2 = GET_OFFSET(); + addr_ret = GET_OFFSET(); + + if (!cond) + *(int64*)(frame_lp + addr_ret) = *(int64*)(frame_lp + addr1); + else + *(int64*)(frame_lp + addr_ret) = *(int64*)(frame_lp + addr2); + HANDLE_OP_END (); + } + + /* variable instructions */ + HANDLE_OP (EXT_OP_SET_LOCAL_FAST): + HANDLE_OP (EXT_OP_TEE_LOCAL_FAST): + { + local_offset = *frame_ip++; + *(int32*)(frame_lp + local_offset) = GET_OPERAND(uint32, 0); + frame_ip += 2; + HANDLE_OP_END (); + } + + HANDLE_OP (EXT_OP_SET_LOCAL_FAST_I64): + HANDLE_OP (EXT_OP_TEE_LOCAL_FAST_I64): + { + local_offset = *frame_ip++; + PUT_I64_TO_ADDR((uint32*)(frame_lp + local_offset), GET_OPERAND(uint64, 0)); + frame_ip += 2; + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_GET_GLOBAL): + { + global_idx = frame_lp[GET_OFFSET()]; + addr_ret = GET_OFFSET(); + + bh_assert(global_idx < module->global_count); + global = globals + global_idx; + global_addr = global_data + global->data_offset; + + switch (global->type) { + case VALUE_TYPE_I32: + case VALUE_TYPE_F32: + frame_lp[addr_ret] = *(uint32*)global_addr; + break; + case VALUE_TYPE_I64: + case VALUE_TYPE_F64: + *(uint64 *)(frame_lp + addr_ret) = GET_I64_FROM_ADDR((uint32*)global_addr); + break; + default: + wasm_set_exception(module, "invalid global type"); + goto got_exception; + } + + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_SET_GLOBAL): + { + global_idx = frame_lp[GET_OFFSET()]; + addr1 = GET_OFFSET(); + + bh_assert(global_idx < module->global_count); + global = globals + global_idx; + global_addr = global_data + global->data_offset; + + switch (global->type) { + case VALUE_TYPE_I32: + case VALUE_TYPE_F32: + *(int32*)global_addr = frame_lp[addr1]; + break; + case VALUE_TYPE_I64: + case VALUE_TYPE_F64: + PUT_I64_TO_ADDR((uint32*)global_addr, *(int64 *)(frame_lp + addr1)); + break; + default: + wasm_set_exception(module, "invalid global type"); + goto got_exception; + } + + HANDLE_OP_END (); + } + + /* memory load instructions */ + HANDLE_OP (WASM_OP_I32_LOAD): + { + uint32 offset, addr; + offset = GET_OPERAND(uint32, 1); + addr = GET_OPERAND(uint32, 3); + frame_ip += 5; + addr_ret = GET_OFFSET(); + CHECK_MEMORY_OVERFLOW_FAST(4); + frame_lp[addr_ret] = LOAD_I32(maddr); + HANDLE_OP_END (); + } + HANDLE_OP (WASM_OP_I64_LOAD): + HANDLE_OP (WASM_OP_F32_LOAD): + HANDLE_OP (WASM_OP_F64_LOAD): + HANDLE_OP (WASM_OP_I32_LOAD8_S): + HANDLE_OP (WASM_OP_I32_LOAD8_U): + HANDLE_OP (WASM_OP_I32_LOAD16_S): + HANDLE_OP (WASM_OP_I32_LOAD16_U): + HANDLE_OP (WASM_OP_I64_LOAD8_S): + HANDLE_OP (WASM_OP_I64_LOAD8_U): + HANDLE_OP (WASM_OP_I64_LOAD16_S): + HANDLE_OP (WASM_OP_I64_LOAD16_U): + HANDLE_OP (WASM_OP_I64_LOAD32_S): + HANDLE_OP (WASM_OP_I64_LOAD32_U): + { + uint32 offset, flags, addr; + GET_OPCODE(); + offset = GET_OPERAND(uint32, 0); + addr = GET_OPERAND(int32, 2); + frame_ip += 4; + addr_ret = GET_OFFSET(); + CHECK_MEMORY_OVERFLOW(); +#if WASM_ENABLE_LABELS_AS_VALUES != 0 + static const void *handle_load_table[] = { + &&HANDLE_LOAD_WASM_OP_I32_LOAD, + &&HANDLE_LOAD_WASM_OP_I64_LOAD, + &&HANDLE_LOAD_WASM_OP_F32_LOAD, + &&HANDLE_LOAD_WASM_OP_F64_LOAD, + &&HANDLE_LOAD_WASM_OP_I32_LOAD8_S, + &&HANDLE_LOAD_WASM_OP_I32_LOAD8_U, + &&HANDLE_LOAD_WASM_OP_I32_LOAD16_S, + &&HANDLE_LOAD_WASM_OP_I32_LOAD16_U, + &&HANDLE_LOAD_WASM_OP_I64_LOAD8_S, + &&HANDLE_LOAD_WASM_OP_I64_LOAD8_U, + &&HANDLE_LOAD_WASM_OP_I64_LOAD16_S, + &&HANDLE_LOAD_WASM_OP_I64_LOAD16_U, + &&HANDLE_LOAD_WASM_OP_I64_LOAD32_S, + &&HANDLE_LOAD_WASM_OP_I64_LOAD32_U + }; + #define HANDLE_OP_LOAD(opcode) HANDLE_LOAD_##opcode + goto *handle_load_table[opcode - WASM_OP_I32_LOAD]; +#else + #define HANDLE_OP_LOAD(opcode) case opcode + switch (opcode) +#endif + { + HANDLE_OP_LOAD(WASM_OP_I32_LOAD): + frame_lp[addr_ret] = LOAD_I32(maddr); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I64_LOAD): + *(int64 *)(frame_lp + addr_ret) = (LOAD_I64(maddr)); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_F32_LOAD): + *(float32 *)(frame_lp + addr_ret) = (LOAD_F32(maddr)); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_F64_LOAD): + *(float64 *)(frame_lp + addr_ret) = (LOAD_F64(maddr)); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I32_LOAD8_S): + frame_lp[addr_ret] = sign_ext_8_32(*(int8*)maddr); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I32_LOAD8_U): + frame_lp[addr_ret] = (uint32)(*(uint8*)maddr); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I32_LOAD16_S): + frame_lp[addr_ret] = sign_ext_16_32(LOAD_I16(maddr)); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I32_LOAD16_U): + frame_lp[addr_ret] = (uint32)(LOAD_U16(maddr)); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I64_LOAD8_S): + *(int64 *)(frame_lp + addr_ret) = sign_ext_8_64(*(int8*)maddr); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I64_LOAD8_U): + *(int64 *)(frame_lp + addr_ret) = (uint64)(*(uint8*)maddr); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I64_LOAD16_S): + *(int64 *)(frame_lp + addr_ret) = sign_ext_16_64(LOAD_I16(maddr)); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I64_LOAD16_U): + *(int64 *)(frame_lp + addr_ret) = (uint64)(LOAD_U16(maddr)); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I64_LOAD32_S): + *(int64 *)(frame_lp + addr_ret) = sign_ext_32_64(LOAD_I32(maddr)); + HANDLE_OP_END(); + HANDLE_OP_LOAD(WASM_OP_I64_LOAD32_U): + *(int64 *)(frame_lp + addr_ret) = (uint64)(LOAD_U32(maddr)); + HANDLE_OP_END(); + } + (void)flags; + HANDLE_OP_END (); + } + + /* memory store instructions */ + HANDLE_OP (WASM_OP_F32_STORE): + { + uint32 offset, addr; + GET_OPCODE(); + offset = GET_OPERAND(uint32, 0); + val = GET_OPERAND(int32, 2); + addr = GET_OPERAND(int32, 4); + frame_ip += 6; + CHECK_MEMORY_OVERFLOW_FAST(4); + STORE_U32(maddr, val); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_F64_STORE): + { + uint32 offset, addr; + int32 val_offset; + GET_OPCODE(); + offset = GET_OPERAND(uint32, 0); + frame_ip += 2; + val_offset = GET_OFFSET(); + addr2 = GET_OFFSET(); + addr = (uint32)frame_lp[addr2]; + CHECK_MEMORY_OVERFLOW_FAST(8); + STORE_U32(maddr, frame_lp[val_offset]); + STORE_U32(maddr + 4, frame_lp[val_offset + 1]); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I32_STORE): + { + uint32 offset, addr; + uint32 sval; + offset = GET_OPERAND(uint32, 1); + sval = GET_OPERAND(uint32, 3); + addr = GET_OPERAND(uint32, 5); + frame_ip += 7; + CHECK_MEMORY_OVERFLOW_FAST(4); + STORE_U32(maddr, sval); + HANDLE_OP_END (); + } + HANDLE_OP (WASM_OP_I32_STORE8): + HANDLE_OP (WASM_OP_I32_STORE16): + { + uint32 offset, addr; + uint32 sval; + GET_OPCODE(); + offset = GET_OPERAND(uint32, 0); + sval = GET_OPERAND(uint32, 2); + addr = GET_OPERAND(uint32, 4); + frame_ip += 6; + CHECK_MEMORY_OVERFLOW(); + switch (opcode) { + case WASM_OP_I32_STORE8: + *(uint8*)maddr = (uint8)sval; + break; + case WASM_OP_I32_STORE16: + STORE_U16(maddr, (uint16)sval); + break; + } + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I64_STORE): + HANDLE_OP (WASM_OP_I64_STORE8): + HANDLE_OP (WASM_OP_I64_STORE16): + HANDLE_OP (WASM_OP_I64_STORE32): + { + uint32 offset, addr; + uint64 sval; + GET_OPCODE(); + offset = GET_OPERAND(uint32, 0); + sval = GET_OPERAND(uint64, 2); + addr = GET_OPERAND(uint32, 4); + frame_ip += 6; + CHECK_MEMORY_OVERFLOW(); + switch (opcode) { + case WASM_OP_I64_STORE: + STORE_I64(maddr, sval); + break; + case WASM_OP_I64_STORE8: + *(uint8*)maddr = (uint8)sval; + break; + case WASM_OP_I64_STORE16: + STORE_U16(maddr, (uint16)sval); + break; + case WASM_OP_I64_STORE32: + STORE_U32(maddr, (uint32)sval); + break; + } + HANDLE_OP_END (); + } + + /* memory size and memory grow instructions */ + HANDLE_OP (WASM_OP_MEMORY_SIZE): + { + uint32 reserved; + addr_ret = GET_OFFSET(); + frame_lp[addr_ret] = memory->cur_page_count; + (void)reserved; + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_MEMORY_GROW): + { + uint32 reserved, delta, prev_page_count = memory->cur_page_count; + + addr1 = GET_OFFSET(); + addr_ret = GET_OFFSET(); + delta = (uint32)frame_lp[addr1]; + + if (!wasm_enlarge_memory(module, delta)) { + /* fail to memory.grow, return -1 */ + frame_lp[addr_ret] = -1; + if (wasm_get_exception(module)) { + bh_printf("%s\n", wasm_get_exception(module)); + wasm_set_exception(module, NULL); + } + } + else { + /* success, return previous page count */ + frame_lp[addr_ret] = prev_page_count; + /* update the memory instance ptr */ + memory = module->default_memory; + memory_data_size = num_bytes_per_page * memory->cur_page_count; + global_data = memory->global_data; + } + + (void)reserved; + HANDLE_OP_END (); + } + + /* comparison instructions of i32 */ + HANDLE_OP (WASM_OP_I32_EQZ): + DEF_OP_EQZ(int32, I32); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_EQ): + DEF_OP_CMP(uint32, I32, ==); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_NE): + DEF_OP_CMP(uint32, I32, !=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_LT_S): + DEF_OP_CMP(int32, I32, <); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_LT_U): + DEF_OP_CMP(uint32, I32, <); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_GT_S): + DEF_OP_CMP(int32, I32, >); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_GT_U): + DEF_OP_CMP(uint32, I32, >); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_LE_S): + DEF_OP_CMP(int32, I32, <=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_LE_U): + DEF_OP_CMP(uint32, I32, <=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_GE_S): + DEF_OP_CMP(int32, I32, >=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_GE_U): + DEF_OP_CMP(uint32, I32, >=); + HANDLE_OP_END (); + + /* comparison instructions of i64 */ + HANDLE_OP (WASM_OP_I64_EQZ): + DEF_OP_EQZ(int64, I64); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_EQ): + DEF_OP_CMP(uint64, I64, ==); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_NE): + DEF_OP_CMP(uint64, I64, !=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_LT_S): + DEF_OP_CMP(int64, I64, <); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_LT_U): + DEF_OP_CMP(uint64, I64, <); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_GT_S): + DEF_OP_CMP(int64, I64, >); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_GT_U): + DEF_OP_CMP(uint64, I64, >); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_LE_S): + DEF_OP_CMP(int64, I64, <=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_LE_U): + DEF_OP_CMP(uint64, I64, <=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_GE_S): + DEF_OP_CMP(int64, I64, >=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_GE_U): + DEF_OP_CMP(uint64, I64, >=); + HANDLE_OP_END (); + + /* comparison instructions of f32 */ + HANDLE_OP (WASM_OP_F32_EQ): + DEF_OP_CMP(float32, F32, ==); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_NE): + DEF_OP_CMP(float32, F32, !=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_LT): + DEF_OP_CMP(float32, F32, <); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_GT): + DEF_OP_CMP(float32, F32, >); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_LE): + DEF_OP_CMP(float32, F32, <=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_GE): + DEF_OP_CMP(float32, F32, >=); + HANDLE_OP_END (); + + /* comparison instructions of f64 */ + HANDLE_OP (WASM_OP_F64_EQ): + DEF_OP_CMP(float64, F64, ==); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_NE): + DEF_OP_CMP(float64, F64, !=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_LT): + DEF_OP_CMP(float64, F64, <); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_GT): + DEF_OP_CMP(float64, F64, >); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_LE): + DEF_OP_CMP(float64, F64, <=); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_GE): + DEF_OP_CMP(float64, F64, >=); + HANDLE_OP_END (); + + /* numberic instructions of i32 */ + HANDLE_OP (WASM_OP_I32_CLZ): + DEF_OP_BIT_COUNT(uint32, I32, clz32); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_CTZ): + DEF_OP_BIT_COUNT(uint32, I32, ctz32); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_POPCNT): + DEF_OP_BIT_COUNT(uint32, I32, popcount32); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_ADD): + DEF_OP_NUMERIC(uint32, uint32, I32, +); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_SUB): + DEF_OP_NUMERIC(uint32, uint32, I32, -); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_MUL): + DEF_OP_NUMERIC(uint32, uint32, I32, *); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_DIV_S): + { + int32 a, b; + + b = frame_lp[GET_OFFSET()]; + a = frame_lp[GET_OFFSET()]; + addr_ret = GET_OFFSET(); + if (a == (int32)0x80000000 && b == -1) { + wasm_set_exception(module, "integer overflow"); + goto got_exception; + } + if (b == 0) { + wasm_set_exception(module, "integer divide by zero"); + goto got_exception; + } + frame_lp[addr_ret] = (a / b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I32_DIV_U): + { + uint32 a, b; + + addr1 = GET_OFFSET(); + addr2 = GET_OFFSET(); + addr_ret = GET_OFFSET(); + + b = (uint32)frame_lp[addr1]; + a = (uint32)frame_lp[addr2]; + if (b == 0) { + wasm_set_exception(module, "integer divide by zero"); + goto got_exception; + } + frame_lp[addr_ret] = (a / b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I32_REM_S): + { + int32 a, b; + + addr1 = GET_OFFSET(); + addr2 = GET_OFFSET(); + addr_ret = GET_OFFSET(); + + b = frame_lp[addr1]; + a = frame_lp[addr2]; + if (a == (int32)0x80000000 && b == -1) { + frame_lp[addr_ret] = 0; + HANDLE_OP_END (); + } + if (b == 0) { + wasm_set_exception(module, "integer divide by zero"); + goto got_exception; + } + frame_lp[addr_ret] = (a % b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I32_REM_U): + { + uint32 a, b; + + addr1 = GET_OFFSET(); + addr2 = GET_OFFSET(); + addr_ret = GET_OFFSET(); + + b = (uint32)frame_lp[addr1]; + a = (uint32)frame_lp[addr2]; + if (b == 0) { + wasm_set_exception(module, "integer divide by zero"); + goto got_exception; + } + frame_lp[addr_ret] = (a % b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I32_AND): + DEF_OP_NUMERIC(uint32, uint32, I32, &); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_OR): + DEF_OP_NUMERIC(uint32, uint32, I32, |); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_XOR): + DEF_OP_NUMERIC(uint32, uint32, I32, ^); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_SHL): + { +#if defined(BUILD_TARGET_X86_64) || defined(BUILD_TARGET_X86_32) + DEF_OP_NUMERIC(uint32, uint32, I32, <<); +#else + DEF_OP_NUMERIC2(uint32, uint32, I32, <<); +#endif + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I32_SHR_S): + { +#if defined(BUILD_TARGET_X86_64) || defined(BUILD_TARGET_X86_32) + DEF_OP_NUMERIC(int32, uint32, I32, >>); +#else + DEF_OP_NUMERIC2(int32, uint32, I32, >>); +#endif + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I32_SHR_U): + { +#if defined(BUILD_TARGET_X86_64) || defined(BUILD_TARGET_X86_32) + DEF_OP_NUMERIC(uint32, uint32, I32, >>); +#else + DEF_OP_NUMERIC2(uint32, uint32, I32, >>); +#endif + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I32_ROTL): + { + uint32 a, b; + + b = (uint32)frame_lp[GET_OFFSET()]; + a = (uint32)frame_lp[GET_OFFSET()]; + frame_lp[GET_OFFSET()] = rotl32(a, b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I32_ROTR): + { + uint32 a, b; + + b = (uint32)frame_lp[GET_OFFSET()]; + a = (uint32)frame_lp[GET_OFFSET()]; + frame_lp[GET_OFFSET()] = rotr32(a, b); + HANDLE_OP_END (); + } + + /* numberic instructions of i64 */ + HANDLE_OP (WASM_OP_I64_CLZ): + DEF_OP_BIT_COUNT(uint64, I64, clz64); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_CTZ): + DEF_OP_BIT_COUNT(uint64, I64, ctz64); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_POPCNT): + DEF_OP_BIT_COUNT(uint64, I64, popcount64); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_ADD): + DEF_OP_NUMERIC_64(uint64, uint64, I64, +); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_SUB): + DEF_OP_NUMERIC_64(uint64, uint64, I64, -); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_MUL): + DEF_OP_NUMERIC_64(uint64, uint64, I64, *); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_DIV_S): + { + int64 a, b; + + b = *(int64*)(frame_lp + GET_OFFSET()); + a = *(int64*)(frame_lp + GET_OFFSET()); + if (a == (int64)0x8000000000000000LL && b == -1) { + wasm_set_exception(module, "integer overflow"); + goto got_exception; + } + if (b == 0) { + wasm_set_exception(module, "integer divide by zero"); + goto got_exception; + } + *(int64*)(frame_lp + GET_OFFSET()) = (a / b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I64_DIV_U): + { + uint64 a, b; + + b = *(uint64*)(frame_lp + GET_OFFSET()); + a = *(uint64*)(frame_lp + GET_OFFSET()); + if (b == 0) { + wasm_set_exception(module, "integer divide by zero"); + goto got_exception; + } + *(uint64*)(frame_lp + GET_OFFSET()) = (a / b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I64_REM_S): + { + int64 a, b; + + b = *(int64*)(frame_lp + GET_OFFSET()); + a = *(int64*)(frame_lp + GET_OFFSET()); + if (a == (int64)0x8000000000000000LL && b == -1) { + *(int64*)(frame_lp + GET_OFFSET()) = 0; + HANDLE_OP_END (); + } + if (b == 0) { + wasm_set_exception(module, "integer divide by zero"); + goto got_exception; + } + *(int64*)(frame_lp + GET_OFFSET()) = (a % b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I64_REM_U): + { + uint64 a, b; + + b = *(uint64*)(frame_lp + GET_OFFSET()); + a = *(uint64*)(frame_lp + GET_OFFSET()); + if (b == 0) { + wasm_set_exception(module, "integer divide by zero"); + goto got_exception; + } + *(uint64*)(frame_lp + GET_OFFSET()) = (a % b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I64_AND): + DEF_OP_NUMERIC_64(uint64, uint64, I64, &); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_OR): + DEF_OP_NUMERIC_64(uint64, uint64, I64, |); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_XOR): + DEF_OP_NUMERIC_64(uint64, uint64, I64, ^); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_SHL): + { +#if defined(BUILD_TARGET_X86_64) || defined(BUILD_TARGET_X86_32) + DEF_OP_NUMERIC_64(uint64, uint64, I64, <<); +#else + DEF_OP_NUMERIC2_64(uint64, uint64, I64, <<); +#endif + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I64_SHR_S): + { +#if defined(BUILD_TARGET_X86_64) || defined(BUILD_TARGET_X86_32) + DEF_OP_NUMERIC_64(int64, uint64, I64, >>); +#else + DEF_OP_NUMERIC2_64(int64, uint64, I64, >>); +#endif + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I64_SHR_U): + { +#if defined(BUILD_TARGET_X86_64) || defined(BUILD_TARGET_X86_32) + DEF_OP_NUMERIC_64(uint64, uint64, I64, >>); +#else + DEF_OP_NUMERIC2_64(uint64, uint64, I64, >>); +#endif + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I64_ROTL): + { + uint64 a, b; + + b = *(int64*)(frame_lp + GET_OFFSET()); + a = *(int64*)(frame_lp + GET_OFFSET()); + *(int64*)(frame_lp + GET_OFFSET()) = rotl64(a, b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I64_ROTR): + { + uint64 a, b; + + b = *(uint64*)(frame_lp + GET_OFFSET()); + a = *(uint64*)(frame_lp + GET_OFFSET()); + *(uint64*)(frame_lp + GET_OFFSET()) = rotr64(a, b); + HANDLE_OP_END (); + } + + /* numberic instructions of f32 */ + HANDLE_OP (WASM_OP_F32_ABS): + DEF_OP_MATH(float32, F32, fabs); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_NEG): + { + int32 i32 = (int32)frame_lp[GET_OFFSET()]; + addr_ret = GET_OFFSET(); + int32 sign_bit = i32 & (1 << 31); + if (sign_bit) + frame_lp[addr_ret] = i32 & ~(1 << 31); + else + frame_lp[addr_ret] = (uint32)(i32 | (1 << 31)); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_F32_CEIL): + DEF_OP_MATH(float32, F32, ceil); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_FLOOR): + DEF_OP_MATH(float32, F32, floor); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_TRUNC): + DEF_OP_MATH(float32, F32, trunc); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_NEAREST): + DEF_OP_MATH(float32, F32, rint); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_SQRT): + DEF_OP_MATH(float32, F32, sqrt); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_ADD): + DEF_OP_NUMERIC(float32, float32, F32, +); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_SUB): + DEF_OP_NUMERIC(float32, float32, F32, -); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_MUL): + DEF_OP_NUMERIC(float32, float32, F32, *); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_DIV): + DEF_OP_NUMERIC(float32, float32, F32, /); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_MIN): + { + float32 a, b; + + b = *(float32*)(frame_lp + GET_OFFSET()); + a = *(float32*)(frame_lp + GET_OFFSET()); + + if (isnan(a)) + *(float32*)(frame_lp + GET_OFFSET()) = a; + else if (isnan(b)) + *(float32*)(frame_lp + GET_OFFSET()) = b; + else + *(float32*)(frame_lp + GET_OFFSET()) = wa_fmin(a, b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_F32_MAX): + { + float32 a, b; + + b = *(float32*)(frame_lp + GET_OFFSET()); + a = *(float32*)(frame_lp + GET_OFFSET()); + + if (isnan(a)) + *(float32*)(frame_lp + GET_OFFSET()) = a; + else if (isnan(b)) + *(float32*)(frame_lp + GET_OFFSET()) = b; + else + *(float32*)(frame_lp + GET_OFFSET()) = wa_fmax(a, b); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_F32_COPYSIGN): + { + float32 a, b; + + b = *(float32*)(frame_lp + GET_OFFSET()); + a = *(float32*)(frame_lp + GET_OFFSET()); + *(float32*)(frame_lp + GET_OFFSET()) = (signbit(b) ? -fabs(a) : fabs(a)); + HANDLE_OP_END (); + } + + /* numberic instructions of f64 */ + HANDLE_OP (WASM_OP_F64_ABS): + DEF_OP_MATH(float64, F64, fabs); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_NEG): + { + int64 i64 = *(int64*)(frame_lp + GET_OFFSET()); + int64 sign_bit = i64 & (((int64)1) << 63); + if (sign_bit) + *(int64*)(frame_lp + GET_OFFSET()) = (uint64)i64 & ~(((uint64)1) << 63); + else + *(int64*)(frame_lp + GET_OFFSET()) = (uint64)i64 | (((uint64)1) << 63); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_F64_CEIL): + DEF_OP_MATH(float64, F64, ceil); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_FLOOR): + DEF_OP_MATH(float64, F64, floor); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_TRUNC): + DEF_OP_MATH(float64, F64, trunc); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_NEAREST): + DEF_OP_MATH(float64, F64, rint); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_SQRT): + DEF_OP_MATH(float64, F64, sqrt); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_ADD): + DEF_OP_NUMERIC_64(float64, float64, F64, +); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_SUB): + DEF_OP_NUMERIC_64(float64, float64, F64, -); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_MUL): + DEF_OP_NUMERIC_64(float64, float64, F64, *); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_DIV): + DEF_OP_NUMERIC_64(float64, float64, F64, /); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_MIN): + { + float64 a, b; + + b = POP_F64(); + a = POP_F64(); + + if (isnan(a)) + PUSH_F64(a); + else if (isnan(b)) + PUSH_F64(b); + else + PUSH_F64(wa_fmin(a, b)); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_F64_MAX): + { + float64 a, b; + + b = POP_F64(); + a = POP_F64(); + + if (isnan(a)) + PUSH_F64(a); + else if (isnan(b)) + PUSH_F64(b); + else + PUSH_F64(wa_fmax(a, b)); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_F64_COPYSIGN): + { + float64 a, b; + + b = POP_F64(); + a = POP_F64(); + PUSH_F64(signbit(b) ? -fabs(a) : fabs(a)); + HANDLE_OP_END (); + } + + /* conversions of i32 */ + HANDLE_OP (WASM_OP_I32_WRAP_I64): + { + int32 value = (int32)(POP_I64() & 0xFFFFFFFFLL); + PUSH_I32(value); + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_I32_TRUNC_S_F32): + /* We don't use INT32_MIN/INT32_MAX/UINT32_MIN/UINT32_MAX, + since float/double values of ieee754 cannot precisely represent + all int32/uint32/int64/uint64 values, e.g.: + UINT32_MAX is 4294967295, but (float32)4294967295 is 4294967296.0f, + but not 4294967295.0f. */ + DEF_OP_TRUNC(int32, I32, float32, F32, <= -2147483904.0f, + >= 2147483648.0f); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_TRUNC_U_F32): + DEF_OP_TRUNC(uint32, I32, float32, F32, <= -1.0f, + >= 4294967296.0f); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_TRUNC_S_F64): + DEF_OP_TRUNC(int32, I32, float64, F64, <= -2147483649.0, + >= 2147483648.0); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I32_TRUNC_U_F64): + DEF_OP_TRUNC(uint32, I32, float64, F64, <= -1.0 , + >= 4294967296.0); + HANDLE_OP_END (); + + /* conversions of i64 */ + HANDLE_OP (WASM_OP_I64_EXTEND_S_I32): + DEF_OP_CONVERT(int64, I64, int32, I32); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_EXTEND_U_I32): + DEF_OP_CONVERT(int64, I64, uint32, I32); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_TRUNC_S_F32): + DEF_OP_TRUNC(int64, I64, float32, F32, <= -9223373136366403584.0f, + >= 9223372036854775808.0f); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_TRUNC_U_F32): + DEF_OP_TRUNC(uint64, I64, float32, F32, <= -1.0f, + >= 18446744073709551616.0f); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_TRUNC_S_F64): + DEF_OP_TRUNC(int64, I64, float64, F64, <= -9223372036854777856.0, + >= 9223372036854775808.0); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_I64_TRUNC_U_F64): + DEF_OP_TRUNC(uint64, I64, float64, F64, <= -1.0, + >= 18446744073709551616.0); + HANDLE_OP_END (); + + /* conversions of f32 */ + HANDLE_OP (WASM_OP_F32_CONVERT_S_I32): + DEF_OP_CONVERT(float32, F32, int32, I32); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_CONVERT_U_I32): + DEF_OP_CONVERT(float32, F32, uint32, I32); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_CONVERT_S_I64): + DEF_OP_CONVERT(float32, F32, int64, I64); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_CONVERT_U_I64): + DEF_OP_CONVERT(float32, F32, uint64, I64); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F32_DEMOTE_F64): + DEF_OP_CONVERT(float32, F32, float64, F64); + HANDLE_OP_END (); + + /* conversions of f64 */ + HANDLE_OP (WASM_OP_F64_CONVERT_S_I32): + DEF_OP_CONVERT(float64, F64, int32, I32); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_CONVERT_U_I32): + DEF_OP_CONVERT(float64, F64, uint32, I32); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_CONVERT_S_I64): + DEF_OP_CONVERT(float64, F64, int64, I64); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_CONVERT_U_I64): + DEF_OP_CONVERT(float64, F64, uint64, I64); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_F64_PROMOTE_F32): + DEF_OP_CONVERT(float64, F64, float32, F32); + HANDLE_OP_END (); + + /* reinterpretations */ + HANDLE_OP (WASM_OP_I32_REINTERPRET_F32): + DEF_OP_REINTERPRET(float32); + HANDLE_OP_END (); + HANDLE_OP (WASM_OP_I64_REINTERPRET_F64): + DEF_OP_REINTERPRET(float64); + HANDLE_OP_END (); + HANDLE_OP (WASM_OP_F32_REINTERPRET_I32): + DEF_OP_REINTERPRET(int32); + HANDLE_OP_END (); + HANDLE_OP (WASM_OP_F64_REINTERPRET_I64): + DEF_OP_REINTERPRET(int64); + HANDLE_OP_END (); + + HANDLE_OP (EXT_OP_COPY_STACK_TOP): + addr1 = GET_OFFSET(); + addr2 = GET_OFFSET(); + frame_lp[addr2] = frame_lp[addr1]; + HANDLE_OP_END (); + + HANDLE_OP (EXT_OP_COPY_STACK_TOP_I64): + addr1 = GET_OFFSET(); + addr2 = GET_OFFSET(); + *(float64*)(frame_lp + addr2) = *(float64*)(frame_lp + addr1); + HANDLE_OP_END (); + + HANDLE_OP (WASM_OP_SET_LOCAL): + HANDLE_OP (WASM_OP_TEE_LOCAL): + { + GET_LOCAL_INDEX_TYPE_AND_OFFSET(); + + switch (local_type) { + case VALUE_TYPE_I32: + case VALUE_TYPE_F32: + *(int32*)(frame_lp + local_offset) = GET_OPERAND(uint32, 0); + break; + case VALUE_TYPE_I64: + case VALUE_TYPE_F64: + PUT_I64_TO_ADDR((uint32*)(frame_lp + local_offset), GET_OPERAND(uint64, 0)); + break; + default: + wasm_set_exception(module, "invalid local type"); + goto got_exception; + } + + HANDLE_OP_END (); + } + + HANDLE_OP (WASM_OP_IMPDEP): + frame = prev_frame; + frame_ip = frame->ip; + goto call_func_from_entry; + +#if WASM_ENABLE_LABELS_AS_VALUES == 0 + default: + wasm_set_exception(module, "WASM interp failed: unsupported opcode."); + goto got_exception; + } +#endif + +#if WASM_ENABLE_LABELS_AS_VALUES != 0 + HANDLE_OP (WASM_OP_UNUSED_0x06): + HANDLE_OP (WASM_OP_UNUSED_0x07): + HANDLE_OP (WASM_OP_UNUSED_0x08): + HANDLE_OP (WASM_OP_UNUSED_0x09): + HANDLE_OP (WASM_OP_UNUSED_0x0a): + HANDLE_OP (WASM_OP_UNUSED_0x12): + HANDLE_OP (WASM_OP_UNUSED_0x13): + HANDLE_OP (WASM_OP_UNUSED_0x14): + HANDLE_OP (WASM_OP_UNUSED_0x15): + HANDLE_OP (WASM_OP_UNUSED_0x16): + HANDLE_OP (WASM_OP_UNUSED_0x17): + HANDLE_OP (WASM_OP_UNUSED_0x18): + HANDLE_OP (WASM_OP_UNUSED_0x19): + HANDLE_OP (WASM_OP_UNUSED_0x1c): + HANDLE_OP (WASM_OP_UNUSED_0x1d): + HANDLE_OP (WASM_OP_UNUSED_0x1e): + HANDLE_OP (WASM_OP_UNUSED_0x1f): + HANDLE_OP (WASM_OP_UNUSED_0x25): + HANDLE_OP (WASM_OP_UNUSED_0x26): + HANDLE_OP (WASM_OP_UNUSED_0x27): + /* optimized op code */ + HANDLE_OP (EXT_OP_GET_LOCAL_FAST): + HANDLE_OP (WASM_OP_GET_LOCAL): + HANDLE_OP (WASM_OP_F64_CONST): + HANDLE_OP (WASM_OP_I64_CONST): + HANDLE_OP (WASM_OP_F32_CONST): + HANDLE_OP (WASM_OP_I32_CONST): + HANDLE_OP (WASM_OP_DROP): + HANDLE_OP (WASM_OP_DROP_64): + HANDLE_OP (WASM_OP_BLOCK): + HANDLE_OP (WASM_OP_LOOP): + HANDLE_OP (WASM_OP_END): + HANDLE_OP (WASM_OP_NOP): + { + wasm_set_exception(module, "WASM interp failed: unsupported opcode."); + goto got_exception; + } +#endif + +#if WASM_ENABLE_LABELS_AS_VALUES == 0 + continue; +#else + FETCH_OPCODE_AND_DISPATCH (); +#endif + + call_func_from_interp: + /* Only do the copy when it's called from interpreter. */ + { + WASMInterpFrame *outs_area = wasm_exec_env_wasm_stack_top(exec_env); + outs_area->lp = outs_area->operand + cur_func->const_cell_num; + for (int i = 0; i < cur_func->param_count; i++) { + if (cur_func->param_types[i] == VALUE_TYPE_I64 + || cur_func->param_types[i] == VALUE_TYPE_F64) { + *(int64*)(outs_area->lp) = + GET_OPERAND(int64, (2 * (cur_func->param_count - i - 1))); + outs_area->lp += 2; + } else { + *(outs_area->lp) = GET_OPERAND(int32, (2 * (cur_func->param_count - i - 1)));; + outs_area->lp ++; + } + } + frame_ip += cur_func->param_count * sizeof(int16); + if (cur_func->ret_cell_num != 0) + frame->ret_offset = GET_OFFSET(); + SYNC_ALL_TO_FRAME(); + prev_frame = frame; + } + + call_func_from_entry: + { + if (cur_func->is_import_func) { + wasm_interp_call_func_native(module, exec_env, cur_func, prev_frame); + prev_frame = frame->prev_frame; + cur_func = frame->function; + UPDATE_ALL_FROM_FRAME(); + + memory = module->default_memory; + if (wasm_get_exception(module)) + goto got_exception; + } + else { + WASMFunction *cur_wasm_func = cur_func->u.func; + + all_cell_num = (uint64)cur_func->param_cell_num + + (uint64)cur_func->local_cell_num + + (uint64)cur_func->const_cell_num + + (uint64)cur_wasm_func->max_stack_cell_num; + if (all_cell_num >= UINT32_MAX) { + wasm_set_exception(module, "WASM interp failed: stack overflow."); + goto got_exception; + } + + frame_size = wasm_interp_interp_frame_size((uint32)all_cell_num); + if (!(frame = ALLOC_FRAME(exec_env, frame_size, prev_frame))) { + frame = prev_frame; + goto got_exception; + } + + /* Initialize the interpreter context. */ + frame->function = cur_func; + frame_ip = wasm_get_func_code(cur_func); + frame_ip_end = wasm_get_func_code_end(cur_func); + + frame_lp = frame->lp = frame->operand + cur_wasm_func->const_cell_num; + + /* Initialize the consts */ + bh_memcpy_s(frame->operand, all_cell_num * 4, + cur_wasm_func->consts, cur_wasm_func->const_cell_num * 4); + + /* Initialize the local varialbes */ + memset(frame_lp + cur_func->param_cell_num, 0, + (uint32)(cur_func->local_cell_num * 4)); + + wasm_exec_env_set_cur_frame(exec_env, (WASMRuntimeFrame*)frame); + } + HANDLE_OP_END (); + } + + return_func: + { + FREE_FRAME(exec_env, frame); + wasm_exec_env_set_cur_frame(exec_env, (WASMRuntimeFrame*)prev_frame); + + if (!prev_frame->ip) + /* Called from native. */ + return; + + RECOVER_CONTEXT(prev_frame); + HANDLE_OP_END (); + } + + (void)frame_ip_end; + + out_of_bounds: + wasm_set_exception(module, "out of bounds memory access"); + + got_exception: + return; + +#if WASM_ENABLE_LABELS_AS_VALUES == 0 + } +#else + FETCH_OPCODE_AND_DISPATCH (); +#endif +} + +#if WASM_ENABLE_FAST_INTERP != 0 +void ** +wasm_interp_get_handle_table() +{ + WASMModuleInstance module; + memset(&module, 0, sizeof(WASMModuleInstance)); + wasm_interp_call_func_bytecode(&module, NULL, NULL, NULL); + return global_handle_table; +} +#endif + +void +wasm_interp_call_wasm(WASMModuleInstance *module_inst, + WASMExecEnv *exec_env, + WASMFunctionInstance *function, + uint32 argc, uint32 argv[]) +{ + WASMRuntimeFrame *prev_frame = wasm_exec_env_get_cur_frame(exec_env); + WASMInterpFrame *frame, *outs_area; + + /* Allocate sufficient cells for all kinds of return values. */ + unsigned all_cell_num = 2, i; + /* This frame won't be used by JITed code, so only allocate interp + frame here. */ + unsigned frame_size = wasm_interp_interp_frame_size(all_cell_num); + + if (argc != function->param_cell_num) { + char buf[128]; + snprintf(buf, sizeof(buf), + "invalid argument count %d, expected %d", + argc, function->param_cell_num); + wasm_set_exception(module_inst, buf); + return; + } + + /* TODO: check stack overflow. */ + + if (!(frame = ALLOC_FRAME(exec_env, frame_size, (WASMInterpFrame*)prev_frame))) + return; + + outs_area = wasm_exec_env_wasm_stack_top(exec_env); + frame->function = NULL; + frame->ip = NULL; + /* There is no local variable. */ + frame->lp = frame->operand + 0; + frame->ret_offset = 0; + + if (argc > 0) + word_copy(outs_area->operand + function->const_cell_num, argv, argc); + + wasm_exec_env_set_cur_frame(exec_env, frame); + + if (function->is_import_func) + wasm_interp_call_func_native(module_inst, exec_env, function, frame); + else + wasm_interp_call_func_bytecode(module_inst, exec_env, function, frame); + + /* Output the return value to the caller */ + if (!wasm_get_exception(module_inst)) { + for (i = 0; i < function->ret_cell_num; i++) + argv[i] = *(frame->lp + i); + } + + wasm_exec_env_set_cur_frame(exec_env, prev_frame); + FREE_FRAME(exec_env, frame); +} diff --git a/core/iwasm/interpreter/wasm_loader.c b/core/iwasm/interpreter/wasm_loader.c index f9968c92d..42da5be16 100644 --- a/core/iwasm/interpreter/wasm_loader.c +++ b/core/iwasm/interpreter/wasm_loader.c @@ -1455,6 +1455,13 @@ wasm_loader_prepare_bytecode(WASMModule *module, WASMFunction *func, BlockAddr *block_addr_cache, char *error_buf, uint32 error_buf_size); +#if WASM_ENABLE_FAST_INTERP != 0 +void ** +wasm_interp_get_handle_table(); + +static void **handle_table; +#endif + static bool load_from_sections(WASMModule *module, WASMSection *sections, char *error_buf, uint32 error_buf_size) @@ -1551,6 +1558,10 @@ load_from_sections(WASMModule *module, WASMSection *sections, section = section->next; } +#if WASM_ENABLE_FAST_INTERP != 0 + handle_table = wasm_interp_get_handle_table(); +#endif + total_size = sizeof(BlockAddr) * (uint64)BLOCK_ADDR_CACHE_SIZE * BLOCK_ADDR_CONFLICT_SIZE; if (total_size >= UINT32_MAX || !(block_addr_cache = wasm_malloc((uint32)total_size))) { @@ -1781,7 +1792,7 @@ create_sections(const uint8 *buf, uint32 size, memset(section, 0, sizeof(WASMSection)); section->section_type = section_type; - section->section_body = p; + section->section_body = (uint8*)p; section->section_body_size = section_size; if (!*p_section_list) @@ -1863,11 +1874,9 @@ load(const uint8 *buf, uint32 size, WASMModule *module, return true; } -const uint8* wasm_file; WASMModule* wasm_loader_load(const uint8 *buf, uint32 size, char *error_buf, uint32 error_buf_size) { - wasm_file = buf; WASMModule *module = wasm_malloc(sizeof(WASMModule)); if (!module) { @@ -1918,6 +1927,12 @@ wasm_loader_unload(WASMModule *module) if (module->functions[i]) { if (module->functions[i]->local_offsets) wasm_free(module->functions[i]->local_offsets); +#if WASM_ENABLE_FAST_INTERP != 0 + if (module->functions[i]->code_compiled) + wasm_free(module->functions[i]->code_compiled); + if (module->functions[i]->consts) + wasm_free(module->functions[i]->consts); +#endif wasm_free(module->functions[i]); } } @@ -2092,9 +2107,9 @@ wasm_loader_find_block_addr(BlockAddr *block_addr_cache, skip_leb_uint32(p, p_end); /* localidx */ break; - case WASM_OP_GET_LOCAL_FAST: - case WASM_OP_SET_LOCAL_FAST: - case WASM_OP_TEE_LOCAL_FAST: + case EXT_OP_GET_LOCAL_FAST: + case EXT_OP_SET_LOCAL_FAST: + case EXT_OP_TEE_LOCAL_FAST: CHECK_BUF(p, p_end, 1); p++; break; @@ -2289,6 +2304,23 @@ wasm_loader_find_block_addr(BlockAddr *block_addr_cache, #define REF_F64_1 VALUE_TYPE_F64 #define REF_F64_2 VALUE_TYPE_F64 +#if WASM_ENABLE_FAST_INTERP != 0 + +#if WASM_DEBUG_PREPROCESSOR != 0 +#define LOG_OP(...) bh_printf(__VA_ARGS__) +#else +#define LOG_OP(...) +#endif + +#define PATCH_ELSE 0 +#define PATCH_END 1 +typedef struct BranchBlockPatch { + struct BranchBlockPatch *next; + uint8 patch_type; + uint8 *code_compiled; +} BranchBlockPatch; +#endif + typedef struct BranchBlock { uint8 block_type; uint8 return_type; @@ -2297,10 +2329,62 @@ typedef struct BranchBlock { uint8 *else_addr; uint8 *end_addr; uint32 stack_cell_num; +#if WASM_ENABLE_FAST_INTERP != 0 + uint16 dynamic_offset; + uint8 *code_compiled; + BranchBlockPatch *patch_list; +#endif } BranchBlock; +typedef struct WASMLoaderContext { + /* frame ref stack */ + uint8 *frame_ref; + uint8 *frame_ref_bottom; + uint8 *frame_ref_boundary; + uint32 frame_ref_size; + uint32 stack_cell_num; + uint32 max_stack_cell_num; + + /* frame csp stack */ + BranchBlock *frame_csp; + BranchBlock *frame_csp_bottom; + BranchBlock *frame_csp_boundary; + uint32 frame_csp_size; + uint32 csp_num; + uint32 max_csp_num; + +#if WASM_ENABLE_FAST_INTERP != 0 + /* frame offset stack */ + int16 *frame_offset; + int16 *frame_offset_bottom; + int16 *frame_offset_boundary; + uint32 frame_offset_size; + int16 dynamic_offset; + int16 start_dynamic_offset; + int16 max_dynamic_offset; + + /* const buffer */ + uint8 *const_buf; + uint16 num_const; + uint16 const_buf_size; + uint16 const_cell_num; + + /* processed code */ + uint8 *p_code_compiled; + uint8 *p_code_compiled_end; + uint32 code_compiled_size; +#endif +} WASMLoaderContext; + +typedef struct Const { + WASMValue value; + uint16 slot_index; + uint8 value_type; +} Const; + static void* -memory_realloc(void *mem_old, uint32 size_old, uint32 size_new) +memory_realloc(void *mem_old, uint32 size_old, uint32 size_new, + char *error_buf, uint32 error_buf_size) { uint8 *mem_new; bh_assert(size_new > size_old); @@ -2309,67 +2393,124 @@ memory_realloc(void *mem_old, uint32 size_old, uint32 size_new) memset(mem_new + size_old, 0, size_new - size_old); wasm_free(mem_old); } + else { + set_error_buf(error_buf, error_buf_size, + "WASM loader prepare bytecode failed: " + "allocate memory failed."); + } return mem_new; } -#define MEM_REALLOC(mem, size_old, size_new) do { \ - void *mem_new = memory_realloc(mem, size_old, size_new);\ - if (!mem_new) { \ - set_error_buf(error_buf, error_buf_size, \ - "WASM loader prepare bytecode failed: " \ - "allocate memory failed."); \ - goto fail; \ - } \ - mem = mem_new; \ +#define MEM_REALLOC(mem, size_old, size_new) do { \ + void *mem_new = memory_realloc(mem, size_old, size_new, \ + error_buf, error_buf_size); \ + if (!mem_new) \ + goto fail; \ + mem = mem_new; \ } while (0) +#define CHECK_CSP_PUSH() do { \ + if (ctx->frame_csp >= ctx->frame_csp_boundary) { \ + MEM_REALLOC(ctx->frame_csp_bottom, ctx->frame_csp_size, \ + (uint32)(ctx->frame_csp_size \ + + 8 * sizeof(BranchBlock))); \ + ctx->frame_csp_size += (uint32)(8 * sizeof(BranchBlock)); \ + ctx->frame_csp_boundary = ctx->frame_csp_bottom + \ + ctx->frame_csp_size / sizeof(BranchBlock); \ + ctx->frame_csp = ctx->frame_csp_bottom + ctx->csp_num; \ + } \ + } while (0) + +#define CHECK_CSP_POP() do { \ + if (ctx->csp_num < 1) { \ + set_error_buf(error_buf, error_buf_size, \ + "WASM module load failed: type mismatch: " \ + "expect data but block stack was empty"); \ + goto fail; \ + } \ + } while (0) + +#if WASM_ENABLE_FAST_INTERP != 0 static bool -check_stack_push(uint8 **p_frame_ref_bottom, uint8 **p_frame_ref_boundary, - uint8 **p_frame_ref, uint32 *p_frame_ref_size, - uint32 stack_cell_num, - char *error_buf, uint32 error_buf_size) +check_offset_push(WASMLoaderContext *ctx, + char *error_buf, uint32 error_buf_size) { - if (*p_frame_ref >= *p_frame_ref_boundary) { - MEM_REALLOC(*p_frame_ref_bottom, *p_frame_ref_size, - *p_frame_ref_size + 16); - *p_frame_ref_size += 16; - *p_frame_ref_boundary = *p_frame_ref_bottom + *p_frame_ref_size; - *p_frame_ref = *p_frame_ref_bottom + stack_cell_num; + if (ctx->frame_offset >= ctx->frame_offset_boundary) { + MEM_REALLOC(ctx->frame_offset_bottom, ctx->frame_offset_size, + ctx->frame_offset_size + 16); + ctx->frame_offset_size += 16; + ctx->frame_offset_boundary = ctx->frame_offset_bottom + + ctx->frame_offset_size / sizeof(int16); + ctx->frame_offset = ctx->frame_offset_bottom + ctx->stack_cell_num; } return true; fail: return false; } -#define CHECK_STACK_PUSH() do { \ - if (!check_stack_push(&frame_ref_bottom, &frame_ref_boundary,\ - &frame_ref, &frame_ref_size, \ - stack_cell_num, \ - error_buf, error_buf_size)) \ - goto fail; \ - } while (0) +static void free_label_patch_list(BranchBlock *frame_csp) +{ + BranchBlockPatch *label_patch = frame_csp->patch_list; + BranchBlockPatch *next; + while (label_patch != NULL) { + next = label_patch->next; + wasm_free(label_patch); + label_patch = next; + } + frame_csp->patch_list = NULL; +} + +static void free_all_label_patch_lists(BranchBlock *frame_csp, uint32 csp_num) +{ + BranchBlock *tmp_csp = frame_csp; + + for (uint32 i = 0; i < csp_num; i++) { + free_label_patch_list(tmp_csp); + tmp_csp ++; + } +} + +#endif static bool -check_stack_pop(uint8 type, uint8 *frame_ref, uint32 stack_cell_num, +check_stack_push(WASMLoaderContext *ctx, + char *error_buf, uint32 error_buf_size) +{ + if (ctx->frame_ref >= ctx->frame_ref_boundary) { + MEM_REALLOC(ctx->frame_ref_bottom, ctx->frame_ref_size, + ctx->frame_ref_size + 16); + ctx->frame_ref_size += 16; + ctx->frame_ref_boundary = ctx->frame_ref_bottom + ctx->frame_ref_size; + ctx->frame_ref = ctx->frame_ref_bottom + ctx->stack_cell_num; + } + return true; +fail: + return false; +} + +static bool +check_stack_pop(WASMLoaderContext *ctx, uint8 type, char *error_buf, uint32 error_buf_size, const char *type_str) { if (((type == VALUE_TYPE_I32 || type == VALUE_TYPE_F32) - && stack_cell_num < 1) + && ctx->stack_cell_num < 1) || ((type == VALUE_TYPE_I64 || type == VALUE_TYPE_F64) - && stack_cell_num < 2)) { + && ctx->stack_cell_num < 2)) { set_error_buf(error_buf, error_buf_size, "WASM module load failed: " "type mismatch: expect data but stack was empty"); return false; } - if ((type == VALUE_TYPE_I32 && *(frame_ref - 1) != REF_I32) - || (type == VALUE_TYPE_F32 && *(frame_ref - 1) != REF_F32) + if ((type == VALUE_TYPE_I32 && *(ctx->frame_ref - 1) != REF_I32) + || (type == VALUE_TYPE_F32 && *(ctx->frame_ref - 1) != REF_F32) || (type == VALUE_TYPE_I64 - && (*(frame_ref - 2) != REF_I64_1 || *(frame_ref - 1) != REF_I64_2)) + && (*(ctx->frame_ref - 2) != REF_I64_1 + || *(ctx->frame_ref - 1) != REF_I64_2)) || (type == VALUE_TYPE_F64 - && (*(frame_ref - 2) != REF_F64_1 || *(frame_ref - 1) != REF_F64_2))) { + && (*(ctx->frame_ref - 2) != REF_F64_1 + || *(ctx->frame_ref - 1) != REF_F64_2))) { if (error_buf != NULL) snprintf(error_buf, error_buf_size, "%s%s%s", "WASM module load failed: type mismatch: expect ", @@ -2379,215 +2520,715 @@ check_stack_pop(uint8 type, uint8 *frame_ref, uint32 stack_cell_num, return true; } -#define CHECK_STACK_POP(TYPE, type) do { \ - if (!check_stack_pop(VALUE_TYPE_##TYPE, \ - frame_ref, stack_cell_num, \ - error_buf, error_buf_size, #type)) \ - goto fail; \ - } while (0) +static void wasm_loader_ctx_destroy(WASMLoaderContext *ctx) +{ + if (ctx) { + if (ctx->frame_ref_bottom) + wasm_free(ctx->frame_ref_bottom); + if (ctx->frame_csp_bottom) { +#if WASM_ENABLE_FAST_INTERP != 0 + free_all_label_patch_lists(ctx->frame_csp_bottom, ctx->csp_num); +#endif + wasm_free(ctx->frame_csp_bottom); + } +#if WASM_ENABLE_FAST_INTERP != 0 + if (ctx->frame_offset_bottom) + wasm_free(ctx->frame_offset_bottom); + if (ctx->const_buf) + wasm_free(ctx->const_buf); +#endif + wasm_free(ctx); + } +} -#define PUSH_I32() do { \ - CHECK_STACK_PUSH(); \ - *frame_ref++ = REF_I32; \ - stack_cell_num++; \ - if (stack_cell_num > max_stack_cell_num) \ - max_stack_cell_num = stack_cell_num; \ - } while (0) +static WASMLoaderContext* +wasm_loader_ctx_init(WASMFunction *func) +{ + WASMLoaderContext *loader_ctx = + wasm_malloc(sizeof(WASMLoaderContext)); + if (!loader_ctx) + return false; + memset(loader_ctx, 0, sizeof(WASMLoaderContext)); -#define PUSH_F32() do { \ - CHECK_STACK_PUSH(); \ - *frame_ref++ = REF_F32; \ - stack_cell_num++; \ - if (stack_cell_num > max_stack_cell_num) \ - max_stack_cell_num = stack_cell_num; \ - } while (0) + loader_ctx->frame_ref_size = 32; + if (!(loader_ctx->frame_ref_bottom = loader_ctx->frame_ref = + wasm_malloc(loader_ctx->frame_ref_size))) + goto fail; + memset(loader_ctx->frame_ref_bottom, 0, loader_ctx->frame_ref_size); + loader_ctx->frame_ref_boundary = loader_ctx->frame_ref_bottom + + loader_ctx->frame_ref_size; -#define PUSH_I64() do { \ - CHECK_STACK_PUSH(); \ - *frame_ref++ = REF_I64_1; \ - stack_cell_num++; \ - CHECK_STACK_PUSH(); \ - *frame_ref++ = REF_I64_2; \ - stack_cell_num++; \ - if (stack_cell_num > max_stack_cell_num) \ - max_stack_cell_num = stack_cell_num; \ - } while (0) + loader_ctx->frame_csp_size = sizeof(BranchBlock) * 8; + if (!(loader_ctx->frame_csp_bottom = loader_ctx->frame_csp = + wasm_malloc(loader_ctx->frame_csp_size))) + goto fail; + memset(loader_ctx->frame_csp_bottom, 0, loader_ctx->frame_csp_size); + loader_ctx->frame_csp_boundary = loader_ctx->frame_csp_bottom + 8; -#define PUSH_F64() do { \ - CHECK_STACK_PUSH(); \ - *frame_ref++ = REF_F64_1; \ - stack_cell_num++; \ - CHECK_STACK_PUSH(); \ - *frame_ref++ = REF_F64_2; \ - stack_cell_num++; \ - if (stack_cell_num > max_stack_cell_num) \ - max_stack_cell_num = stack_cell_num; \ - } while (0) +#if WASM_ENABLE_FAST_INTERP != 0 + loader_ctx->frame_offset_size = sizeof(int16) * 32; + if (!(loader_ctx->frame_offset_bottom = loader_ctx->frame_offset = + wasm_malloc(loader_ctx->frame_offset_size))) + goto fail; + memset(loader_ctx->frame_offset_bottom, 0, + loader_ctx->frame_offset_size); + loader_ctx->frame_offset_boundary = loader_ctx->frame_offset_bottom + 32; -#define POP_I32() do { \ - CHECK_STACK_POP(I32, i32); \ - stack_cell_num--; \ - frame_ref--; \ - } while (0) + loader_ctx->num_const = 0; + loader_ctx->const_buf_size = sizeof(Const) * 8; + if (!(loader_ctx->const_buf = wasm_malloc(loader_ctx->const_buf_size))) + goto fail; + memset(loader_ctx->const_buf, 0, loader_ctx->const_buf_size); -#define POP_I64() do { \ - CHECK_STACK_POP(I64, i64); \ - stack_cell_num -= 2; \ - frame_ref -= 2; \ - } while (0) + loader_ctx->start_dynamic_offset = loader_ctx->dynamic_offset = + loader_ctx->max_dynamic_offset = func->param_cell_num + + func->local_cell_num; +#endif + return loader_ctx; -#define POP_F32() do { \ - CHECK_STACK_POP(F32, f32); \ - stack_cell_num--; \ - frame_ref--; \ - } while (0) - -#define POP_F64() do { \ - CHECK_STACK_POP(F64, f64); \ - stack_cell_num -= 2; \ - frame_ref -= 2; \ - } while (0) +fail: + wasm_loader_ctx_destroy(loader_ctx); + return NULL; +} static bool -push_type(uint8 type, uint8 **p_frame_ref_bottom, - uint8 **p_frame_ref_boundary, - uint8 **p_frame_ref, uint32 *p_frame_ref_size, - uint32 *p_stack_cell_num, uint32 *p_max_stack_cell_num, - char *error_buf, uint32 error_buf_size) +wasm_loader_push_frame_ref(WASMLoaderContext *ctx, uint8 type, + char *error_buf, uint32 error_buf_size) { - uint8 *frame_ref = *p_frame_ref; - uint32 frame_ref_size = *p_frame_ref_size; - uint32 max_stack_cell_num = *p_max_stack_cell_num; - uint32 stack_cell_num = *p_stack_cell_num; + if (type == VALUE_TYPE_VOID) + return true; - switch (type) { - case VALUE_TYPE_I64: - case VALUE_TYPE_F64: - if (!check_stack_push(p_frame_ref_bottom, p_frame_ref_boundary, - &frame_ref, &frame_ref_size, - stack_cell_num, - error_buf, error_buf_size)) - goto fail; - *frame_ref++ = type; - stack_cell_num++; - if (stack_cell_num > max_stack_cell_num) - max_stack_cell_num = stack_cell_num; - goto handle_i32_f32; -handle_i32_f32: - case VALUE_TYPE_I32: - case VALUE_TYPE_F32: - if (!check_stack_push(p_frame_ref_bottom, p_frame_ref_boundary, - &frame_ref, &frame_ref_size, - stack_cell_num, - error_buf, error_buf_size)) - goto fail; - *frame_ref++ = type; - stack_cell_num++; - if (stack_cell_num > max_stack_cell_num) - max_stack_cell_num = stack_cell_num; - break; - } + if (!check_stack_push(ctx, error_buf, error_buf_size)) + return false; - *p_frame_ref = frame_ref; - *p_frame_ref_size = frame_ref_size; - *p_max_stack_cell_num = max_stack_cell_num; - *p_stack_cell_num = stack_cell_num; + *ctx->frame_ref++ = type; + ctx->stack_cell_num++; + if (ctx->stack_cell_num > ctx->max_stack_cell_num) + ctx->max_stack_cell_num = ctx->stack_cell_num; + + if (type == VALUE_TYPE_I32 || type == VALUE_TYPE_F32) + return true; + + if (!check_stack_push(ctx, error_buf, error_buf_size)) + return false; + *ctx->frame_ref++ = type; + ctx->stack_cell_num++; + if (ctx->stack_cell_num > ctx->max_stack_cell_num) + ctx->max_stack_cell_num = ctx->stack_cell_num; + return true; +} + +static bool +wasm_loader_pop_frame_ref(WASMLoaderContext *ctx, uint8 type, + char *error_buf, uint32 error_buf_size) +{ + char *type_str[] = { "f64", "f32", "i64", "i32" }; + if (type == VALUE_TYPE_VOID) + return true; + + if (!check_stack_pop(ctx, type, error_buf, error_buf_size, + type_str[type - VALUE_TYPE_F64])) + return false; + + ctx->frame_ref--; + ctx->stack_cell_num--; + + if (type == VALUE_TYPE_I32 || type == VALUE_TYPE_F32) + return true; + + ctx->frame_ref--; + ctx->stack_cell_num--; + return true; +} + +static bool +wasm_loader_push_frame_csp(WASMLoaderContext *ctx, uint8 type, + uint8 ret_type, uint8* start_addr, + char *error_buf, uint32 error_buf_size) +{ + CHECK_CSP_PUSH(); + memset(ctx->frame_csp, 0, sizeof(BranchBlock)); + ctx->frame_csp->block_type = type; + ctx->frame_csp->return_type = ret_type; + ctx->frame_csp->start_addr = start_addr; + ctx->frame_csp->stack_cell_num = ctx->stack_cell_num; +#if WASM_ENABLE_FAST_INTERP != 0 + ctx->frame_csp->dynamic_offset = ctx->dynamic_offset; + ctx->frame_csp->patch_list = NULL; +#endif + ctx->frame_csp++; + ctx->csp_num++; + if (ctx->csp_num > ctx->max_csp_num) + ctx->max_csp_num = ctx->csp_num; return true; fail: return false; } -#define PUSH_TYPE(type) do { \ - if (!push_type(type, &frame_ref_bottom, \ - &frame_ref_boundary, \ - &frame_ref, &frame_ref_size, \ - &stack_cell_num, &max_stack_cell_num, \ - error_buf, error_buf_size)) \ - goto fail; \ - } while (0) +static bool +wasm_loader_pop_frame_csp(WASMLoaderContext *ctx, + char *error_buf, uint32 error_buf_size) +{ + CHECK_CSP_POP(); + ctx->frame_csp--; + ctx->csp_num--; + return true; +fail: + return false; +} static bool -pop_type(uint8 type, uint8 **p_frame_ref, uint32 *p_stack_cell_num, - char *error_buf, uint32 error_buf_size) +wasm_loader_check_br(WASMLoaderContext *ctx, uint32 depth, + char *error_buf, uint32 error_buf_size) { - char *type_str[] = { "f64", "f32", "i64", "i32" }; - switch (type) { - case VALUE_TYPE_I64: - case VALUE_TYPE_F64: - if (!check_stack_pop(type, *p_frame_ref, *p_stack_cell_num, - error_buf, error_buf_size, - type_str[type - VALUE_TYPE_F64])) - return false; - *p_frame_ref -= 2; - *p_stack_cell_num -= 2; - break; - case VALUE_TYPE_I32: - case VALUE_TYPE_F32: - if (!check_stack_pop(type, *p_frame_ref, *p_stack_cell_num, - error_buf, error_buf_size, - type_str[type - VALUE_TYPE_F64])) - return false; - *p_frame_ref -= 1; - *p_stack_cell_num -= 1; - break; + if (ctx->csp_num < depth + 1) { + set_error_buf(error_buf, error_buf_size, + "WASM module load failed: type mismatch: " + "unexpected end of section or function"); + return false; + } + if ((ctx->frame_csp - (depth + 1))->block_type != BLOCK_TYPE_LOOP) { + uint8 tmp_ret_type = (ctx->frame_csp - (depth + 1))->return_type; + if ((tmp_ret_type == VALUE_TYPE_I32 + && (ctx->stack_cell_num < 1 || *(ctx->frame_ref - 1) != REF_I32)) + || (tmp_ret_type == VALUE_TYPE_F32 + && (ctx->stack_cell_num < 1 || *(ctx->frame_ref - 1) != REF_F32)) + || (tmp_ret_type == VALUE_TYPE_I64 + && (ctx->stack_cell_num < 2 + || *(ctx->frame_ref - 2) != REF_I64_1 + || *(ctx->frame_ref - 1) != REF_I64_2)) + || (tmp_ret_type == VALUE_TYPE_F64 + && (ctx->stack_cell_num < 2 + || *(ctx->frame_ref - 2) != REF_F64_1 + || *(ctx->frame_ref - 1) != REF_F64_2))) { + set_error_buf(error_buf, error_buf_size, + "WASM module load failed: type mismatch: " + "expect data but stack was empty or other type"); + return false; + } + (ctx->frame_csp - (depth + 1))->is_block_reachable = true; } return true; } -#define POP_TYPE(type) do { \ - if (!pop_type(type, &frame_ref, &stack_cell_num,\ - error_buf, error_buf_size)) \ - goto fail; \ +#if WASM_ENABLE_FAST_INTERP != 0 +static bool +wasm_loader_ctx_reinit(WASMLoaderContext *ctx) +{ + if (!(ctx->p_code_compiled = wasm_malloc(ctx->code_compiled_size))) + return false; + memset(ctx->p_code_compiled, 0, ctx->code_compiled_size); + ctx->p_code_compiled_end = ctx->p_code_compiled + + ctx->code_compiled_size; + + /* clean up frame ref */ + memset(ctx->frame_ref_bottom, 0, ctx->frame_ref_size); + ctx->frame_ref = ctx->frame_ref_bottom; + ctx->stack_cell_num = 0; + + /* clean up frame csp */ + memset(ctx->frame_csp_bottom, 0, ctx->frame_csp_size); + ctx->frame_csp = ctx->frame_csp_bottom; + ctx->csp_num = 0; + ctx->max_csp_num = 0; + + /* clean up frame offset */ + memset(ctx->frame_offset_bottom, 0, ctx->frame_offset_size); + ctx->frame_offset = ctx->frame_offset_bottom; + ctx->dynamic_offset = ctx->start_dynamic_offset; + + /* const buf is reserved */ + return true; +} + +static void +wasm_loader_emit_int16(WASMLoaderContext *ctx, int16 value) +{ + if (ctx->p_code_compiled) { + *(int16*)(ctx->p_code_compiled) = value; + ctx->p_code_compiled += sizeof(int16); + } + else + ctx->code_compiled_size += sizeof(int16); +} + +static void +wasm_loader_emit_uint8(WASMLoaderContext *ctx, uint8 value) +{ + if (ctx->p_code_compiled) { + *(ctx->p_code_compiled) = value; + ctx->p_code_compiled += sizeof(uint8); + } + else + ctx->code_compiled_size += sizeof(uint8); +} + +static void +wasm_loader_emit_ptr(WASMLoaderContext *ctx, void *value) +{ + if (ctx->p_code_compiled) { + *(uint8**)(ctx->p_code_compiled) = value; + ctx->p_code_compiled += sizeof(void *); + } + else + ctx->code_compiled_size += sizeof(void *); +} + +static void +wasm_loader_emit_backspace(WASMLoaderContext *ctx, uint32 size) +{ + if (ctx->p_code_compiled) { + ctx->p_code_compiled -= size; + } + else + ctx->code_compiled_size -= size; +} + +static void +wasm_loader_emit_leb(WASMLoaderContext *ctx, uint8* start, uint8* end) +{ + if (ctx->p_code_compiled) { + bh_memcpy_s(ctx->p_code_compiled, + ctx->p_code_compiled_end - ctx->p_code_compiled, + start, end - start); + ctx->p_code_compiled += (end - start); + } + else { + ctx->code_compiled_size += (end - start); + } + +} + +static bool +add_label_patch_to_list(BranchBlock *frame_csp, + uint8 patch_type, uint8 *p_code_compiled, + char *error_buf, uint32 error_buf_size) +{ + BranchBlockPatch *patch = wasm_malloc(sizeof(BranchBlockPatch)); + if (!patch) { + set_error_buf(error_buf, error_buf_size, + "WASM loader prepare bytecode failed: " + "allocate memory failed"); + return false; + } + patch->patch_type = patch_type; + patch->code_compiled = p_code_compiled; + if (!frame_csp->patch_list) { + frame_csp->patch_list = patch; + patch->next = NULL; + } + else { + patch->next = frame_csp->patch_list; + frame_csp->patch_list = patch; + } + return true; +} + +static void +apply_label_patch(WASMLoaderContext *ctx, uint8 depth, + uint8 patch_type, uint8 *frame_ip) +{ + BranchBlock *frame_csp = ctx->frame_csp - depth; + BranchBlockPatch *node = frame_csp->patch_list; + BranchBlockPatch *node_prev = NULL, *node_next; + + if (!ctx->p_code_compiled) + return; + + while (node) { + node_next = node->next; + if (node->patch_type == patch_type) { + *((uint8**)node->code_compiled) = ctx->p_code_compiled; + if (node_prev == NULL) { + frame_csp->patch_list = node_next; + } + else { + node_prev->next = node_next; + } + wasm_free(node); + } + else { + node_prev = node; + } + node = node_next; + } +} + +#define emit_operand(ctx, offset) do { \ + wasm_loader_emit_int16(ctx, offset); \ + LOG_OP("%d\t", offset); \ } while (0) -#define CHECK_CSP_PUSH() do { \ - if (frame_csp >= frame_csp_boundary) { \ - MEM_REALLOC(frame_csp_bottom, frame_csp_size, \ - (uint32)(frame_csp_size \ - + 8 * sizeof(BranchBlock))); \ - frame_csp_size += (uint32)(8 * sizeof(BranchBlock)); \ - frame_csp_boundary = frame_csp_bottom + \ - frame_csp_size / sizeof(BranchBlock); \ - frame_csp = frame_csp_bottom + csp_num; \ - } \ +#define emit_byte(ctx, byte) do { \ + wasm_loader_emit_uint8(ctx, byte); \ + LOG_OP("%d\t", byte); \ } while (0) -#define CHECK_CSP_POP() do { \ - if (csp_num < 1) { \ - set_error_buf(error_buf, error_buf_size, \ - "WASM module load failed: type mismatch: "\ - "expect data but block stack was empty"); \ - goto fail; \ - } \ +#define emit_leb() do { \ + wasm_loader_emit_leb(loader_ctx, p_org, p); \ } while (0) -#define PUSH_CSP(type, ret_type, _start_addr) do { \ - CHECK_CSP_PUSH(); \ - frame_csp->block_type = type; \ - frame_csp->is_block_reachable = false; \ - frame_csp->return_type = ret_type; \ - frame_csp->start_addr = _start_addr; \ - frame_csp->else_addr = NULL; \ - frame_csp->end_addr = NULL; \ - frame_csp->stack_cell_num = stack_cell_num; \ - frame_csp++; \ - csp_num++; \ - if (csp_num > max_csp_num) \ - max_csp_num = csp_num; \ +#define emit_const(value) do { \ + GET_CONST_OFFSET(VALUE_TYPE_I32, value); \ + emit_operand(loader_ctx, operand_offset); \ } while (0) -#define POP_CSP() do { \ - CHECK_CSP_POP(); \ - frame_csp--; \ - csp_num--; \ +static bool +wasm_loader_emit_br_info(WASMLoaderContext *ctx, BranchBlock *frame_csp, + char *error_buf, uint32 error_buf_size) +{ + emit_operand(ctx, frame_csp->dynamic_offset); + if (frame_csp->return_type == VALUE_TYPE_I32 + || frame_csp->return_type == VALUE_TYPE_F32) { + emit_byte(ctx, 1); + emit_operand(ctx, *(int16*)(ctx->frame_offset - 1)); + } + else if (frame_csp->return_type == VALUE_TYPE_I64 + || frame_csp->return_type == VALUE_TYPE_F64) { + emit_byte(ctx, 2); + emit_operand(ctx, *(int16*)(ctx->frame_offset - 2)); + } + else { + emit_byte(ctx, 0); + emit_operand(ctx, 0); + } + if (frame_csp->block_type == BLOCK_TYPE_LOOP) { + wasm_loader_emit_ptr(ctx, frame_csp->code_compiled); + } + else { + if (!add_label_patch_to_list(frame_csp, PATCH_END, + ctx->p_code_compiled, + error_buf, error_buf_size)) + return false; + /* label address, to be patched */ + wasm_loader_emit_ptr(ctx, NULL); + } + return true; +} + +static bool +wasm_loader_push_frame_offset(WASMLoaderContext *ctx, uint8 type, + bool disable_emit, int16 operand_offset, + char *error_buf, uint32 error_buf_size) +{ + if (type == VALUE_TYPE_VOID) + return true; + + // only check memory overflow in first traverse + if (ctx->p_code_compiled == NULL) { + if (!check_offset_push(ctx, error_buf, error_buf_size)) + return false; + } + + if (disable_emit) + *(ctx->frame_offset)++ = operand_offset; + else { + emit_operand(ctx, ctx->dynamic_offset); + *(ctx->frame_offset)++ = ctx->dynamic_offset; + ctx->dynamic_offset++; + if (ctx->dynamic_offset > ctx->max_dynamic_offset) + ctx->max_dynamic_offset = ctx->dynamic_offset; + } + + if (type == VALUE_TYPE_I32 || type == VALUE_TYPE_F32) + return true; + + if (ctx->p_code_compiled == NULL) { + if (!check_offset_push(ctx, error_buf, error_buf_size)) + return false; + } + + ctx->frame_offset++; + ctx->dynamic_offset++; + if (ctx->dynamic_offset > ctx->max_dynamic_offset) + ctx->max_dynamic_offset = ctx->dynamic_offset; + return true; +} + +/* The frame_offset stack should always keep the same depth with + frame_ref, so we don't check pop of frame_offset */ +static bool +wasm_loader_pop_frame_offset(WASMLoaderContext *ctx, uint8 type, + char *error_buf, uint32 error_buf_size) +{ + if (type == VALUE_TYPE_VOID) + return true; + + if (type == VALUE_TYPE_I32 || type == VALUE_TYPE_F32) { + ctx->frame_offset -= 1; + if (*(ctx->frame_offset) > ctx->start_dynamic_offset) + ctx->dynamic_offset -= 1; + } + else { + ctx->frame_offset -= 2; + if (*(ctx->frame_offset) > ctx->start_dynamic_offset) + ctx->dynamic_offset -= 2; + } + emit_operand(ctx, *(ctx->frame_offset)); + return true; +} + +static bool +wasm_loader_push_frame_ref_offset(WASMLoaderContext *ctx, uint8 type, + bool disable_emit, int16 operand_offset, + char *error_buf, uint32 error_buf_size) +{ + if (!(wasm_loader_push_frame_ref(ctx, type, error_buf, error_buf_size))) + return false; + if (!(wasm_loader_push_frame_offset(ctx, type, disable_emit, operand_offset, + error_buf, error_buf_size))) + return false; + + return true; +} + +static bool +wasm_loader_pop_frame_ref_offset(WASMLoaderContext *ctx, uint8 type, + char *error_buf, uint32 error_buf_size) +{ + if (!wasm_loader_pop_frame_ref(ctx, type, error_buf, error_buf_size)) + return false; + if (!wasm_loader_pop_frame_offset(ctx, type, error_buf, error_buf_size)) + return false; + + return true; +} + + +static bool +wasm_loader_get_const_offset(WASMLoaderContext *ctx, uint8 type, + int64 val_int, float32 val_f32, + float64 val_f64, int16 *offset, + char *error_buf, uint32 error_buf_size) +{ + int16 operand_offset = 0; + Const *c; + for (c = (Const *)ctx->const_buf; + (uint8*)c < ctx->const_buf + ctx->num_const * sizeof(Const); c ++) { + if ((type == c->value_type) + && ((type == VALUE_TYPE_I64 && (int64)val_int == c->value.i64) + || (type == VALUE_TYPE_I32 && (int32)val_int == c->value.i32) + || (type == VALUE_TYPE_F64 && (float64)val_f64 == c->value.f64) + || (type == VALUE_TYPE_F32 && (float32)val_f32 == c->value.f32))) { + operand_offset = c->slot_index; + break; + } + if (c->value_type == VALUE_TYPE_I64 + || c->value_type == VALUE_TYPE_F64) + operand_offset += 2; + else + operand_offset += 1; + } + if ((uint8 *)c == ctx->const_buf + ctx->num_const * sizeof(Const)) { + if ((uint8 *)c == ctx->const_buf + ctx->const_buf_size) { + MEM_REALLOC(ctx->const_buf, + ctx->const_buf_size, + ctx->const_buf_size + 4 * sizeof(Const)); + ctx->const_buf_size += 4 * sizeof(Const); + c = (Const *)(ctx->const_buf + ctx->num_const * sizeof(Const)); + } + c->value_type = type; + switch (type) { + case VALUE_TYPE_F64: + c->value.f64 = (float64)val_f64; + ctx->const_cell_num += 2; + /* The const buf will be reversed, we use the second cell */ + /* of the i64/f64 const so the finnal offset is corrent */ + operand_offset ++; + break; + case VALUE_TYPE_I64: + c->value.i64 = (int64)val_int; + ctx->const_cell_num += 2; + operand_offset ++; + break; + case VALUE_TYPE_F32: + c->value.f32 = (float32)val_f32; + ctx->const_cell_num ++; + break; + case VALUE_TYPE_I32: + c->value.i32 = (int32)val_int; + ctx->const_cell_num ++; + break; + default: + break; + } + c->slot_index = operand_offset; + ctx->num_const ++; + LOG_OP("#### new const [%d]: %ld\n", num_const, (int64)val); + } + /* use negetive index for const */ + operand_offset = -(operand_offset + 1); + *offset = operand_offset; + return true; +fail: + return false; +} + +/* + PUSH(POP)_XXX = push(pop) frame_ref + push(pop) frame_offset + -- Mostly used for the binary / compare operation + PUSH(POP)_OFFSET_TYPE only push(pop) the frame_offset stack + -- Mostly used in block / control instructions + + The POP will always emit the offset on the top of the frame_offset stack + PUSH can be used in two ways: + 1. directly PUSH: + PUSH_XXX(); + will allocate a dynamic space and emit + 2. silent PUSH: + operand_offset = xxx; disable_emit = true; + PUSH_XXX(); + only push the frame_offset stack, no emit +*/ +#define PUSH_I32() do { \ + if (!wasm_loader_push_frame_ref_offset(loader_ctx, VALUE_TYPE_I32, \ + disable_emit, operand_offset,\ + error_buf, error_buf_size)) \ + goto fail; \ } while (0) +#define PUSH_F32() do { \ + if (!wasm_loader_push_frame_ref_offset(loader_ctx, VALUE_TYPE_F32, \ + disable_emit, operand_offset,\ + error_buf, error_buf_size)) \ + goto fail; \ + } while (0) + +#define PUSH_I64() do { \ + if (!wasm_loader_push_frame_ref_offset(loader_ctx, VALUE_TYPE_I64, \ + disable_emit, operand_offset,\ + error_buf, error_buf_size)) \ + goto fail; \ + } while (0) + +#define PUSH_F64() do { \ + if (!wasm_loader_push_frame_ref_offset(loader_ctx, VALUE_TYPE_F64, \ + disable_emit, operand_offset,\ + error_buf, error_buf_size)) \ + goto fail; \ + } while (0) + +#define POP_I32() do { \ + if (!wasm_loader_pop_frame_ref_offset(loader_ctx, VALUE_TYPE_I32, \ + error_buf, error_buf_size)) \ + goto fail; \ + } while (0) + +#define POP_F32() do { \ + if (!wasm_loader_pop_frame_ref_offset(loader_ctx, VALUE_TYPE_F32, \ + error_buf, error_buf_size)) \ + goto fail; \ + } while (0) + +#define POP_I64() do { \ + if (!wasm_loader_pop_frame_ref_offset(loader_ctx, VALUE_TYPE_I64, \ + error_buf, error_buf_size)) \ + goto fail; \ + } while (0) + +#define POP_F64() do { \ + if (!wasm_loader_pop_frame_ref_offset(loader_ctx, VALUE_TYPE_F64, \ + error_buf, error_buf_size)) \ + goto fail; \ + } while (0) + +#define PUSH_OFFSET_TYPE(type) do { \ + if (!(wasm_loader_push_frame_offset(loader_ctx, type, \ + disable_emit, operand_offset, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define POP_OFFSET_TYPE(type) do { \ + if (!(wasm_loader_pop_frame_offset(loader_ctx, type, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#else /* WASM_ENABLE_FAST_INTERP */ + +#define PUSH_I32() do { \ + if (!(wasm_loader_push_frame_ref(loader_ctx, VALUE_TYPE_I32, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define PUSH_F32() do { \ + if (!(wasm_loader_push_frame_ref(loader_ctx, VALUE_TYPE_F32, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define PUSH_I64() do { \ + if (!(wasm_loader_push_frame_ref(loader_ctx, VALUE_TYPE_I64, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define PUSH_F64() do { \ + if (!(wasm_loader_push_frame_ref(loader_ctx, VALUE_TYPE_F64, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define POP_I32() do { \ + if (!(wasm_loader_pop_frame_ref(loader_ctx, VALUE_TYPE_I32, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define POP_F32() do { \ + if (!(wasm_loader_pop_frame_ref(loader_ctx, VALUE_TYPE_F32, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define POP_I64() do { \ + if (!(wasm_loader_pop_frame_ref(loader_ctx, VALUE_TYPE_I64, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define POP_F64() do { \ + if (!(wasm_loader_pop_frame_ref(loader_ctx, VALUE_TYPE_F64, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#endif /* WASM_ENABLE_FAST_INTERP */ + +#define PUSH_TYPE(type) do { \ + if (!(wasm_loader_push_frame_ref(loader_ctx, type, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define POP_TYPE(type) do { \ + if (!(wasm_loader_pop_frame_ref(loader_ctx, type, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define PUSH_CSP(type, ret_type, _start_addr) do { \ + if (!wasm_loader_push_frame_csp(loader_ctx, type, ret_type, \ + _start_addr, error_buf, \ + error_buf_size)) \ + goto fail; \ + } while (0) + +#define POP_CSP() do { \ + if (!wasm_loader_pop_frame_csp(loader_ctx, \ + error_buf, error_buf_size)) \ + goto fail; \ + } while (0) + + #define GET_LOCAL_INDEX_TYPE_AND_OFFSET() do { \ read_leb_uint32(p, p_end, local_idx); \ if (local_idx >= param_count + local_count) { \ set_error_buf(error_buf, error_buf_size, \ "WASM module load failed: " \ - "local index out of range"); \ + "local index out of range"); \ goto fail; \ } \ local_type = local_idx < param_count \ @@ -2597,33 +3238,9 @@ pop_type(uint8 type, uint8 **p_frame_ref, uint32 *p_stack_cell_num, } while (0) #define CHECK_BR(depth) do { \ - if (csp_num < depth + 1) { \ - set_error_buf(error_buf, error_buf_size, \ - "WASM module load failed: type mismatch: " \ - "unexpected end of section or function"); \ - goto fail; \ - } \ - if ((frame_csp - (depth + 1))->block_type != BLOCK_TYPE_LOOP) { \ - uint8 tmp_ret_type = (frame_csp - (depth + 1))->return_type; \ - if ((tmp_ret_type == VALUE_TYPE_I32 \ - && (stack_cell_num < 1 || *(frame_ref - 1) != REF_I32)) \ - || (tmp_ret_type == VALUE_TYPE_F32 \ - && (stack_cell_num < 1 || *(frame_ref - 1) != REF_F32))\ - || (tmp_ret_type == VALUE_TYPE_I64 \ - && (stack_cell_num < 2 \ - || *(frame_ref - 2) != REF_I64_1 \ - || *(frame_ref - 1) != REF_I64_2)) \ - || (tmp_ret_type == VALUE_TYPE_F64 \ - && (stack_cell_num < 2 \ - || *(frame_ref - 2) != REF_F64_1 \ - || *(frame_ref - 1) != REF_F64_2))) { \ - set_error_buf(error_buf, error_buf_size, \ - "WASM module load failed: type mismatch: " \ - "expect data but stack was empty or other type"); \ + if (!wasm_loader_check_br(loader_ctx, depth, \ + error_buf, error_buf_size)) \ goto fail; \ - } \ - (frame_csp - (depth + 1))->is_block_reachable = true; \ - } \ } while (0) static bool @@ -2640,32 +3257,146 @@ check_memory(WASMModule *module, return true; } -#define CHECK_MEMORY() do { \ - if (!check_memory(module, error_buf, error_buf_size)) \ - goto fail; \ +#define CHECK_MEMORY() do { \ + if (!check_memory(module, error_buf, error_buf_size)) \ + goto fail; \ } while (0) +#if WASM_ENABLE_FAST_INTERP != 0 +#if WASM_ENABLE_ABS_LABEL_ADDR != 0 + +#define emit_label(opcode) do { \ + wasm_loader_emit_ptr(loader_ctx, handle_table[opcode]); \ + LOG_OP("\nemit_op [%02x]\t", opcode); \ + } while (0) + +#define skip_label() do { \ + wasm_loader_emit_backspace(loader_ctx, sizeof(void *)); \ + LOG_OP("\ndelete last op\n"); \ + } while (0) + +#else + +#define emit_label(opcode) do { \ + int32 offset = (int32)(handle_table[opcode] - handle_table[0]); \ + if (!(offset >= INT16_MIN && offset < INT16_MAX)) { \ + set_error_buf(error_buf, error_buf_size, \ + "WASM module load failed: " \ + "pre-compiled label offset out of range"); \ + goto fail; \ + } \ + wasm_loader_emit_int16(loader_ctx, offset); \ + LOG_OP("\nemit_op [%02x]\t", opcode); \ + } while (0) + +// drop local.get / const / block / loop / end +#define skip_label() do { \ + wasm_loader_emit_backspace(loader_ctx, sizeof(int16)); \ + LOG_OP("\ndelete last op\n"); \ + } while (0) + +#endif /* WASM_ENABLE_ABS_LABEL_ADDR */ + +#define emit_empty_label_addr_and_frame_ip(type) do { \ + if (!add_label_patch_to_list(loader_ctx->frame_csp - 1, type, \ + loader_ctx->p_code_compiled, \ + error_buf, error_buf_size)) \ + goto fail; \ + /* label address, to be patched */ \ + wasm_loader_emit_ptr(loader_ctx, NULL); \ + } while (0) + +#define emit_br_info(frame_csp) do { \ + if (!wasm_loader_emit_br_info(loader_ctx, frame_csp, \ + error_buf, error_buf_size)) \ + goto fail; \ + } while (0) + +#define LAST_OP_OUTPUT_I32() (last_op >= WASM_OP_I32_EQZ \ + && last_op <= WASM_OP_I32_ROTR) \ + || (last_op == WASM_OP_I32_LOAD \ + || last_op == WASM_OP_F32_LOAD) \ + || (last_op >= WASM_OP_I32_LOAD8_S \ + && last_op <= WASM_OP_I32_LOAD16_U) \ + || (last_op >= WASM_OP_F32_ABS \ + && last_op <= WASM_OP_F32_COPYSIGN) \ + || (last_op >= WASM_OP_I32_WRAP_I64 \ + && last_op <= WASM_OP_I32_TRUNC_U_F64) \ + || (last_op >= WASM_OP_F32_CONVERT_S_I32 \ + && last_op <= WASM_OP_F32_DEMOTE_F64) \ + || (last_op == WASM_OP_I32_REINTERPRET_F32) \ + || (last_op == WASM_OP_F32_REINTERPRET_I32) \ + || (last_op == EXT_OP_COPY_STACK_TOP) + +#define LAST_OP_OUTPUT_I64() (last_op >= WASM_OP_I64_CLZ \ + && last_op <= WASM_OP_I64_ROTR) \ + || (last_op >= WASM_OP_F64_ABS \ + && last_op <= WASM_OP_F64_COPYSIGN) \ + || (last_op == WASM_OP_I64_LOAD \ + || last_op == WASM_OP_F64_LOAD) \ + || (last_op >= WASM_OP_I64_LOAD8_S \ + && last_op <= WASM_OP_I64_LOAD32_U) \ + || (last_op >= WASM_OP_I64_EXTEND_S_I32 \ + && last_op <= WASM_OP_I64_TRUNC_U_F64) \ + || (last_op >= WASM_OP_F64_CONVERT_S_I32 \ + && last_op <= WASM_OP_F64_PROMOTE_F32) \ + || (last_op == WASM_OP_I64_REINTERPRET_F64) \ + || (last_op == WASM_OP_F64_REINTERPRET_I64) \ + || (last_op == EXT_OP_COPY_STACK_TOP_I64) + +#define GET_CONST_OFFSET(type, val) do { \ + if (!(wasm_loader_get_const_offset(loader_ctx, type, \ + val, 0, 0, &operand_offset, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define GET_CONST_F32_OFFSET(type, fval) do { \ + if (!(wasm_loader_get_const_offset(loader_ctx, type, \ + 0, fval, 0, &operand_offset, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#define GET_CONST_F64_OFFSET(type, fval) do { \ + if (!(wasm_loader_get_const_offset(loader_ctx, type, \ + 0, 0, fval, &operand_offset, \ + error_buf, error_buf_size))) \ + goto fail; \ + } while (0) + +#endif /* WASM_ENABLE_FAST_INTERP */ + static bool wasm_loader_prepare_bytecode(WASMModule *module, WASMFunction *func, BlockAddr *block_addr_cache, char *error_buf, uint32 error_buf_size) { uint8 *p = func->code, *p_end = func->code + func->code_size, *p_org; - uint8 *frame_ref_bottom = NULL, *frame_ref_boundary, *frame_ref; - BranchBlock *frame_csp_bottom = NULL, *frame_csp_boundary, *frame_csp; uint32 param_count, local_count, global_count; - uint32 max_stack_cell_num = 0, max_csp_num = 0; - uint32 stack_cell_num = 0, csp_num = 0; - uint32 frame_ref_size, frame_csp_size; uint8 *param_types, ret_type, *local_types, local_type, global_type; uint16 *local_offsets, local_offset; - uint32 count, i, local_idx, global_idx, depth, u32; + uint32 count, i, local_idx, global_idx, depth, u32, align, mem_offset; uint32 cache_index, item_index; int32 i32, i32_const = 0; int64 i64; uint8 opcode, u8, block_return_type; bool return_value = false, is_i32_const = false; BlockAddr *cache_items; + WASMLoaderContext *loader_ctx; +#if WASM_ENABLE_FAST_INTERP != 0 + uint8 *func_const_end, *func_const; + int16 operand_offset; + uint8 last_op = 0; + bool disable_emit; + float32 f32; + float64 f64; + + LOG_OP("\nProcessing func | [%d] params | [%d] locals | [%d] return\n", + func->param_cell_num, + func->local_cell_num, + func->ret_cell_num); +#endif global_count = module->import_global_count + module->global_count; @@ -2678,50 +3409,66 @@ wasm_loader_prepare_bytecode(WASMModule *module, WASMFunction *func, local_types = func->local_types; local_offsets = func->local_offsets; - frame_ref_size = 32; - if (!(frame_ref_bottom = frame_ref = wasm_malloc(frame_ref_size))) { - set_error_buf(error_buf, error_buf_size, - "WASM loader prepare bytecode failed: " - "allocate memory failed"); - goto fail; - } - memset(frame_ref_bottom, 0, frame_ref_size); - frame_ref_boundary = frame_ref_bottom + frame_ref_size; - - frame_csp_size = sizeof(BranchBlock) * 8; - if (!(frame_csp_bottom = frame_csp = wasm_malloc(frame_csp_size))) { + if (!(loader_ctx = wasm_loader_ctx_init(func))) { set_error_buf(error_buf, error_buf_size, "WASM loader prepare bytecode failed: " "allocate memory failed"); goto fail; } - memset(frame_csp_bottom, 0, frame_csp_size); - frame_csp_boundary = frame_csp_bottom + 8; +#if WASM_ENABLE_FAST_INTERP != 0 +re_scan: + if (loader_ctx->code_compiled_size > 0) { + if (!wasm_loader_ctx_reinit(loader_ctx)) { + set_error_buf(error_buf, error_buf_size, + "WASM loader prepare bytecode failed: " + "allocate memory failed"); + goto fail; + } + p = func->code; + func->code_compiled = loader_ctx->p_code_compiled; + } +#endif PUSH_CSP(BLOCK_TYPE_FUNCTION, ret_type, p); - (frame_csp - 1)->is_block_reachable = true; + (loader_ctx->frame_csp - 1)->is_block_reachable = true; while (p < p_end) { opcode = *p++; +#if WASM_ENABLE_FAST_INTERP != 0 + p_org = p; + disable_emit = false; + emit_label(opcode); +#endif switch (opcode) { case WASM_OP_UNREACHABLE: goto handle_next_reachable_block; case WASM_OP_NOP: +#if WASM_ENABLE_FAST_INTERP != 0 + skip_label(); +#endif break; case WASM_OP_BLOCK: /* 0x40/0x7F/0x7E/0x7D/0x7C */ block_return_type = read_uint8(p); PUSH_CSP(BLOCK_TYPE_BLOCK, block_return_type, p); +#if WASM_ENABLE_FAST_INTERP != 0 + skip_label(); +#endif break; case WASM_OP_LOOP: /* 0x40/0x7F/0x7E/0x7D/0x7C */ block_return_type = read_uint8(p); PUSH_CSP(BLOCK_TYPE_LOOP, block_return_type, p); +#if WASM_ENABLE_FAST_INTERP != 0 + skip_label(); + (loader_ctx->frame_csp - 1)->code_compiled = + loader_ctx->p_code_compiled; +#endif break; case WASM_OP_IF: @@ -2729,144 +3476,253 @@ wasm_loader_prepare_bytecode(WASMModule *module, WASMFunction *func, /* 0x40/0x7F/0x7E/0x7D/0x7C */ block_return_type = read_uint8(p); PUSH_CSP(BLOCK_TYPE_IF, block_return_type, p); +#if WASM_ENABLE_FAST_INTERP != 0 + emit_empty_label_addr_and_frame_ip(PATCH_ELSE); + emit_empty_label_addr_and_frame_ip(PATCH_END); +#endif if (!is_i32_const) - (frame_csp - 1)->is_block_reachable = true; + (loader_ctx->frame_csp - 1)->is_block_reachable = true; else { if (!i32_const) { - cache_index = ((uintptr_t)(frame_csp - 1)->start_addr) + cache_index = ((uintptr_t)(loader_ctx->frame_csp - 1)->start_addr) & (uintptr_t)(BLOCK_ADDR_CACHE_SIZE - 1); cache_items = block_addr_cache + BLOCK_ADDR_CONFLICT_SIZE * cache_index; for (item_index = 0; item_index < BLOCK_ADDR_CONFLICT_SIZE; item_index++) { if (cache_items[item_index].start_addr == - (frame_csp - 1)->start_addr) { - (frame_csp - 1)->else_addr = cache_items[item_index].else_addr; - (frame_csp - 1)->end_addr = cache_items[item_index].end_addr; + (loader_ctx->frame_csp - 1)->start_addr) { + (loader_ctx->frame_csp - 1)->else_addr = + cache_items[item_index].else_addr; + (loader_ctx->frame_csp - 1)->end_addr = + cache_items[item_index].end_addr; break; } } - if(item_index == BLOCK_ADDR_CONFLICT_SIZE - && !wasm_loader_find_block_addr(block_addr_cache, - (frame_csp - 1)->start_addr, + if (item_index == BLOCK_ADDR_CONFLICT_SIZE + && !wasm_loader_find_block_addr(block_addr_cache, + (loader_ctx->frame_csp - 1)->start_addr, p_end, - (frame_csp - 1)->block_type, - &(frame_csp - 1)->else_addr, - &(frame_csp - 1)->end_addr, + (loader_ctx->frame_csp - 1)->block_type, + &(loader_ctx->frame_csp - 1)->else_addr, + &(loader_ctx->frame_csp - 1)->end_addr, error_buf, error_buf_size)) goto fail; - if ((frame_csp - 1)->else_addr) - p = (frame_csp - 1)->else_addr; + if ((loader_ctx->frame_csp - 1)->else_addr) + p = (loader_ctx->frame_csp - 1)->else_addr; else - p = (frame_csp - 1)->end_addr; + p = (loader_ctx->frame_csp - 1)->end_addr; + + is_i32_const = false; + continue; } } break; case WASM_OP_ELSE: - if (csp_num < 2 - || (frame_csp - 1)->block_type != BLOCK_TYPE_IF) { + if (loader_ctx->csp_num < 2 + || (loader_ctx->frame_csp - 1)->block_type != BLOCK_TYPE_IF) { set_error_buf(error_buf, error_buf_size, "WASM loader prepare bytecode failed: " "opcode else found without matched opcode if"); goto fail; } - (frame_csp - 1)->else_addr = p - 1; - stack_cell_num = (frame_csp - 1)->stack_cell_num; - frame_ref = frame_ref_bottom + stack_cell_num; + (loader_ctx->frame_csp - 1)->else_addr = p - 1; + loader_ctx->stack_cell_num = (loader_ctx->frame_csp - 1)->stack_cell_num; + loader_ctx->frame_ref = loader_ctx->frame_ref_bottom + + loader_ctx->stack_cell_num; +#if WASM_ENABLE_FAST_INTERP != 0 + // if the result of if branch is in local or const area, add a copy op + if ((loader_ctx->frame_csp - 1)->return_type != VALUE_TYPE_VOID) { + uint8 return_cells; + if ((loader_ctx->frame_csp - 1)->return_type == VALUE_TYPE_I32 + || (loader_ctx->frame_csp - 1)->return_type == VALUE_TYPE_F32) + return_cells = 1; + else + return_cells = 2; + if ((loader_ctx->frame_csp - 1)->dynamic_offset != + *(loader_ctx->frame_offset - return_cells)) { + skip_label(); + if (return_cells == 1) + emit_label(EXT_OP_COPY_STACK_TOP); + else + emit_label(EXT_OP_COPY_STACK_TOP_I64); + emit_operand(loader_ctx, *(loader_ctx->frame_offset - return_cells)); + emit_operand(loader_ctx, (loader_ctx->frame_csp - 1)->dynamic_offset); + *(loader_ctx->frame_offset - return_cells) = + loader_ctx->frame_csp->dynamic_offset; + emit_label(opcode); + } + } + loader_ctx->frame_offset = loader_ctx->frame_offset_bottom + + loader_ctx->stack_cell_num; + emit_empty_label_addr_and_frame_ip(PATCH_END); + apply_label_patch(loader_ctx, 1, PATCH_ELSE, p); +#endif break; case WASM_OP_END: { POP_CSP(); - POP_TYPE(frame_csp->return_type); - PUSH_TYPE(frame_csp->return_type); + POP_TYPE(loader_ctx->frame_csp->return_type); + PUSH_TYPE(loader_ctx->frame_csp->return_type); - if (csp_num > 0) { - frame_csp->end_addr = p - 1; +#if WASM_ENABLE_FAST_INTERP != 0 + skip_label(); + // copy the result to the block return address + if (loader_ctx->frame_csp->return_type != VALUE_TYPE_VOID) { + uint8 return_cells; + if (loader_ctx->frame_csp->return_type == VALUE_TYPE_I32 + || loader_ctx->frame_csp->return_type == VALUE_TYPE_F32) + return_cells = 1; + else + return_cells = 2; + if (loader_ctx->frame_csp->dynamic_offset != + *(loader_ctx->frame_offset - return_cells)) { + if (return_cells == 1) + emit_label(EXT_OP_COPY_STACK_TOP); + else + emit_label(EXT_OP_COPY_STACK_TOP_I64); + emit_operand(loader_ctx, *(loader_ctx->frame_offset - return_cells)); + emit_operand(loader_ctx, loader_ctx->frame_csp->dynamic_offset); + } + // the frame_offset stack top should be the return address of the block + loader_ctx->frame_offset -= return_cells; + loader_ctx->dynamic_offset = loader_ctx->frame_csp->dynamic_offset; + PUSH_OFFSET_TYPE(loader_ctx->frame_csp->return_type); + wasm_loader_emit_backspace(loader_ctx, sizeof(int16)); + } + + apply_label_patch(loader_ctx, 0, PATCH_END, p); + free_label_patch_list(loader_ctx->frame_csp); + if (loader_ctx->frame_csp->block_type == BLOCK_TYPE_FUNCTION) { + emit_label(WASM_OP_RETURN); + POP_OFFSET_TYPE(loader_ctx->frame_csp->return_type); + } +#endif + if (loader_ctx->csp_num > 0) { + loader_ctx->frame_csp->end_addr = p - 1; } else { /* end of function block, function will return, ignore the following bytecodes */ p = p_end; + + is_i32_const = false; + continue; } break; } case WASM_OP_BR: { +#if WASM_ENABLE_FAST_INTERP != 0 + BranchBlock *frame_csp_tmp; +#endif read_leb_uint32(p, p_end, depth); CHECK_BR(depth); +#if WASM_ENABLE_FAST_INTERP != 0 + frame_csp_tmp = loader_ctx->frame_csp - depth - 1; + emit_br_info(frame_csp_tmp); +#endif + handle_next_reachable_block: - for (i = 1; i <= csp_num; i++) - if ((frame_csp - i)->is_block_reachable) + for (i = 1; i <= loader_ctx->csp_num; i++) + if ((loader_ctx->frame_csp - i)->is_block_reachable) break; - block_return_type = (frame_csp - i)->return_type; + block_return_type = (loader_ctx->frame_csp - i)->return_type; - cache_index = ((uintptr_t)(frame_csp - i)->start_addr) + cache_index = ((uintptr_t)(loader_ctx->frame_csp - i)->start_addr) & (uintptr_t)(BLOCK_ADDR_CACHE_SIZE - 1); cache_items = block_addr_cache + BLOCK_ADDR_CONFLICT_SIZE * cache_index; for (item_index = 0; item_index < BLOCK_ADDR_CONFLICT_SIZE; item_index++) { - if (cache_items[item_index].start_addr == (frame_csp - i)->start_addr) { - (frame_csp - i)->else_addr = cache_items[item_index].else_addr; - (frame_csp - i)->end_addr = cache_items[item_index].end_addr; + if (cache_items[item_index].start_addr == (loader_ctx->frame_csp - i)->start_addr) { + (loader_ctx->frame_csp - i)->else_addr = cache_items[item_index].else_addr; + (loader_ctx->frame_csp - i)->end_addr = cache_items[item_index].end_addr; break; } } if(item_index == BLOCK_ADDR_CONFLICT_SIZE && !wasm_loader_find_block_addr(block_addr_cache, - (frame_csp - i)->start_addr, + (loader_ctx->frame_csp - i)->start_addr, p_end, - (frame_csp - i)->block_type, - &(frame_csp - i)->else_addr, - &(frame_csp - i)->end_addr, + (loader_ctx->frame_csp - i)->block_type, + &(loader_ctx->frame_csp - i)->else_addr, + &(loader_ctx->frame_csp - i)->end_addr, error_buf, error_buf_size)) goto fail; - stack_cell_num = (frame_csp - i)->stack_cell_num; - frame_ref = frame_ref_bottom + stack_cell_num; - csp_num -= i - 1; - frame_csp -= i - 1; + loader_ctx->stack_cell_num = (loader_ctx->frame_csp - i)->stack_cell_num; + loader_ctx->frame_ref = loader_ctx->frame_ref_bottom + + loader_ctx->stack_cell_num; + loader_ctx->csp_num -= i - 1; + loader_ctx->frame_csp -= i - 1; - if ((frame_csp - 1)->block_type == BLOCK_TYPE_IF - && (frame_csp - 1)->else_addr != NULL - && p <= (frame_csp - 1)->else_addr) - p = (frame_csp - 1)->else_addr; + if ((loader_ctx->frame_csp - 1)->block_type == BLOCK_TYPE_IF + && (loader_ctx->frame_csp - 1)->else_addr != NULL + && p <= (loader_ctx->frame_csp - 1)->else_addr) + p = (loader_ctx->frame_csp - 1)->else_addr; else { - p = (frame_csp - 1)->end_addr; + p = (loader_ctx->frame_csp - 1)->end_addr; PUSH_TYPE(block_return_type); +#if WASM_ENABLE_FAST_INTERP != 0 + loader_ctx->frame_offset = loader_ctx->frame_offset_bottom + + loader_ctx->stack_cell_num; +#endif } - break; + is_i32_const = false; + continue; } case WASM_OP_BR_IF: + { +#if WASM_ENABLE_FAST_INTERP != 0 + BranchBlock *frame_csp_tmp; +#endif read_leb_uint32(p, p_end, depth); POP_I32(); CHECK_BR(depth); +#if WASM_ENABLE_FAST_INTERP != 0 + frame_csp_tmp = loader_ctx->frame_csp - depth - 1; + emit_br_info(frame_csp_tmp); +#endif if (!is_i32_const) - (frame_csp - (depth + 1))->is_block_reachable = true; + (loader_ctx->frame_csp - (depth + 1))->is_block_reachable = true; else { if (i32_const) goto handle_next_reachable_block; } break; + } case WASM_OP_BR_TABLE: { +#if WASM_ENABLE_FAST_INTERP != 0 + BranchBlock *frame_csp_tmp; +#endif + read_leb_uint32(p, p_end, count); +#if WASM_ENABLE_FAST_INTERP != 0 + emit_const(count); +#endif POP_I32(); /* TODO: check the const */ for (i = 0; i <= count; i++) { read_leb_uint32(p, p_end, depth); CHECK_BR(depth); +#if WASM_ENABLE_FAST_INTERP != 0 + frame_csp_tmp = loader_ctx->frame_csp - depth - 1; + emit_br_info(frame_csp_tmp); +#endif } + goto handle_next_reachable_block; } @@ -2875,39 +3731,49 @@ handle_next_reachable_block: POP_TYPE(ret_type); PUSH_TYPE(ret_type); - cache_index = ((uintptr_t)(frame_csp - 1)->start_addr) + cache_index = ((uintptr_t)(loader_ctx->frame_csp - 1)->start_addr) & (uintptr_t)(BLOCK_ADDR_CACHE_SIZE - 1); cache_items = block_addr_cache + BLOCK_ADDR_CONFLICT_SIZE * cache_index; for (item_index = 0; item_index < BLOCK_ADDR_CONFLICT_SIZE; item_index++) { if (cache_items[item_index].start_addr == - (frame_csp - 1)->start_addr) { - (frame_csp - 1)->else_addr = cache_items[item_index].else_addr; - (frame_csp - 1)->end_addr = cache_items[item_index].end_addr; + (loader_ctx->frame_csp - 1)->start_addr) { + (loader_ctx->frame_csp - 1)->else_addr = cache_items[item_index].else_addr; + (loader_ctx->frame_csp - 1)->end_addr = cache_items[item_index].end_addr; break; } } if(item_index == BLOCK_ADDR_CONFLICT_SIZE && !wasm_loader_find_block_addr(block_addr_cache, - (frame_csp - 1)->start_addr, + (loader_ctx->frame_csp - 1)->start_addr, p_end, - (frame_csp - 1)->block_type, - &(frame_csp - 1)->else_addr, - &(frame_csp - 1)->end_addr, + (loader_ctx->frame_csp - 1)->block_type, + &(loader_ctx->frame_csp - 1)->else_addr, + &(loader_ctx->frame_csp - 1)->end_addr, error_buf, error_buf_size)) goto fail; - stack_cell_num = (frame_csp - 1)->stack_cell_num; - frame_ref = frame_ref_bottom + stack_cell_num; - if ((frame_csp - 1)->block_type == BLOCK_TYPE_IF - && p <= (frame_csp - 1)->else_addr) { - p = (frame_csp - 1)->else_addr; + loader_ctx->stack_cell_num = (loader_ctx->frame_csp - 1)->stack_cell_num; + loader_ctx->frame_ref = loader_ctx->frame_ref_bottom + loader_ctx->stack_cell_num; + + if ((loader_ctx->frame_csp - 1)->block_type == BLOCK_TYPE_IF + && p <= (loader_ctx->frame_csp - 1)->else_addr) { + p = (loader_ctx->frame_csp - 1)->else_addr; } else { - p = (frame_csp - 1)->end_addr; - PUSH_TYPE((frame_csp - 1)->return_type); + p = (loader_ctx->frame_csp - 1)->end_addr; + PUSH_TYPE((loader_ctx->frame_csp - 1)->return_type); } - break; + +#if WASM_ENABLE_FAST_INTERP != 0 + // emit the offset after return opcode + POP_OFFSET_TYPE(ret_type); + loader_ctx->frame_offset = loader_ctx->frame_offset_bottom + + loader_ctx->stack_cell_num; +#endif + + is_i32_const = false; + continue; } case WASM_OP_CALL: @@ -2917,6 +3783,10 @@ handle_next_reachable_block: int32 idx; read_leb_uint32(p, p_end, func_idx); +#if WASM_ENABLE_FAST_INTERP != 0 + // we need to emit func_idx before arguments + emit_const(func_idx); +#endif if (func_idx >= module->import_function_count + module->function_count) { set_error_buf(error_buf, error_buf_size, @@ -2932,12 +3802,20 @@ handle_next_reachable_block: module->functions[func_idx - module->import_function_count]->func_type; if (func_type->param_count > 0) { - for (idx = (int32)(func_type->param_count - 1); idx >= 0; idx--) + for (idx = (int32)(func_type->param_count - 1); idx >= 0; idx--) { POP_TYPE(func_type->types[idx]); +#if WASM_ENABLE_FAST_INTERP != 0 + POP_OFFSET_TYPE(func_type->types[idx]); +#endif + } } - if (func_type->result_count) + if (func_type->result_count) { PUSH_TYPE(func_type->types[func_type->param_count]); +#if WASM_ENABLE_FAST_INTERP != 0 + PUSH_OFFSET_TYPE(func_type->types[func_type->param_count]); +#endif + } func->has_op_func_call = true; break; @@ -2958,6 +3836,10 @@ handle_next_reachable_block: } read_leb_uint32(p, p_end, type_idx); +#if WASM_ENABLE_FAST_INTERP != 0 + // we need to emit func_idx before arguments + emit_const(type_idx); +#endif /* reserved byte 0x00 */ if (*p++ != 0x00) { @@ -2979,12 +3861,20 @@ handle_next_reachable_block: func_type = module->types[type_idx]; if (func_type->param_count > 0) { - for (idx = (int32)(func_type->param_count - 1); idx >= 0; idx--) + for (idx = (int32)(func_type->param_count - 1); idx >= 0; idx--) { POP_TYPE(func_type->types[idx]); +#if WASM_ENABLE_FAST_INTERP != 0 + POP_OFFSET_TYPE(func_type->types[idx]); +#endif + } } - if (func_type->result_count > 0) + if (func_type->result_count > 0) { PUSH_TYPE(func_type->types[func_type->param_count]); +#if WASM_ENABLE_FAST_INTERP != 0 + PUSH_OFFSET_TYPE(func_type->types[func_type->param_count]); +#endif + } func->has_op_func_call = true; break; @@ -2992,28 +3882,44 @@ handle_next_reachable_block: case WASM_OP_DROP: { - if (stack_cell_num <= 0) { + if (loader_ctx->stack_cell_num <= 0) { set_error_buf(error_buf, error_buf_size, "WASM loader prepare bytecode failed: " "opcode drop was found but stack was empty"); goto fail; } - if (*(frame_ref - 1) == REF_I32 - || *(frame_ref - 1) == REF_F32) { - frame_ref--; - stack_cell_num--; + if (*(loader_ctx->frame_ref - 1) == REF_I32 + || *(loader_ctx->frame_ref - 1) == REF_F32) { + loader_ctx->frame_ref--; + loader_ctx->stack_cell_num--; +#if WASM_ENABLE_FAST_INTERP != 0 + skip_label(); + loader_ctx->frame_offset--; + if (*(loader_ctx->frame_offset) > + loader_ctx->start_dynamic_offset) + loader_ctx->dynamic_offset --; +#endif } else { - if (stack_cell_num <= 1) { + if (loader_ctx->stack_cell_num <= 1) { set_error_buf(error_buf, error_buf_size, "WASM loader prepare bytecode failed: " "opcode drop was found but stack was empty"); goto fail; } - frame_ref -= 2; - stack_cell_num -= 2; + loader_ctx->frame_ref -= 2; + loader_ctx->stack_cell_num -= 2; +#if WASM_ENABLE_FAST_INTERP == 0 *(p - 1) = WASM_OP_DROP_64; +#endif +#if WASM_ENABLE_FAST_INTERP != 0 + skip_label(); + loader_ctx->frame_offset -= 2; + if (*(loader_ctx->frame_offset) > + loader_ctx->start_dynamic_offset) + loader_ctx->dynamic_offset -= 2; +#endif } break; } @@ -3024,40 +3930,64 @@ handle_next_reachable_block: POP_I32(); - if (stack_cell_num <= 0) { + if (loader_ctx->stack_cell_num <= 0) { set_error_buf(error_buf, error_buf_size, "WASM loader prepare bytecode failed: " "opcode select was found but stack was empty"); goto fail; } - switch (*(frame_ref - 1)) { + switch (*(loader_ctx->frame_ref - 1)) { case REF_I32: case REF_F32: break; case REF_I64_2: case REF_F64_2: +#if WASM_ENABLE_FAST_INTERP == 0 *(p - 1) = WASM_OP_SELECT_64; +#endif +#if WASM_ENABLE_FAST_INTERP != 0 + if (loader_ctx->p_code_compiled) { +#if WASM_ENABLE_ABS_LABEL_ADDR != 0 + *(void**)(loader_ctx->p_code_compiled - 10) = + handle_table[WASM_OP_SELECT_64]; +#else + *((int16*)loader_ctx->p_code_compiled - 2) = (int16) + (handle_table[WASM_OP_SELECT_64] - handle_table[0]); +#endif + } +#endif break; } - ref_type = *(frame_ref - 1); + ref_type = *(loader_ctx->frame_ref - 1); POP_TYPE(ref_type); POP_TYPE(ref_type); PUSH_TYPE(ref_type); +#if WASM_ENABLE_FAST_INTERP != 0 + POP_OFFSET_TYPE(ref_type); + POP_OFFSET_TYPE(ref_type); + PUSH_OFFSET_TYPE(ref_type); +#endif break; } case WASM_OP_GET_LOCAL: { p_org = p - 1; - GET_LOCAL_INDEX_TYPE_AND_OFFSET(); PUSH_TYPE(local_type); +#if WASM_ENABLE_FAST_INTERP != 0 + /* Get Local is optimized out */ + skip_label(); + disable_emit = true; + operand_offset = local_offset; + PUSH_OFFSET_TYPE(local_type); +#else #if (WASM_ENABLE_WAMR_COMPILER == 0) && (WASM_ENABLE_JIT == 0) if (local_offset < 0x80) { - *p_org++ = WASM_OP_GET_LOCAL_FAST; + *p_org++ = EXT_OP_GET_LOCAL_FAST; if (local_type == VALUE_TYPE_I32 || local_type == VALUE_TYPE_F32) *p_org++ = (uint8)local_offset; @@ -3066,6 +3996,7 @@ handle_next_reachable_block: while (p_org < p) *p_org++ = WASM_OP_NOP; } +#endif #endif break; } @@ -3073,13 +4004,46 @@ handle_next_reachable_block: case WASM_OP_SET_LOCAL: { p_org = p - 1; - GET_LOCAL_INDEX_TYPE_AND_OFFSET(); POP_TYPE(local_type); +#if WASM_ENABLE_FAST_INTERP != 0 + if (local_offset < 256) { + skip_label(); + if (LAST_OP_OUTPUT_I32()) { + if (loader_ctx->p_code_compiled) + *(int16*)(loader_ctx->p_code_compiled - 2) = local_offset; + loader_ctx->frame_offset --; + loader_ctx->dynamic_offset --; + } + else if (LAST_OP_OUTPUT_I64()) { + if (loader_ctx->p_code_compiled) + *(int16*)(loader_ctx->p_code_compiled - 2) = local_offset; + loader_ctx->frame_offset -= 2; + loader_ctx->dynamic_offset -= 2; + } + else { + if (local_type == VALUE_TYPE_I32 + || local_type == VALUE_TYPE_F32) { + emit_label(EXT_OP_SET_LOCAL_FAST); + emit_byte(loader_ctx, local_offset); + } + else { + emit_label(EXT_OP_SET_LOCAL_FAST_I64); + emit_byte(loader_ctx, local_offset); + } + POP_OFFSET_TYPE(local_type); + } + } + else { /* local index larger than 255, reserve leb */ + p_org ++; + emit_leb(); + POP_OFFSET_TYPE(local_type); + } +#else #if (WASM_ENABLE_WAMR_COMPILER == 0) && (WASM_ENABLE_JIT == 0) if (local_offset < 0x80) { - *p_org++ = WASM_OP_SET_LOCAL_FAST; + *p_org++ = EXT_OP_SET_LOCAL_FAST; if (local_type == VALUE_TYPE_I32 || local_type == VALUE_TYPE_F32) *p_org++ = (uint8)local_offset; @@ -3089,21 +4053,40 @@ handle_next_reachable_block: *p_org++ = WASM_OP_NOP; } #endif - +#endif break; } case WASM_OP_TEE_LOCAL: { p_org = p - 1; - GET_LOCAL_INDEX_TYPE_AND_OFFSET(); POP_TYPE(local_type); PUSH_TYPE(local_type); +#if WASM_ENABLE_FAST_INTERP != 0 + if (local_offset < 256) { + skip_label(); + if (local_type == VALUE_TYPE_I32 + || local_type == VALUE_TYPE_F32) { + emit_label(EXT_OP_TEE_LOCAL_FAST); + emit_byte(loader_ctx, local_offset); + } + else { + emit_label(EXT_OP_TEE_LOCAL_FAST_I64); + emit_byte(loader_ctx, local_offset); + } + } + else { /* local index larger than 255, reserve leb */ + p_org ++; + emit_leb(); + } + emit_operand(loader_ctx, *(loader_ctx->frame_offset - + wasm_value_type_cell_num(local_type))); +#else #if (WASM_ENABLE_WAMR_COMPILER == 0) && (WASM_ENABLE_JIT == 0) if (local_offset < 0x80) { - *p_org++ = WASM_OP_TEE_LOCAL_FAST; + *p_org++ = EXT_OP_TEE_LOCAL_FAST; if (local_type == VALUE_TYPE_I32 || local_type == VALUE_TYPE_F32) *p_org++ = (uint8)local_offset; @@ -3113,7 +4096,7 @@ handle_next_reachable_block: *p_org++ = WASM_OP_NOP; } #endif - +#endif break; } @@ -3132,6 +4115,10 @@ handle_next_reachable_block: :module->globals[global_idx - module->import_global_count].type; PUSH_TYPE(global_type); +#if WASM_ENABLE_FAST_INTERP != 0 + emit_const(global_idx); + PUSH_OFFSET_TYPE(global_type); +#endif break; } @@ -3150,21 +4137,19 @@ handle_next_reachable_block: : module->globals[global_idx - module->import_global_count].type; POP_TYPE(global_type); +#if WASM_ENABLE_FAST_INTERP != 0 + emit_const(global_idx); + POP_OFFSET_TYPE(global_type); +#endif break; } + /* load */ case WASM_OP_I32_LOAD: case WASM_OP_I32_LOAD8_S: case WASM_OP_I32_LOAD8_U: case WASM_OP_I32_LOAD16_S: case WASM_OP_I32_LOAD16_U: - CHECK_MEMORY(); - read_leb_uint32(p, p_end, u32); /* align */ - read_leb_uint32(p, p_end, u32); /* offset */ - POP_I32(); - PUSH_I32(); - break; - case WASM_OP_I64_LOAD: case WASM_OP_I64_LOAD8_S: case WASM_OP_I64_LOAD8_U: @@ -3172,65 +4157,82 @@ handle_next_reachable_block: case WASM_OP_I64_LOAD16_U: case WASM_OP_I64_LOAD32_S: case WASM_OP_I64_LOAD32_U: - CHECK_MEMORY(); - read_leb_uint32(p, p_end, u32); /* align */ - read_leb_uint32(p, p_end, u32); /* offset */ - POP_I32(); - PUSH_I64(); - break; - case WASM_OP_F32_LOAD: - CHECK_MEMORY(); - read_leb_uint32(p, p_end, u32); /* align */ - read_leb_uint32(p, p_end, u32); /* offset */ - POP_I32(); - PUSH_F32(); - break; - case WASM_OP_F64_LOAD: - CHECK_MEMORY(); - read_leb_uint32(p, p_end, u32); /* align */ - read_leb_uint32(p, p_end, u32); /* offset */ - POP_I32(); - PUSH_F64(); - break; - + /* store */ case WASM_OP_I32_STORE: case WASM_OP_I32_STORE8: case WASM_OP_I32_STORE16: - CHECK_MEMORY(); - read_leb_uint32(p, p_end, u32); /* align */ - read_leb_uint32(p, p_end, u32); /* offset */ - POP_I32(); - POP_I32(); - break; - case WASM_OP_I64_STORE: case WASM_OP_I64_STORE8: case WASM_OP_I64_STORE16: case WASM_OP_I64_STORE32: - CHECK_MEMORY(); - read_leb_uint32(p, p_end, u32); /* align */ - read_leb_uint32(p, p_end, u32); /* offset */ - POP_I64(); - POP_I32(); - break; - case WASM_OP_F32_STORE: - CHECK_MEMORY(); - read_leb_uint32(p, p_end, u32); /* align */ - read_leb_uint32(p, p_end, u32); /* offset */ - POP_F32(); - POP_I32(); - break; - case WASM_OP_F64_STORE: + { CHECK_MEMORY(); - read_leb_uint32(p, p_end, u32); /* align */ - read_leb_uint32(p, p_end, u32); /* offset */ - POP_F64(); - POP_I32(); + read_leb_uint32(p, p_end, align); /* align */ + read_leb_uint32(p, p_end, mem_offset); /* offset */ +#if WASM_ENABLE_FAST_INTERP != 0 + emit_byte(loader_ctx, opcode); + emit_const(mem_offset); +#endif + switch (opcode) + { + /* load */ + case WASM_OP_I32_LOAD: + case WASM_OP_I32_LOAD8_S: + case WASM_OP_I32_LOAD8_U: + case WASM_OP_I32_LOAD16_S: + case WASM_OP_I32_LOAD16_U: + POP_I32(); + PUSH_I32(); + break; + case WASM_OP_I64_LOAD: + case WASM_OP_I64_LOAD8_S: + case WASM_OP_I64_LOAD8_U: + case WASM_OP_I64_LOAD16_S: + case WASM_OP_I64_LOAD16_U: + case WASM_OP_I64_LOAD32_S: + case WASM_OP_I64_LOAD32_U: + POP_I32(); + PUSH_I64(); + break; + case WASM_OP_F32_LOAD: + POP_I32(); + PUSH_F32(); + break; + case WASM_OP_F64_LOAD: + POP_I32(); + PUSH_F64(); + break; + /* store */ + case WASM_OP_I32_STORE: + case WASM_OP_I32_STORE8: + case WASM_OP_I32_STORE16: + POP_I32(); + POP_I32(); + break; + case WASM_OP_I64_STORE: + case WASM_OP_I64_STORE8: + case WASM_OP_I64_STORE16: + case WASM_OP_I64_STORE32: + POP_I64(); + POP_I32(); + break; + case WASM_OP_F32_STORE: + POP_F32(); + POP_I32(); + break; + case WASM_OP_F64_STORE: + POP_F64(); + POP_I32(); + break; + default: + break; + } break; + } case WASM_OP_MEMORY_SIZE: CHECK_MEMORY(); @@ -3264,21 +4266,43 @@ handle_next_reachable_block: read_leb_int32(p, p_end, i32_const); /* Currently we only track simple I32_CONST opcode. */ is_i32_const = true; +#if WASM_ENABLE_FAST_INTERP != 0 + skip_label(); + disable_emit = true; + GET_CONST_OFFSET(VALUE_TYPE_I32, i32_const); +#endif PUSH_I32(); break; case WASM_OP_I64_CONST: read_leb_int64(p, p_end, i64); +#if WASM_ENABLE_FAST_INTERP != 0 + skip_label(); + disable_emit = true; + GET_CONST_OFFSET(VALUE_TYPE_I64, i64); +#endif PUSH_I64(); break; case WASM_OP_F32_CONST: p += sizeof(float32); +#if WASM_ENABLE_FAST_INTERP != 0 + skip_label(); + disable_emit = true; + f32 = *(float32 *)p_org; + GET_CONST_F32_OFFSET(VALUE_TYPE_F32, f32); +#endif PUSH_F32(); break; case WASM_OP_F64_CONST: p += sizeof(float64); +#if WASM_ENABLE_FAST_INTERP != 0 + skip_label(); + disable_emit = true; + f64 = *(float64 *)p_org; + GET_CONST_F64_OFFSET(VALUE_TYPE_F64, f64); +#endif PUSH_F64(); break; @@ -3545,24 +4569,56 @@ handle_next_reachable_block: if (opcode != WASM_OP_I32_CONST) is_i32_const = false; + +#if WASM_ENABLE_FAST_INTERP != 0 + last_op = opcode; +#endif } - if (csp_num > 0) { + if (loader_ctx->csp_num > 0) { set_error_buf(error_buf, error_buf_size, "WASM module load failed: " "function body must end with END opcode."); goto fail; } - func->max_stack_cell_num = max_stack_cell_num; - func->max_block_num = max_csp_num; +#if WASM_ENABLE_FAST_INTERP != 0 + if (loader_ctx->p_code_compiled == NULL) + goto re_scan; + + func->const_cell_num = loader_ctx->const_cell_num; + if (!(func->consts = func_const = wasm_malloc(func->const_cell_num * 4))) { + set_error_buf(error_buf, error_buf_size, + "WASM loader prepare bytecode failed: " + "allocate memory failed"); + goto fail; + } + memset(func->consts, 0, func->const_cell_num * 4); + func_const_end = func->consts + func->const_cell_num * 4; + // reverse the const buf + for (int i = loader_ctx->num_const - 1; i >= 0; i--) { + Const *c = (Const*)(loader_ctx->const_buf + i * sizeof(Const)); + if (c->value_type == VALUE_TYPE_F64 + || c->value_type == VALUE_TYPE_I64) { + bh_memcpy_s(func_const, func_const_end - func_const, + &c->value.f64, sizeof(int64)); + func_const += sizeof(int64); + } else { + *(uint32*)func_const = c->value.i32; + func_const += sizeof(int32); + } + } + + func->max_stack_cell_num = loader_ctx->max_dynamic_offset - + loader_ctx->start_dynamic_offset + 1; +#else + func->max_stack_cell_num = loader_ctx->max_stack_cell_num; +#endif + func->max_block_num = loader_ctx->max_csp_num; return_value = true; fail: - if (frame_ref_bottom) - wasm_free(frame_ref_bottom); - if (frame_csp_bottom) - wasm_free(frame_csp_bottom); + wasm_loader_ctx_destroy(loader_ctx); (void)u8; (void)u32; @@ -3570,5 +4626,7 @@ fail: (void)i64; (void)local_offset; (void)p_org; + (void)mem_offset; + (void)align; return return_value; } diff --git a/core/iwasm/interpreter/wasm_opcode.h b/core/iwasm/interpreter/wasm_opcode.h index e96dc7869..e99db696a 100644 --- a/core/iwasm/interpreter/wasm_opcode.h +++ b/core/iwasm/interpreter/wasm_opcode.h @@ -238,13 +238,19 @@ typedef enum WASMOpcode { WASM_OP_F64_REINTERPRET_I64 = 0xbf, /* f64.reinterpret/i64 */ /* drop/select specified types*/ - WASM_OP_DROP_64 = 0xc0, - WASM_OP_SELECT_64 = 0xc1, - WASM_OP_GET_LOCAL_FAST = 0xc2, - WASM_OP_SET_LOCAL_FAST = 0xc3, - WASM_OP_TEE_LOCAL_FAST = 0xc4, + WASM_OP_DROP_64 = 0xc0, + WASM_OP_SELECT_64 = 0xc1, - WASM_OP_IMPDEP = 0xc5 + /* extend op code */ + EXT_OP_GET_LOCAL_FAST = 0xc2, + EXT_OP_SET_LOCAL_FAST_I64 = 0xc3, + EXT_OP_SET_LOCAL_FAST = 0xc4, + EXT_OP_TEE_LOCAL_FAST = 0xc5, + EXT_OP_TEE_LOCAL_FAST_I64 = 0xc6, + EXT_OP_COPY_STACK_TOP = 0xc7, + EXT_OP_COPY_STACK_TOP_I64 = 0xc8, + + WASM_OP_IMPDEP = 0xc9 } WASMOpcode; #ifdef __cplusplus @@ -450,12 +456,16 @@ static const void *_name[WASM_INSTRUCTION_NUM] = { \ HANDLE_OPCODE (WASM_OP_I64_REINTERPRET_F64), /* 0xbd */ \ HANDLE_OPCODE (WASM_OP_F32_REINTERPRET_I32), /* 0xbe */ \ HANDLE_OPCODE (WASM_OP_F64_REINTERPRET_I64), /* 0xbf */ \ - HANDLE_OPCODE (WASM_OP_DROP_64), /* 0xc0 */ \ - HANDLE_OPCODE (WASM_OP_SELECT_64), /* 0xc1 */ \ - HANDLE_OPCODE (WASM_OP_GET_LOCAL_FAST),/* 0xc2 */ \ - HANDLE_OPCODE (WASM_OP_SET_LOCAL_FAST),/* 0xc3 */ \ - HANDLE_OPCODE (WASM_OP_TEE_LOCAL_FAST),/* 0xc4 */ \ - HANDLE_OPCODE (WASM_OP_IMPDEP), /* 0xc5 */ \ + HANDLE_OPCODE (WASM_OP_DROP_64), /* 0xc0 */ \ + HANDLE_OPCODE (WASM_OP_SELECT_64), /* 0xc1 */ \ + HANDLE_OPCODE (EXT_OP_GET_LOCAL_FAST), /* 0xc2 */ \ + HANDLE_OPCODE (EXT_OP_SET_LOCAL_FAST_I64), /* 0xc3 */ \ + HANDLE_OPCODE (EXT_OP_SET_LOCAL_FAST), /* 0xc4 */ \ + HANDLE_OPCODE (EXT_OP_TEE_LOCAL_FAST), /* 0xc5 */ \ + HANDLE_OPCODE (EXT_OP_TEE_LOCAL_FAST_I64), /* 0xc6 */ \ + HANDLE_OPCODE (EXT_OP_COPY_STACK_TOP), /* 0xc7 */ \ + HANDLE_OPCODE (EXT_OP_COPY_STACK_TOP_I64), /* 0xc8 */ \ + HANDLE_OPCODE (WASM_OP_IMPDEP), /* 0xc9 */ \ } #endif /* end of _WASM_OPCODE_H */ diff --git a/core/iwasm/interpreter/wasm_runtime.c b/core/iwasm/interpreter/wasm_runtime.c index 6f56cce69..5b97f15dc 100644 --- a/core/iwasm/interpreter/wasm_runtime.c +++ b/core/iwasm/interpreter/wasm_runtime.c @@ -355,6 +355,10 @@ functions_instantiate(const WASMModule *module, function->local_offsets = function->u.func->local_offsets; +#if WASM_ENABLE_FAST_INTERP != 0 + function->const_cell_num = function->u.func->const_cell_num; +#endif + function++; } diff --git a/core/iwasm/interpreter/wasm_runtime.h b/core/iwasm/interpreter/wasm_runtime.h index aa025fcc6..bf172b284 100644 --- a/core/iwasm/interpreter/wasm_runtime.h +++ b/core/iwasm/interpreter/wasm_runtime.h @@ -87,6 +87,10 @@ typedef struct WASMFunctionInstance { uint16 ret_cell_num; /* cell num of local variables, 0 for import function */ uint16 local_cell_num; +#if WASM_ENABLE_FAST_INTERP != 0 + /* cell num of consts */ + uint16 const_cell_num; +#endif uint16 *local_offsets; /* parameter types */ uint8 *param_types; @@ -165,7 +169,11 @@ typedef struct WASMInterpFrame WASMRuntimeFrame; static inline uint8* wasm_get_func_code(WASMFunctionInstance *func) { +#if WASM_ENABLE_FAST_INTERP == 0 return func->is_import_func ? NULL : func->u.func->code; +#else + return func->is_import_func ? NULL : func->u.func->code_compiled; +#endif } /** @@ -178,8 +186,13 @@ wasm_get_func_code(WASMFunctionInstance *func) static inline uint8* wasm_get_func_code_end(WASMFunctionInstance *func) { +#if WASM_ENABLE_FAST_INTERP == 0 return func->is_import_func ? NULL : func->u.func->code + func->u.func->code_size; +#else + return func->is_import_func + ? NULL : func->u.func->code_compiled + func->u.func->code_compiled_size; +#endif } WASMModule * diff --git a/core/shared/include/config.h b/core/shared/include/config.h index bb98980ee..66bb55f57 100644 --- a/core/shared/include/config.h +++ b/core/shared/include/config.h @@ -98,6 +98,13 @@ enum { /* WASM Interpreter labels-as-values feature */ #define WASM_ENABLE_LABELS_AS_VALUES 1 +#if WASM_ENABLE_FAST_INTERP != 0 +#define WASM_ENABLE_ABS_LABEL_ADDR 1 +#define WASM_DEBUG_PREPROCESSOR 0 +#else +#define WASM_ENABLE_ABS_LABEL_ADDR 0 +#endif + /* Heap and stack profiling */ #define BEIHAI_ENABLE_MEMORY_PROFILING 0 diff --git a/core/shared/platform/alios/bh_platform.c b/core/shared/platform/alios/bh_platform.c index 9d3a1f6bd..91a2af0cb 100644 --- a/core/shared/platform/alios/bh_platform.c +++ b/core/shared/platform/alios/bh_platform.c @@ -9,20 +9,8 @@ #include #include -char *bh_strdup(const char *s) -{ - uint32 size; - char *s1 = NULL; - - if (s) { - size = (uint32)(strlen(s) + 1); - if ((s1 = bh_malloc(size))) - bh_memcpy_s(s1, size, s, size); - } - return s1; -} - -int bh_platform_init() +int +bh_platform_init() { return 0; } diff --git a/core/shared/platform/android/bh_definition.c b/core/shared/platform/android/bh_definition.c deleted file mode 100644 index 8fb58d2f8..000000000 --- a/core/shared/platform/android/bh_definition.c +++ /dev/null @@ -1,67 +0,0 @@ -/* - * Copyright (C) 2019 Intel Corporation. All rights reserved. - * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception - */ - -#include "bh_platform.h" - -#define RSIZE_MAX 0x7FFFFFFF - -int b_memcpy_s(void * s1, unsigned int s1max, const void * s2, unsigned int n) -{ - char *dest = (char*) s1; - char *src = (char*) s2; - if (n == 0) { - return 0; - } - - if (s1 == NULL || s1max > RSIZE_MAX) { - return -1; - } - if (s2 == NULL || n > s1max) { - memset(dest, 0, s1max); - return -1; - } - memcpy(dest, src, n); - return 0; -} - -int b_strcat_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s1) + strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strcat(s1, s2); - - return 0; -} - -int b_strcpy_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strcpy(s1, s2); - - return 0; -} - -int fopen_s(FILE ** pFile, const char *filename, const char *mode) -{ - if (NULL == pFile || NULL == filename || NULL == mode) { - return -1; - } - - *pFile = fopen(filename, mode); - - if (NULL == *pFile) - return -1; - - return 0; -} diff --git a/core/shared/platform/android/bh_platform.c b/core/shared/platform/android/bh_platform.c index 4eda405b0..44e28ab07 100755 --- a/core/shared/platform/android/bh_platform.c +++ b/core/shared/platform/android/bh_platform.c @@ -12,20 +12,8 @@ #include #include -char *bh_strdup(const char *s) -{ - uint32 size; - char *s1 = NULL; - - if (s) { - size = (uint32)(strlen(s) + 1); - if ((s1 = bh_malloc(size))) - bh_memcpy_s(s1, size, s, size); - } - return s1; -} - -int bh_platform_init() +int +bh_platform_init() { return 0; } diff --git a/core/shared/platform/android/bh_platform.h b/core/shared/platform/android/bh_platform.h index fa221aa2e..b449616e0 100644 --- a/core/shared/platform/android/bh_platform.h +++ b/core/shared/platform/android/bh_platform.h @@ -104,8 +104,6 @@ int b_memcpy_s(void * s1, unsigned int s1max, const void * s2, int b_strcat_s(char * s1, size_t s1max, const char * s2); int b_strcpy_s(char * s1, size_t s1max, const char * s2); -int fopen_s(FILE ** pFile, const char *filename, const char *mode); - char *bh_read_file_to_buffer(const char *filename, uint32 *ret_size); char *bh_strdup(const char *s); diff --git a/core/shared/platform/darwin/bh_definition.c b/core/shared/platform/darwin/bh_definition.c deleted file mode 100644 index f093cc9c5..000000000 --- a/core/shared/platform/darwin/bh_definition.c +++ /dev/null @@ -1,71 +0,0 @@ -/* - * Copyright (C) 2019 Intel Corporation. All rights reserved. - * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception - */ - -#include "bh_platform.h" - -#ifdef RSIZE_MAX -#undef RSIZE_MAX -#endif - -#define RSIZE_MAX 0x7FFFFFFF - -int b_memcpy_s(void * s1, unsigned int s1max, const void * s2, unsigned int n) -{ - char *dest = (char*) s1; - char *src = (char*) s2; - if (n == 0) { - return 0; - } - - if (s1 == NULL || s1max > RSIZE_MAX) { - return -1; - } - if (s2 == NULL || n > s1max) { - memset(dest, 0, s1max); - return -1; - } - memcpy(dest, src, n); - return 0; -} - -int b_strcat_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s1) + strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strcat(s1, s2); - - return 0; -} - -int b_strcpy_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strcpy(s1, s2); - - return 0; -} - -int fopen_s(FILE ** pFile, const char *filename, const char *mode) -{ - if (NULL == pFile || NULL == filename || NULL == mode) { - return -1; - } - - *pFile = fopen(filename, mode); - - if (NULL == *pFile) - return -1; - - return 0; -} diff --git a/core/shared/platform/darwin/bh_platform.c b/core/shared/platform/darwin/bh_platform.c index 8a660b6d3..b6e251145 100755 --- a/core/shared/platform/darwin/bh_platform.c +++ b/core/shared/platform/darwin/bh_platform.c @@ -12,20 +12,8 @@ #include #include -char *bh_strdup(const char *s) -{ - uint32 size; - char *s1 = NULL; - - if (s) { - size = (uint32)(strlen(s) + 1); - if ((s1 = bh_malloc(size))) - bh_memcpy_s(s1, size, s, size); - } - return s1; -} - -int bh_platform_init() +int +bh_platform_init() { return 0; } diff --git a/core/shared/platform/darwin/bh_platform.h b/core/shared/platform/darwin/bh_platform.h index 11a577bbc..7deaec68c 100644 --- a/core/shared/platform/darwin/bh_platform.h +++ b/core/shared/platform/darwin/bh_platform.h @@ -101,8 +101,6 @@ int b_memcpy_s(void * s1, unsigned int s1max, const void * s2, int b_strcat_s(char * s1, size_t s1max, const char * s2); int b_strcpy_s(char * s1, size_t s1max, const char * s2); -int fopen_s(FILE ** pFile, const char *filename, const char *mode); - char *bh_read_file_to_buffer(const char *filename, uint32 *ret_size); char *bh_strdup(const char *s); diff --git a/core/shared/platform/linux-sgx/bh_definition.c b/core/shared/platform/linux-sgx/bh_definition.c deleted file mode 100644 index 2b69af0ca..000000000 --- a/core/shared/platform/linux-sgx/bh_definition.c +++ /dev/null @@ -1,54 +0,0 @@ -/* - * Copyright (C) 2019 Intel Corporation. All rights reserved. - * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception - */ - -#include "bh_platform.h" - -#define RSIZE_MAX 0x7FFFFFFF - -int b_memcpy_s(void * s1, unsigned int s1max, const void * s2, unsigned int n) -{ - char *dest = (char*) s1; - char *src = (char*) s2; - if (n == 0) { - return 0; - } - - if (s1 == NULL || s1max > RSIZE_MAX) { - return -1; - } - if (s2 == NULL || n > s1max) { - memset(dest, 0, s1max); - return -1; - } - memcpy(dest, src, n); - return 0; -} - -int b_strcat_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s1) + strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strncat(s1, s2, strlen(s2)); - - return 0; -} - -int b_strcpy_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strncpy(s1, s2, s1max); - - return 0; -} - diff --git a/core/shared/platform/linux-sgx/bh_platform.c b/core/shared/platform/linux-sgx/bh_platform.c index 5b66a13d0..9257b6a50 100644 --- a/core/shared/platform/linux-sgx/bh_platform.c +++ b/core/shared/platform/linux-sgx/bh_platform.c @@ -14,19 +14,6 @@ #define FIXED_BUFFER_SIZE (1<<9) static bh_print_function_t print_function = NULL; -char *bh_strdup(const char *s) -{ - uint32 size; - char *s1 = NULL; - - if (s) { - size = (uint32)(strlen(s) + 1); - if ((s1 = bh_malloc(size))) - bh_memcpy_s(s1, size, s, size); - } - return s1; -} - int bh_platform_init() { return 0; diff --git a/core/shared/platform/linux/bh_definition.c b/core/shared/platform/linux/bh_definition.c deleted file mode 100644 index 8fb58d2f8..000000000 --- a/core/shared/platform/linux/bh_definition.c +++ /dev/null @@ -1,67 +0,0 @@ -/* - * Copyright (C) 2019 Intel Corporation. All rights reserved. - * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception - */ - -#include "bh_platform.h" - -#define RSIZE_MAX 0x7FFFFFFF - -int b_memcpy_s(void * s1, unsigned int s1max, const void * s2, unsigned int n) -{ - char *dest = (char*) s1; - char *src = (char*) s2; - if (n == 0) { - return 0; - } - - if (s1 == NULL || s1max > RSIZE_MAX) { - return -1; - } - if (s2 == NULL || n > s1max) { - memset(dest, 0, s1max); - return -1; - } - memcpy(dest, src, n); - return 0; -} - -int b_strcat_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s1) + strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strcat(s1, s2); - - return 0; -} - -int b_strcpy_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strcpy(s1, s2); - - return 0; -} - -int fopen_s(FILE ** pFile, const char *filename, const char *mode) -{ - if (NULL == pFile || NULL == filename || NULL == mode) { - return -1; - } - - *pFile = fopen(filename, mode); - - if (NULL == *pFile) - return -1; - - return 0; -} diff --git a/core/shared/platform/linux/bh_platform.c b/core/shared/platform/linux/bh_platform.c index 4eda405b0..44e28ab07 100755 --- a/core/shared/platform/linux/bh_platform.c +++ b/core/shared/platform/linux/bh_platform.c @@ -12,20 +12,8 @@ #include #include -char *bh_strdup(const char *s) -{ - uint32 size; - char *s1 = NULL; - - if (s) { - size = (uint32)(strlen(s) + 1); - if ((s1 = bh_malloc(size))) - bh_memcpy_s(s1, size, s, size); - } - return s1; -} - -int bh_platform_init() +int +bh_platform_init() { return 0; } diff --git a/core/shared/platform/linux/bh_platform.h b/core/shared/platform/linux/bh_platform.h index 2fc4e1a30..dd7483668 100644 --- a/core/shared/platform/linux/bh_platform.h +++ b/core/shared/platform/linux/bh_platform.h @@ -102,8 +102,6 @@ int b_memcpy_s(void * s1, unsigned int s1max, const void * s2, int b_strcat_s(char * s1, size_t s1max, const char * s2); int b_strcpy_s(char * s1, size_t s1max, const char * s2); -int fopen_s(FILE ** pFile, const char *filename, const char *mode); - char *bh_read_file_to_buffer(const char *filename, uint32 *ret_size); char *bh_strdup(const char *s); diff --git a/core/shared/platform/vxworks/bh_definition.c b/core/shared/platform/vxworks/bh_definition.c deleted file mode 100644 index 4cdf98259..000000000 --- a/core/shared/platform/vxworks/bh_definition.c +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Copyright (C) 2019 Intel Corporation. All rights reserved. - * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception - */ - -#include "bh_platform.h" - -#define RSIZE_MAX 0x7FFFFFFF - -int b_memcpy_s(void * s1, unsigned int s1max, const void * s2, unsigned int n) -{ - char *dest = (char*) s1; - char *src = (char*) s2; - if (n == 0) { - return 0; - } - - if (s1 == NULL || s1max > RSIZE_MAX) { - return -1; - } - if (s2 == NULL || n > s1max) { - memset(dest, 0, s1max); - return -1; - } - memcpy(dest, src, n); - return 0; -} - -int b_strcat_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s1) + strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strcat(s1, s2); - - return 0; -} - -int b_strcpy_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strcpy(s1, s2); - - return 0; -} diff --git a/core/shared/platform/vxworks/bh_platform.c b/core/shared/platform/vxworks/bh_platform.c index ea5666b30..c2960485d 100644 --- a/core/shared/platform/vxworks/bh_platform.c +++ b/core/shared/platform/vxworks/bh_platform.c @@ -16,21 +16,8 @@ #include #include - -char *bh_strdup(const char *s) -{ - uint32 size; - char *s1 = NULL; - - if (s) { - size = (uint32)(strlen(s) + 1); - if ((s1 = bh_malloc(size))) - bh_memcpy_s(s1, size, s, size); - } - return s1; -} - -int bh_platform_init() +int +bh_platform_init() { return 0; } diff --git a/core/shared/platform/zephyr/bh_definition.c b/core/shared/platform/zephyr/bh_definition.c deleted file mode 100644 index aae3ed823..000000000 --- a/core/shared/platform/zephyr/bh_definition.c +++ /dev/null @@ -1,53 +0,0 @@ -/* - * Copyright (C) 2019 Intel Corporation. All rights reserved. - * SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception - */ - -#include "bh_platform.h" - -#define RSIZE_MAX 0x7FFFFFFF - -int b_memcpy_s(void * s1, unsigned int s1max, const void * s2, unsigned int n) -{ - char *dest = (char*) s1; - char *src = (char*) s2; - if (n == 0) { - return 0; - } - - if (s1 == NULL || s1max > RSIZE_MAX) { - return -1; - } - if (s2 == NULL || n > s1max) { - memset(dest, 0, s1max); - return -1; - } - memcpy(dest, src, n); - return 0; -} - -int b_strcat_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1 || NULL == s2 - || s1max < (strlen(s1) + strlen(s2) + 1) - || s1max > RSIZE_MAX) { - return -1; - } - - strcat(s1, s2); - - return 0; -} - -int b_strcpy_s(char * s1, size_t s1max, const char * s2) -{ - if (NULL == s1|| NULL == s2 - || s1max < (strlen(s2) + 1) || s1max > RSIZE_MAX) { - return -1; - } - - strcpy(s1, s2); - - return 0; -} - diff --git a/core/shared/platform/zephyr/bh_platform.c b/core/shared/platform/zephyr/bh_platform.c index f2bc2e380..a96483da7 100755 --- a/core/shared/platform/zephyr/bh_platform.c +++ b/core/shared/platform/zephyr/bh_platform.c @@ -12,19 +12,6 @@ #include #endif -char *bh_strdup(const char *s) -{ - uint32 size; - char *s1 = NULL; - - if (s) { - size = (uint32)(strlen(s) + 1); - if ((s1 = bh_malloc(size))) - bh_memcpy_s(s1, size, s, size); - } - return s1; -} - #ifdef CONFIG_ARM_MPU /** * This function will allow execute from sram region. @@ -54,7 +41,8 @@ _stdout_hook_iwasm(int c) return 1; } -int bh_platform_init() +int +bh_platform_init() { extern void __stdout_hook_install(int (*hook)(int)); /* Enable printf() in Zephyr */ diff --git a/core/shared/platform/alios/bh_definition.c b/core/shared/utils/bh_definition.c similarity index 55% rename from core/shared/platform/alios/bh_definition.c rename to core/shared/utils/bh_definition.c index 1bd85b125..b417a764d 100644 --- a/core/shared/platform/alios/bh_definition.c +++ b/core/shared/utils/bh_definition.c @@ -4,11 +4,17 @@ */ #include "bh_platform.h" +#include "bh_common.h" + +#ifdef RSIZE_MAX +#undef RSIZE_MAX +#endif #define RSIZE_MAX 0x7FFFFFFF -int b_memcpy_s(void * s1, unsigned int s1max, - const void * s2, unsigned int n) +int +b_memcpy_s(void * s1, unsigned int s1max, + const void * s2, unsigned int n) { char *dest = (char*)s1; char *src = (char*)s2; @@ -27,7 +33,8 @@ int b_memcpy_s(void * s1, unsigned int s1max, return 0; } -int b_strcat_s(char * s1, size_t s1max, const char * s2) +int +b_strcat_s(char * s1, size_t s1max, const char * s2) { if (NULL == s1 || NULL == s2 || s1max < (strlen(s1) + strlen(s2) + 1) @@ -35,11 +42,12 @@ int b_strcat_s(char * s1, size_t s1max, const char * s2) return -1; } - strcat(s1, s2); + memcpy(s1 + strlen(s1), s2, strlen(s2) + 1); return 0; } -int b_strcpy_s(char * s1, size_t s1max, const char * s2) +int +b_strcpy_s(char * s1, size_t s1max, const char * s2) { if (NULL == s1 || NULL == s2 || s1max < (strlen(s2) + 1) @@ -47,7 +55,21 @@ int b_strcpy_s(char * s1, size_t s1max, const char * s2) return -1; } - strcpy(s1, s2); + memcpy(s1, s2, strlen(s2) + 1); return 0; } +char * +bh_strdup(const char *s) +{ + uint32 size; + char *s1 = NULL; + + if (s) { + size = (uint32)(strlen(s) + 1); + if ((s1 = bh_malloc(size))) + bh_memcpy_s(s1, size, s, size); + } + return s1; +} + diff --git a/doc/build_wamr.md b/doc/build_wamr.md index f5c3e48fc..2f845fa50 100644 --- a/doc/build_wamr.md +++ b/doc/build_wamr.md @@ -1,12 +1,72 @@ Build WAMR core (iwasm) ========================= -Please follow the instructions below to build the WAMR VM core on different platforms. +It is recommended to use the [WAMR SDK](../wamr-sdk) tools to build a project that embedes the WAMR. This document introduces how to build the WAMR minimal product which is vmcore only (no app-framework and app-mgr) for multiple platforms. + + + +## iwasm VM core CMake building configurations + +By including the cmake scripts under folder [build-scripts](../build-scripts), it is easy to build minimal product with CMake. WAMR provides a number of features which can be easily configured through cmake variables: + +``` Bash +cmake -DWAMR_BUILD_INTERP=1/0 to enable or disable WASM intepreter +cmake -DWAMR_BUILD_FAST_INTERP=1/0 to build fast (default) or classic WASM intepreter. +cmake -DWAMR_BUILD_AOT=1/0 to enable or disable WASM AOT +cmake -DWAMR_BUILD_JIT=1/0 to enable or disable WASM JIT. (Disabled by default) +cmake -DWAMR_BUILD_LIBC_BUILTIN=1/0 enable or disable Libc builtin API's. (Enabled by default) +cmake -DWAMR_BUILD_LIBC_WASI=1/0 enable or disable Libc WASI API's +cmake -DWAMR_BUILD_TARGET= to set the building target, including: + X86_64, X86_32, ARM, THUMB, XTENSA and MIPS + For ARM and THUMB, the format is [][_VFP] where is the ARM sub-architecture and the "_VFP" suffix means VFP coprocessor registers s0-s15 (d0-d7) are used for passing arguments or returning results in standard procedure-call. Both and [_VFP] are optional. e.g. ARMV7, ARMV7_VFP, THUMBV7, THUMBV7_VFP and so on. +``` + +For example, if we want to enable classic interpreter, we can: + +``` Bash +cmake .. -DWAMR_BUILD_FAST_INTERP=0 +``` + +**Note** the fast interpreter will run ~2X faster than classic interpreter, but it consumes about 2X memory to hold the WASM bytecode code. + +If we want to disable interpreter, enable AOT and WASI, we can: + +``` Bash +cmake .. -DWAMR_BUILD_INTERP=0 -DWAMR_BUILD_AOT=1 -DWAMR_BUILD_LIBC_WASI=0 +``` + +Or if we want to enable inerpreter, disable AOT and WASI, and build as X86_32, we can: + +``` Bash +cmake .. -DWAMR_BUILD_INTERP=1 -DWAMR_BUILD_AOT=0 -DWAMR_BUILD_LIBC_WASI=0 -DWAMR_BUILD_TARGET=X86_32 +``` + +By default in Linux, the interpreter, AOT and WASI are enabled, and JIT is disabled. And the build target is +set to X86_64 or X86_32 depending on the platform's bitwidth. + +To enable WASM JIT, firstly we should build LLVM: + +``` Bash +cd product-mini/platforms/linux/ +./build_llvm.sh (The llvm source code is cloned under /core/deps/llvm and auto built) +``` + +Then pass option -DWAMR_BUILD_JIT=1 to cmake to enable WASM JIT: + +``` Bash +mkdir build +cd build +cmake .. -DWAMR_BUILD_JIT=1 +make +``` + + Linux ------------------------- First of all please install the dependent packages. Run command below in Ubuntu-18.04: + ``` Bash sudo apt install build-essential cmake g++-multilib libgcc-8-dev lib32gcc-8-dev ``` @@ -29,46 +89,13 @@ make ``` The binary file iwasm will be generated under build folder. -Note: -WAMR provides some features which can be easily configured by passing options to cmake: -``` Bash -cmake -DWAMR_BUILD_INTERP=1/0 to enable or disable WASM intepreter -cmake -DWAMR_BUILD_AOT=1/0 to enable or disable WASM AOT -cmake -DWAMR_BUILD_JIT=1/0 to enable or disable WASM JIT -cmake -DWAMR_BUILD_LIBC_BUILTIN=1/0 enable or disable Libc builtin API's -cmake -DWAMR_BUILD_LIBC_WASI=1/0 enable or disable Libc WASI API's -cmake -DWAMR_BUILD_TARGET= to set the building target, including: - X86_64, X86_32, ARM, THUMB, XTENSA and MIPS - For ARM and THUMB, the format is [][_VFP] where is the ARM sub-architecture and the "_VFP" suffix means VFP coprocessor registers s0-s15 (d0-d7) are used for passing arguments or returning results in standard procedure-call. Both and [_VFP] are optional. e.g. ARMV7, ARMV7_VFP, THUMBV7, THUMBV7_VFP and so on. -``` -For example, if we want to disable interpreter, enable AOT and WASI, we can: -``` Bash -cmake .. -DWAMR_BUILD_INTERP=0 -DWAMR_BUILD_AOT=1 -DWAMR_BUILD_LIBC_WASI=0 -``` -Or if we want to enable inerpreter, disable AOT and WASI, and build as X86_32, we can: -``` Bash -cmake .. -DWAMR_BUILD_INTERP=1 -DWAMR_BUILD_AOT=0 -DWAMR_BUILD_LIBC_WASI=0 -DWAMR_BUILD_TARGET=X86_32 -``` -By default in Linux, the interpreter, AOT and WASI are enabled, and JIT is disabled. And the build target is -set to X86_64 or X86_32 depending on the platform's bitwidth. -To enable WASM JIT, firstly we should build LLVM: -``` Bash -cd product-mini/platforms/linux/ -./build_llvm.sh (The llvm source code is cloned under /core/deps/llvm and auto built) -``` -Then pass option -DWAMR_BUILD_JIT=1 to cmake to enable WASM JIT: -``` Bash -mkdir build -cd build -cmake .. -DWAMR_BUILD_JIT=1 -make -``` Linux SGX (Intel Software Guard Extention) ------------------------- + First of all please install the [Intel SGX SDK](https://software.intel.com/en-us/sgx/sdk). After installing dependencies, build the source code: @@ -122,11 +149,11 @@ VxWorks 7 SR0620 release is validated. First you need to build a VSB. Make sure *UTILS_UNIX* layer is added in the VSB. After the VSB is built, export the VxWorks toolchain path by: -``` +```bash export /host/vx-compiler/bin:$PATH ``` Now switch to iwasm source tree to build the source code: -``` +```bash cd product-mini/platforms/vxworks/ mkdir build cd build diff --git a/doc/embed_wamr.md b/doc/embed_wamr.md index 3a2c530aa..79e5b7cc3 100644 --- a/doc/embed_wamr.md +++ b/doc/embed_wamr.md @@ -20,20 +20,20 @@ Embedding WAMR guideline // all the WAMR heap and WASM applications are limited in this buffer bh_memory_init_with_pool(global_heap_buf, sizeof(global_heap_buf)); - + wasm_runtime_init(); // read WASM file into a memory buffer buffer = read_wasm_binary_to_buffer(…, &size); - + // parse the WASM file from buffer and create a WASM module module = wasm_runtime_load(buffer, size, error_buf, sizeof(error_buf)); // create an instance of the WASM module (WASM linear memory is ready) - module_inst = wasm_runtime_instantiate(module, - stack_size, + module_inst = wasm_runtime_instantiate(module, + stack_size, heap_size, - error_buf, + error_buf, sizeof(error_buf)); ``` @@ -59,7 +59,6 @@ After a module is instantiated, the runtime native can lookup WASM functions by // call the WASM function if (wasm_runtime_call_wasm(exec_env, func, 1, argv) ) { - /* the return value is stored in argv[0] */ printf("fib function return: %d\n", argv[0]); } @@ -79,25 +78,24 @@ The parameters are transferred in an array of 32 bits elements. For parameters t double arg3 = 1.0; int 64 arg4 = 100; double ret; - + argv[0] = arg1; argv[1] = arg2; - - // use memory copy for 8 bytes parameters rather than - // *(double*)(&argv[2]) = arg3 here because some archs + + // use memory copy for 8 bytes parameters rather than + // *(double*)(&argv[2]) = arg3 here because some archs // like ARM, MIPS requires address is 8 aligned. // Or use the aligned malloc or compiler align attribute // to ensure the array address is 8 bytes aligned memcpy(&argv[2], &arg3, sizeof(arg3)); memcpy(&argv[4], &arg4, sizeof(arg4)); - - // + // // attention: the arg number is 6 here since both // arg3 and arg4 each takes 2 elements // wasm_runtime_call_wasm(exec_env, func, 6, argv); - - // if the return value is type of 8 bytes, it takes + + // if the return value is type of 8 bytes, it takes // the first two array elements memcpy(&ret, &argv[0], sizeof(ret)); @@ -109,7 +107,7 @@ The parameters are transferred in an array of 32 bits elements. For parameters t -If we need to transfer a buffer to WASM function, we can pass the buffer address through a parameter. **Attention**: The sandbox will forbid the WASM code to access outside memory, we must **allocate the buffer from WASM instance's own memory space and pass the buffer address in instance's space (not the runtime native address)**. +If we need to transfer a buffer to WASM function, we can pass the buffer address through a parameter. **Attention**: The sandbox will forbid the WASM code to access outside memory, we must **allocate the buffer from WASM instance's own memory space and pass the buffer address in instance's space (not the runtime native address)**. @@ -124,10 +122,10 @@ There are two runtime APIs available for this purpose. * size: the buffer size to allocate */ int32_t -wasm_runtime_module_malloc(wasm_module_inst_t module_inst, +wasm_runtime_module_malloc(wasm_module_inst_t module_inst, uint32_t size, - void **p_native_addr); - + void **p_native_addr); + /* * description: malloc a buffer from instance's private memory space, * and copy the data from another native buffer to it. @@ -137,8 +135,8 @@ wasm_runtime_module_malloc(wasm_module_inst_t module_inst, */ int32 wasm_runtime_module_dup_data(WASMModuleInstanceCommon *module_inst, - const char *src, - uint32 size); + const char *src, + uint32 size); ``` @@ -154,7 +152,7 @@ if(buffer_for_wasm != 0) { unit32 argv[2]; strncpy(buffer, "hello", 100); // use native address for accessing in runtime - argv[0] = buffer_for_wasm; // pass the buffer address for WASM space. + argv[0] = buffer_for_wasm; // pass the buffer address for WASM space. argv[1] = 100; // the size of buffer wasm_runtime_call_wasm(exec_env, func, 2, argv); } diff --git a/doc/port_wamr.md b/doc/port_wamr.md index ed7d4af00..50a6e619f 100644 --- a/doc/port_wamr.md +++ b/doc/port_wamr.md @@ -20,8 +20,6 @@ Create folders: Implement folder core/shared/platform/super-os. Normally in this folder you should implement the following files: - bh_platform.h and bh_platform.c: define the platform related macros, data types and APIs. - bh_assert.c: implement function bh_assert_internal() and bh_debug_internal(). -- bh_definition.c: implement function b_memcpy_s, b_strcat_s and b_strcpy_s. And implement fopen_s - if we need to read wasm file from file system. - bh_platform_log.c: implement function bh_log_emit, bh_fprintf and bh_fflush. - bh_time.c: implement several time related functions. - bh_thread.c: implement thread, mutex, condition related functions. diff --git a/product-mini/platforms/alios-things/aos.mk b/product-mini/platforms/alios-things/aos.mk index 461cb5102..83b611beb 100644 --- a/product-mini/platforms/alios-things/aos.mk +++ b/product-mini/platforms/alios-things/aos.mk @@ -71,7 +71,6 @@ GLOBAL_INCLUDES += ${IWASM_ROOT}/aot endif $(NAME)_SOURCES := ${SHARED_ROOT}/platform/alios/bh_assert.c \ - ${SHARED_ROOT}/platform/alios/bh_definition.c \ ${SHARED_ROOT}/platform/alios/bh_math.c \ ${SHARED_ROOT}/platform/alios/bh_platform.c \ ${SHARED_ROOT}/platform/alios/bh_platform_log.c \ @@ -82,6 +81,7 @@ $(NAME)_SOURCES := ${SHARED_ROOT}/platform/alios/bh_assert.c \ ${SHARED_ROOT}/mem-alloc/ems/ems_kfc.c \ ${SHARED_ROOT}/mem-alloc/ems/ems_alloc.c \ ${SHARED_ROOT}/mem-alloc/ems/ems_hmu.c \ + ${SHARED_ROOT}/utils/bh_definition.c \ ${SHARED_ROOT}/utils/bh_hashmap.c \ ${SHARED_ROOT}/utils/bh_list.c \ ${SHARED_ROOT}/utils/bh_log.c \ diff --git a/product-mini/platforms/linux/CMakeLists.txt b/product-mini/platforms/linux/CMakeLists.txt index 193d9ec0e..2b5a3e1dd 100644 --- a/product-mini/platforms/linux/CMakeLists.txt +++ b/product-mini/platforms/linux/CMakeLists.txt @@ -52,6 +52,11 @@ if (NOT DEFINED WAMR_BUILD_LIBC_WASI) set (WAMR_BUILD_LIBC_WASI 1) endif () +if (NOT DEFINED WAMR_BUILD_FAST_INTERP) + # Enable fast interpreter + set (WAMR_BUILD_FAST_INTERP 1) +endif () + set (WAMR_ROOT_DIR ${CMAKE_CURRENT_SOURCE_DIR}/../../..) include (${WAMR_ROOT_DIR}/build-scripts/runtime_lib.cmake) diff --git a/product-mini/platforms/linux/main.c b/product-mini/platforms/linux/main.c index 5f0ba1525..7bdf7a185 100644 --- a/product-mini/platforms/linux/main.c +++ b/product-mini/platforms/linux/main.c @@ -155,6 +155,7 @@ int main(int argc, char *argv[]) uint32 wasm_file_size; wasm_module_t wasm_module = NULL; wasm_module_inst_t wasm_module_inst = NULL; + RuntimeInitArgs init_args; char error_buf[128] = { 0 }; #if WASM_ENABLE_LOG != 0 int log_verbose_level = 2; @@ -228,35 +229,35 @@ int main(int argc, char *argv[]) app_argc = argc; app_argv = argv; + memset(&init_args, 0, sizeof(RuntimeInitArgs)); + #if USE_GLOBAL_HEAP_BUF != 0 - if (bh_memory_init_with_pool(global_heap_buf, sizeof(global_heap_buf)) - != 0) { - bh_printf("Init memory with global heap buffer failed.\n"); - return -1; - } + init_args.mem_alloc_type = Alloc_With_Pool; + init_args.mem_alloc.pool.heap_buf = global_heap_buf; + init_args.mem_alloc.pool.heap_size = sizeof(global_heap_buf); #else - if (bh_memory_init_with_allocator(malloc, free)) { - bh_printf("Init memory with memory allocator failed.\n"); - return -1; - } + init_args.mem_alloc_type = Alloc_With_Allocator; + init_args.mem_alloc.allocator.malloc_func = malloc; + init_args.mem_alloc.allocator.free_func = free; #endif - /* initialize runtime environment */ - if (!wasm_runtime_init()) - goto fail1; + if (!wasm_runtime_full_init(&init_args)) { + bh_printf("Init runtime environment failed.\n"); + return -1; + } bh_log_set_verbose_level(log_verbose_level); /* load WASM byte buffer from WASM bin file */ if (!(wasm_file_buf = (uint8*) bh_read_file_to_buffer(wasm_file, &wasm_file_size))) - goto fail2; + goto fail1; /* load WASM module */ if (!(wasm_module = wasm_runtime_load(wasm_file_buf, wasm_file_size, error_buf, sizeof(error_buf)))) { bh_printf("%s\n", error_buf); - goto fail3; + goto fail2; } #if WASM_ENABLE_LIBC_WASI != 0 @@ -274,7 +275,7 @@ int main(int argc, char *argv[]) error_buf, sizeof(error_buf)))) { bh_printf("%s\n", error_buf); - goto fail4; + goto fail3; } if (is_repl_mode) @@ -287,20 +288,17 @@ int main(int argc, char *argv[]) /* destroy the module instance */ wasm_runtime_deinstantiate(wasm_module_inst); -fail4: +fail3: /* unload the module */ wasm_runtime_unload(wasm_module); -fail3: +fail2: /* free the file buffer */ bh_free(wasm_file_buf); -fail2: - /* destroy runtime environment */ - wasm_runtime_destroy(); - fail1: - bh_memory_destroy(); + /* destroy runtime environment */ + wasm_runtime_full_destroy(); return 0; }