86 Commits

Author	SHA1	Message	Date
YAMAMOTO Takashi	735b510437	core/iwasm/libraries/wasi-nn/test/build.sh: set -e	2025-08-19 14:58:27 +09:00
YAMAMOTO Takashi	c661592edd	wasi_nn_onnx.cpp: fix debug build (#4564)	2025-08-19 08:23:00 +08:00
dongsheng28849455	14ced7c86c	Add onnxruntime as wasi-nn backend (#4485) ... * Add onnxruntime as wasi-nn backend * remove global context * put checks under the lock * tensor type will not support legacy wasi-nn abi * Manually set the imported target with name space	2025-08-15 09:48:37 +08:00
YAMAMOTO Takashi	6b51c61f5e	wasi-nn: fix set_input memory address validation for the legacy abi (#4534) ... cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/3223	2025-08-06 10:40:56 +08:00
YAMAMOTO Takashi	29d465b44e	wasi_nn_tensorflowlite.cpp: make this compatible with wasmedge (#4517) ... for wasi_ephemeral_nn, * implement u8 input * stop dealing with quantization. * wasi-nn doesn't have a concept of quantization or pre/post-processing. i can't think of any ways to make the backend perform zero-point/scale processing without risking to break other applications. * there seems to be applications which just use u8 inputs/outputs for a quantized model. (see [1] for an example.) for certain kinds of inputs/outputs, it usually just works. this commit keeps the legacy wasi_nn logic intact for now. tested with [1] with [2] applied. WAMR with this patch: ``` Read graph weights, size in bytes: 3561598 [wasi_nn.c:297 WARNING] load_by_name_with_config() not found [wasi_nn_tensorflowlite.cpp:272 WARNING] Default encoding is CPU. Loaded graph into wasi-nn with ID: Graph#0 Read input tensor, size in bytes: 150528 1.) [166](198)Aix galericulata 2.) [34](1)Gallus gallus domesticus 3.) [158](1)Coccothraustes coccothraustes 4.) [778](1)Sitta europaea 5.) [819](1)Anas platyrhynchos ``` wasmedge: ``` Read graph weights, size in bytes: 3561598 Loaded graph into wasi-nn with ID: Graph#0 Read input tensor, size in bytes: 150528 1.) [166](198)Aix galericulata 2.) [34](1)Gallus gallus domesticus 3.) [158](1)Coccothraustes coccothraustes 4.) [778](1)Sitta europaea 5.) [819](1)Anas platyrhynchos ``` and "Aix galericulata" seems like a reasonable classification of the image to my eyes. [1] 67f174bab5/tflite-birds_v1-image [2] https://github.com/second-state/WasmEdge-WASINN-examples/pull/204 Related: https://github.com/bytecodealliance/wasm-micro-runtime/issues/3555 https://github.com/bytecodealliance/wasm-micro-runtime/issues/2611	2025-08-01 14:31:02 +08:00
YAMAMOTO Takashi	a0de8c7b7d	wasi-nn: fix buffer overruns in load_by_name/load_by_name_with_config (#4491) ... also, ensure NUL terminations here to simplify backend implementations. (note that, although names with '\0' in the middle are usually valid in wasm, wamr doesn't support them in general. we might revisit this later.) also, add a missing address validation in load_by_name_with_config. cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4487	2025-07-30 10:27:20 +08:00
YAMAMOTO Takashi	e4d2673a7c	wasi-nn: add a comment on load_by_name_with_config (#4492) ... this api has been introduced by https://github.com/bytecodealliance/wasm-micro-runtime/pull/3709 in which i couldn't find any explanations. this comment is my best guess.	2025-07-17 09:50:49 +08:00
YAMAMOTO Takashi	56f87b7ee9	wasi-nn: do not pretend to support legacy abi in openvino and llamacpp (#4468) ... as tested by core/iwasm/libraries/wasi-nn/test/test_tensorflow.c, the legacy "wasi_nn" abi uses the number of fp32 for get_output. because these backends don't implement the abi, bail out explicitly in build time. cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4376	2025-07-10 08:28:08 +08:00
YAMAMOTO Takashi	fbd27e5e03	wasi_nn_llamacpp.c: explicitly reject unimplemented input index (#4446) ... note: wasmedge seems to accept index 1 for metadata. we don't implement it.	2025-07-09 10:35:46 +08:00
YAMAMOTO Takashi	ee056d8076	wasi_nn_llamacpp.c: validate input tensor type/dimensions (#4442)	2025-07-03 10:17:19 +08:00
YAMAMOTO Takashi	da6019f749	wasi_nn_llamacpp.c: reject invalid graph and execution context (#4422) ... * return valid graph and execution context instead of using stack garbage. (always 0 for now because we don't implement multiple graph/context for this backend.) * validate user-given graph and execution context values. reject invalid ones.	2025-07-01 19:31:00 +08:00
YAMAMOTO Takashi	ebf1404ad1	wasi_nn_openvino.c: avoid self-assignment warning (#4434)	2025-07-01 19:19:36 +08:00
YAMAMOTO Takashi	38fe056cc6	wasi-nn: reduce code duplication a bit (#4433)	2025-07-01 10:37:12 +08:00
YAMAMOTO Takashi	7a6a6a39e9	wasi_nn_openvino.c: fix a debug build (#4416) ... after "wasi_nn_openvino.c: implement multiple models per instance" change. (https://github.com/bytecodealliance/wasm-micro-runtime/pull/4380)	2025-06-30 12:57:44 +08:00
YAMAMOTO Takashi	8ed89e2ab2	wasi_nn_llamacpp.c: remove an unused variable (#4415)	2025-06-27 11:55:08 +08:00
YAMAMOTO Takashi	d6876f1e9f	wasi_nn_llamacpp.c: fix buffer overruns in set_input (#4420) ... note: for some reasons, wasmedge seems to ignore type/dimensions for the input of ggml. some user code relies on it. cf. https://github.com/second-state/WasmEdge-WASINN-examples/issues/196 note: despite the comment in our code, the input doesn't seem nul-terminated.	2025-06-27 11:51:03 +08:00
YAMAMOTO Takashi	2372a472aa	wasi-nn: make the host use the wasi_ephemeral_nn version of tensor_data (#4411) ... the motivations: * make the actual input size available to the backends. (currently the backends have to make a guess from shape/type.) * make the host logic look a bit similar to wasi_ephemeral_nn. this is a backend api/abi change.	2025-06-27 07:41:42 +08:00
YAMAMOTO Takashi	a7aae9d2cc	wasi_nn_llamacpp.c: make this compilable (#4403)	2025-06-26 07:05:45 +08:00
YAMAMOTO Takashi	8289452abb	wasi_nn_tensorflowlite.cpp: fix get_output return size (#4390) ... it should be byte size, not the number of (fp32) values. i'm ambivalent about how to deal with the compatibility for the legacy wamr-specific "wasi_nn". for now, i avoided changing it. (so that existing tests using the legacy abi, namely test_tensorflow.c and test_tensorflow_quantized.c, passes as they are.) if we have any users who still want to use the legacy abi, i suppose they consider the compatibility is more important than the consistency with other backends. cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4376	2025-06-24 20:38:19 +08:00
YAMAMOTO Takashi	70c39bae77	wasi-nn: fix context lifetime issues (#4396) ... * wasi-nn: fix context lifetime issues use the module instance context api instead of trying to roll our own with a hashmap. this fixes context lifetime problems mentioned in https://github.com/bytecodealliance/wasm-micro-runtime/issues/4313. namely, * wasi-nn resources will be freed earlier now. before this change, they used to be kept until the runtime shutdown. (wasm_runtime_destroy) after this change, they will be freed together with the associated instances. * wasm_module_inst_t pointer uniqueness assumption (which is wrong after wasm_runtime_deinstantiate) was lifted. as a side effect, this change also makes a context shared among threads within a cluster. note that this is a user-visible api/abi breaking change. before this change, wasi-nn "handles" like wasi_ephemeral_nn_graph were thread-local. after this change, they are shared among threads within a cluster, similarly to wasi file descriptors. spec-wise, either behavior should be ok simply because wasi officially doesn't have threads yet. althogh i feel the latter semantics is more intuitive, if your application depends on the thread-local behavior, this change breaks your application. tested with wamr-wasi-extensions/samples/nn-cli, modified to call each wasi-nn operations on different threads. (if you are interested, you can find the modification at https://github.com/yamt/wasm-micro-runtime/tree/yamt-nn-wip-20250619.) cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4313 https://github.com/bytecodealliance/wasm-micro-runtime/issues/2430 * runtime_lib.cmake: enable WAMR_BUILD_MODULE_INST_CONTEXT for wasi-nn as we do for wasi (WAMR_BUILD_LIBC_WASI)	2025-06-24 20:37:56 +08:00
YAMAMOTO Takashi	f449b79a31	wasi_nn_openvino.c: implement multiple models per instance (#4380) ... tested with two models: ``` --load-graph=id=graph1,file=public/license-plate-recognition-barrier-0007/FP32/license-plate-recognition-barrier-0007.xml,file=public/license-plate-recognition-barrier-0007/FP32/license-plate-recognition-barrier-0007.bin \ --load-graph=id=graph2,file=classify/model.xml,file=classify/model.bin \ --init-execution-context=id=exec1,graph-id=graph1 \ --init-execution-context=id=exec2,graph-id=graph2 \ --set-input=context-id=exec1,dim=1,dim=24,dim=94,dim=3,file=out.bin \ --set-input=context-id=exec2,file=classify/banana-3x224x224-bgr.bin,dim=1,dim=3,dim=224,dim=224 \ --compute=context-id=exec1 \ --compute=context-id=exec2 \ --get-output=context-id=exec1,file=exec1-result.bin \ --get-output=context-id=exec2,file=exec2-result.bin ``` a detailed HOWTO: https://github.com/bytecodealliance/wasm-micro-runtime/pull/4380#issuecomment-2986882718	2025-06-20 15:50:29 +08:00
YAMAMOTO Takashi	ea408ab6c0	wasi-nn: add minimum serialization on WASINNContext (#4387) ... currently this is not necessary because context (WASINNContext) is local to instance. (wasm_module_instance_t) i plan to make a context shared among instances in a cluster when fixing https://github.com/bytecodealliance/wasm-micro-runtime/issues/4313. this is a preparation for that direction. an obvious alternative is to tweak the module instance context APIs to allow declaring some kind of contexts instance-local. but i feel, in this particular case, it's more natural to make "wasi-nn handles" shared among threads within a "process". note that, spec-wise, how wasi-nn behaves wrt threads is not defined at all because wasi officially doesn't have threads yet. i suppose, at this point, that how wasi-nn interacts with wasi-threads is something we need to define by ourselves, especially when we are using an outdated wasi-nn version. with this change, if a thread attempts to access a context while another thread is using it, we simply make the operation fail with the "busy" error. this is intended for the mimimum serialization to avoid problems like crashes/leaks/etc. this is not intended to allow parallelism or such. no functional changes are intended at this point yet. cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4313 https://github.com/bytecodealliance/wasm-micro-runtime/issues/2430	2025-06-20 09:48:55 +08:00
YAMAMOTO Takashi	71c07f3e4e	deprecate legacy WAMR-specific "wasi_nn" module (#4382) ... wasi_nn.h: deprecate legacy "wasi_nn" cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4326	2025-06-19 14:32:26 +08:00
YAMAMOTO Takashi	aa53d648fa	wasi-nn: fix tensor_data abi for wasi_ephemeral_nn (#4379) ... it's "(list u8)" in the witx definition. the new definition matches both of our own host definition (struct tensor_wasm) and wasmtime. cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4352	2025-06-19 14:18:36 +08:00
YAMAMOTO Takashi	a29f3943ef	core/iwasm/libraries/wasi-nn/test: use the correct version of keras (#4383)	2025-06-18 19:24:06 +08:00
YAMAMOTO Takashi	db7714f0f5	wasi_nn_tensorflowlite.cpp: reject non-fp32 input earlier (#4388) ... this backend assumes fp32 here and there. it's safer to reject unexpected inputs explicitly.	2025-06-18 19:08:57 +08:00
YAMAMOTO Takashi	4bf799c3af	core/iwasm/libraries/wasi-nn/test/build.sh: add a tip for intel mac (#4389) ... i keep forgetting this and had to re-investigate it at least twice. hopefully this can be helpful for others too.	2025-06-18 19:06:57 +08:00
YAMAMOTO Takashi	cba9001749	wasi-nn: don't try to deinit uninitialized backend (#4375) ... cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4339	2025-06-17 17:40:53 +08:00
YAMAMOTO Takashi	c9b8c16088	wasi_nn_openvino.c: remove pre/postprocessing and layout assumptions (#4361) ... as wasi-nn doesn't have these concepts, the best we can do without risking breaking certain applications here is to pass through tensors as they are. this matches wasmtime's behavior. tested with: * wasmtime classification-example (with this change, this example fails on tensor size mismatch instead of implicitly resizing it.) * license-plate-recognition-barrier-0007, a converted version with non-fp32 output. [1] (with this change, this model outputs integers as expected.) [1] cd7ebe313b/models/public/license-plate-recognition-barrier-0007	2025-06-17 13:01:46 +08:00
YAMAMOTO Takashi	2f0750a6fe	wasi_nn_openvino.c: add a missing buffer overflow check in get_output (#4353) ... cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4351	2025-06-17 11:17:00 +08:00
YAMAMOTO Takashi	20be1d33fe	wasi_ephemeral_nn.h: prefix identfiers to avoid too generic names (#4358)	2025-06-17 11:15:01 +08:00
YAMAMOTO Takashi	745da82cd6	wasi_nn_openvino.c: remove broken xml check (#4365) ... `xml.buf[xml.size]` check is broken because it accesses past the end of the buffer. anyway, openvino doesn't seem to care the NUL termination.	2025-06-17 11:02:36 +08:00
YAMAMOTO Takashi	0d001c4c38	wasi-nn: fix backend leak on multiple loads (#4366) ... cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4340	2025-06-17 11:01:07 +08:00
YAMAMOTO Takashi	5478d267f4	wasi_nn_openvino.c: remove the tensor layout adjustment logic (#4308) ... the logic in question seems like an attempt to work around some application bugs. my wild guess is that it was for classification-example. cf. https://github.com/bytecodealliance/wasmtime/issues/10867	2025-06-12 09:34:14 +08:00
YAMAMOTO Takashi	3a087c4244	wamr-wasi-extensions: add a cmake package to provide our wasi extension (#4344) ... * wasi_ephemeral_nn.h: add a convenience wrapper header * wamr-wasi-extensions: add a cmake package to provide our wasi extension the sample app was tested with: * wasmtime * iwasm with https://github.com/bytecodealliance/wasm-micro-runtime/pull/4308 currently only contains wasi-nn. maybe it makes sense to add lib-socket things as well. cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4288	2025-06-12 09:33:25 +08:00
YAMAMOTO Takashi	c932597057	wasi_nn_types.h: remove a seemingly stale comment (#4348)	2025-06-12 09:29:59 +08:00
YAMAMOTO Takashi	ea5757f1d7	wasi-nn: do not assign wasi_nn_ctx->backend multiple times (#4329)	2025-06-09 11:36:31 +08:00
YAMAMOTO Takashi	4d6b8dcd5d	wasi_nn.h: make this compatible with wasi_ephemeral_nn (#4330) ... - wasi_nn.h: make this compatible with wasi_ephemeral_nn cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4323 - fix WASM_ENABLE_WASI_EPHEMERAL_NN build this structure is used by host logic as well. ideally definitions for wasm and host should be separated. until it happens, check __wasm__ to avoid the breakage.	2025-06-09 11:36:05 +08:00
YAMAMOTO Takashi	933f8124b0	wasi-nn: fix the size of tensor->type (#4333) ... * this enum is (@witx tag u8) in witx * it seems that some wasm modules actually use non-zero padding and cause errors * it's a bad practice to use C enum for ABI description anyway	2025-06-06 15:08:18 +08:00
YAMAMOTO Takashi	769d16eaab	wasi-nn: move some host-only things out of wasi_nn_types.h (#4334) ... cf. https://github.com/bytecodealliance/wasm-micro-runtime/issues/4324	2025-06-06 15:07:29 +08:00
YAMAMOTO Takashi	79cb4366ae	wasi-nn: remove unused wasi_nn_dump_tensor_dimension prototype (#4325)	2025-06-05 09:48:28 +08:00
YAMAMOTO Takashi	b20ebc2724	wasi_nn.h: add import_name attribute (#4328) ... this would fix undefined symbol errors by making it clear these functions are imported. references: e2c698c7e8/llvm/lib/MC/WasmObjectWriter.cpp (L1798-L1799) e2c698c7e8/llvm/lib/Object/WasmObjectFile.cpp (L749-L752) e2c698c7e8/lld/wasm/Symbols.cpp (L203) e2c698c7e8/lld/wasm/Relocations.cpp (L36-L40)	2025-06-05 09:48:00 +08:00
YAMAMOTO Takashi	85efe08431	wasi-nn: protect the backend lookup table with a lock (#4319) ... this would avoid potential issues when multiple instances happen to make an attempt to load a backend at the same time. Fixes: https://github.com/bytecodealliance/wasm-micro-runtime/issues/4314	2025-06-05 09:19:46 +08:00
YAMAMOTO Takashi	6a00874f2f	wasi_nn_openvino.c: make this buildable (#4305)	2025-06-03 13:28:13 +08:00
YAMAMOTO Takashi	61cb97221e	wasi-nn: fix shared library filenames for macOS (#4306) ... tested with openvino	2025-06-03 13:23:19 +08:00
YAMAMOTO Takashi	ae6e490ad5	fix wasi-nn abi definitions (#4307) ... sync with a more appropriate version of the definitions. as we use the "wasi_ephemeral_nn", which is p1-based, it seems more appropriate to use definitions from witx, not wit. it's a bit unfortunate p2-based wasi-nn made gratuitous changes like this from p1. note: this is an ABI change.	2025-06-03 13:22:48 +08:00
YAMAMOTO Takashi	16c46751ac	wasi-nn: remove "backends" argument from detect_and_load_backend() (#4309) ... it seems meaningless and quite confusing to access a table with two aliases ("lookup" and "backends") within a function. no functional changes are intended.	2025-06-03 13:22:27 +08:00
YAMAMOTO Takashi	1c12a32066	wasi_nn_openvino.c: fix a few printf formats (#4310)	2025-06-03 13:21:32 +08:00
hongxia	aa1ff778b9	add load_by_name in wasi-nn (#4298)	2025-06-03 06:26:58 +08:00
hongxia	3ab9f84026	Dockerfile.vx-delegate build error fix (#4273) ... - specify tensorflow version & bugfix	2025-05-28 20:29:41 +08:00