Skip to content

feat: import plugin sources, tests, and CI utils from WasmEdge core#44

Merged
0yi0 merged 623 commits into
mainfrom
import/wasmedge-master
Jun 12, 2026
Merged

feat: import plugin sources, tests, and CI utils from WasmEdge core#44
0yi0 merged 623 commits into
mainfrom
import/wasmedge-master

Conversation

@0yi0

@0yi0 0yi0 commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

PR summary

Implements #23 (epic #22; upstream WasmEdge/WasmEdge#3840): imports the C++ plugin sources, their tests, and plugin-related CI utilities from WasmEdge/WasmEdge with git history, rebased linearly onto main — no merge commit required.

Implementation design

  • Fresh full clone of upstream master @ d67087e (2026-06-11)
  • git filter-repo over: plugins, test/plugins, thirdparty/wasi_crypto, utils/docker, utils/ffmpeg, utils/opencvmini, utils/wasi-crypto, utils/wasi-nn, utils/wasi-test, utils/build_libpiper.sh623 commits, 0 merges (linear)
  • git rebase --onto main --root --empty=drop --committer-date-is-author-date: replays all 623 commits onto main — original authors, dates, messages, and DCO sign-offs preserved; committer rewritten (rebase semantics); every replayed tree keeps main's existing files
  • Pushed unmodified as import/wasmedge-master

Design decisions

  • Linear rebase instead of unrelated-histories merge (maintainer direction): keeps main linear, avoids enabling merge commits in repo settings
  • Merge mechanics: prefer fast-forwarding main to this exact branch head (git push origin import/wasmedge-master:main) — preserves reviewed SHAs and auto-marks this PR merged. The "Rebase and merge" button also works but rewrites all 623 SHAs again
  • utils/ selection ("test or docker related"): in = docker, ffmpeg, opencvmini, wasi-crypto, wasi-nn, wasi-test, build_libpiper.sh; out = installer files (stay upstream → feat: installer integration: fetch plugin assets from cpp-plugins releases #27), corpus, wasi-cpp-header, ohos, openwrt
  • History depth: path filtering doesn't follow renames across upstream's ~2022 plugins/ restructure — e.g. plugins/wasi_crypto carries 34 commits; earlier history remains reachable upstream. Same precedent as the 2024 import (feat: Initialization #1)
  • Path layout kept exactly as upstream for git blame continuity

Commit slicing

No authored commits — this PR is exactly the 623 filtered upstream commits replayed onto main. Follow-ups land in separate PRs (#24 build system, #25 pilot).

Test plan

(no build/test execution — imported sources are not expected to build standalone yet; that is #24)

  • HEAD tree = main's files + plugins/ (15 plugins) + test/plugins/ + thirdparty/wasi_crypto + selected utils/ — 493 files, none outside expected paths
  • git rev-list --count main..HEAD = 623; root commit's parent = main head fd798bc
  • git log --oneline -- plugins/wasi_crypto | wc -l = 34; git blame plugins/wasi_crypto/ctx.h shows original upstream authors
  • 0 merge commits (linear)
  • DCO: ❌ 95/623 commits flagged — artifact of upstream's squash-merge flow (author rewritten to GitHub noreply identity while the sign-off kept the contributor's real email; also one name change grorgeHan-Wen Tsao). Not introduced by this import; see options in the PR comment below
  • Lint: ❌ clang-format violations in thirdparty/wasi_crypto/api.hpp (generated from witx), test/plugins/wasm_bpf/assets/bpf-sources/ (test fixtures), test/plugins/wasi_crypto/asymmetric.cpp — paths upstream does not format-enforce, plus probable clang-format version drift. commitlint did not flag — the predicted risk did not materialize

Non-goals / afterwards

Closes #23


🤖 Generated by Claude Fable 5 with Claude Code

CaptainVincent and others added 30 commits June 2, 2024 22:57
Signed-off-by: vincent <vincent@secondstate.io>
Use -DWASMEDGE_PLUGIN_WASI_NN_RUST_MODEL=squeezenet or whisper for switch plugin model

Signed-off-by: vincent <vincent@secondstate.io>
* update hostfunc's fromTuple to update Span of ValInterface
* add emplace helper
* fix `Wit` type converter, it still convert `vector` rather than `List`

Signed-off-by: Lîm Tsú-thuàn <dannypsnl@secondstate.io>
Signed-off-by: Lîm Tsú-thuàn <dannypsnl@secondstate.io>
Signed-off-by: grorge <g97116@yahoo.com>
Signed-off-by: grorge <g97116@yahoo.com>
Signed-off-by: grorge <g97116@yahoo.com>
Signed-off-by: grorge <g97116@yahoo.com>
Signed-off-by: grorge <g97116@yahoo.com>
Signed-off-by: vincent <vincent@secondstate.io>
Signed-off-by: YiYing He <yiying@secondstate.io>
Signed-off-by: YiYing He <yiying@secondstate.io>
Signed-off-by: Shen-Ta Hsieh <ibmibmibm.tw@gmail.com>
Signed-off-by: grorge <g97116@yahoo.com>
Signed-off-by: hydai <z54981220@gmail.com>
Signed-off-by: grorge <g97116@yahoo.com>
* update piper patch to support static linking piper-phonemize and espeak-ng and support using system onnxruntime

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* force BUILD_SHARED_LIBS OFF and CMAKE_POSITION_INDEPENDENT_CODE ON for piper

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* avoid unwanted targets from piper

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* add onnxruntime install script

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* update piper patch to use PRIVATE scope for target-specific commands to avoid include and link pollution

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* add clean include directory for piper
disable tests for piper_phonemize
use normal variable instead of CACHE for piper FetchContent

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* remove unnecessary target_include_directories

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* install onnxruntime in workflow

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* remove the `-ubuntu` suffix because this should also be used in the manylinux(CentOS) distributions

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* remove redundant lines
will be done later by `wasmedge_setup_wasinn_target(wasmedgePluginWasiNN)`

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* find onnxruntime early and fail if not found

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

---------

Signed-off-by: PeterD1524 <qaz246135@gmail.com>
Signed-off-by: grorge <g97116@yahoo.com>
Signed-off-by: grorge <g97116@yahoo.com>
Signed-off-by: dm4 <dm4@secondstate.io>
Signed-off-by: hydai <z54981220@gmail.com>
…NN Piper CI (#3622)

* install onnxruntime in manylinux_2_28-plugins-deps for WASI-NN Piper CI

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* update utils/wasi-nn/install-onnxruntime.sh to support both x86_64 and aarch64

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

---------

Signed-off-by: PeterD1524 <qaz246135@gmail.com>
* enable wasi_nn-piper build for manylinux_2_28_x86_64

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* update piper patch to allow passing CMAKE_POSITION_INDEPENDENT_CODE to espeak-ng
It worked on ubuntu before because gcc on ubuntu (wasmedge/wasmedge:ubuntu-build-gcc) is configured with --enable-default-pie.
This will turn R_X86_64_32S into R_X86_64_PC32.
gcc on wasmedge/wasmedge:manylinux_2_28_x86_64-plugins-deps is not configured with --enable-default-pie,
so the error `libespeak-ng.a(case.c.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC` will occur.
The configuration of gcc can be obtained by running `gcc -v`.
As a result, passing CMAKE_POSITION_INDEPENDENT_CODE=ON to espeak-ng to compile with -fPIC is required.

Some references:
https://stackoverflow.com/a/46493456
https://stackoverflow.com/a/6093910
https://www.ucw.cz/~hubicka/papers/abi/node19.html

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

* enable wasi_nn-piper build for manylinux_2_28_aarch64

Signed-off-by: PeterD1524 <qaz246135@gmail.com>

---------

Signed-off-by: PeterD1524 <qaz246135@gmail.com>
Signed-off-by: grorge <g97116@yahoo.com>
Signed-off-by: YiYing He <yiying@secondstate.io>
Signed-off-by: YiYing He <yiying@secondstate.io>
Signed-off-by: Lîm Tsú-thuàn <dannypsnl@secondstate.io>
Signed-off-by: Lîm Tsú-thuàn <dannypsnl@secondstate.io>
Signed-off-by: dm4 <dm4@secondstate.io>
…nd spdlog

Signed-off-by: PeterD1524 <qaz246135@gmail.com>
kalamkaar9404 and others added 20 commits January 28, 2026 21:21
feat(wasi-nn): support HIP backend for llama.cpp

Signed-off-by: Khushi <khushisingh82072@gmail.com>
…ckend (#4622)

Signed-off-by: Divyansh Khatri <divyanshkhatri200102@gmail.com>
…4634)

fix: install espeak-ng-data for Piper backend tests in CI

Signed-off-by: Divyansh Khatri <divyanshkhatri200102@gmail.com>
… changes (#4443)

feat(wasi-nn): upgrade Piper backend to use new upstream

Signed-off-by: Divyansh Khatri <divyanshkhatri200102@gmail.com>
* fix(plugin): correct timeout unit conversion in process plugin

The timeout calculation divided microseconds by 1000000 (converting to
seconds) instead of 1000 (converting to milliseconds). Since the
seconds part is already multiplied by 1000 to convert to milliseconds,
the microseconds part should also be in milliseconds.

This gave the timeout only second-level granularity instead of the
intended millisecond-level granularity, making sub-second timeouts
effectively zero.

Signed-off-by: Yi LIU <yi@quantstamp.com>

* test(plugin): add process plugin timeout precision test

Signed-off-by: Yi LIU <yi@quantstamp.com>

---------

Signed-off-by: Yi LIU <yi@quantstamp.com>
Signed-off-by: YiYing He <yiying@secondstate.io>
Assisted-by: Codex (OpenAI)
Signed-off-by: grorge <g97116@yahoo.com>
Signed-off-by: grorge <g97116@yahoo.com>
chore: fix some minor issues

Signed-off-by: purelualight <purelualight@outlook.com>
Assisted-by: OpenAI Codex (GPT-5.5)

Signed-off-by: hydai <z54981220@gmail.com>
Co-authored-by: YiYing He <yiying@secondstate.io>
…n (#4910)

Signed-off-by: Pranjal Kole <pranjal.kole7@gmail.com>
… (#4913)

Signed-off-by: Pranjal Kole <pranjal.kole7@gmail.com>
Signed-off-by: hydai <z54981220@gmail.com>
…s (#4923)

imdecode/imshow/imwrite/imencode fetched the input buffer with
getPointer<char*>, which only validates one byte, then read a
guest-controlled length past it (host out-of-bounds read). Switch to
getSpan<char>(Ptr, Len) and reject a short span, matching the
output-buffer check already present in imencode.

Signed-off-by: aizu-m <aizumusheer2@gmail.com>
WasiNNSetInput already retrieves the tensor data through Tensor.Tensor before constructing the RPC request.

This change reuses that existing span when calling RPCTensor.set_data() instead of retrieving the pointer and length again from WasiTensor.

The behavior for valid inputs remains unchanged.

Signed-off-by: Yashika <ssyashika1311@gmail.com>
Signed-off-by: vishal2005025 <vishalmalyan2005@gmail.com>
…776)

* feat(wasi_crypto): implement managed keypair generation for EDDSA

- Implemented keypairGenerateManaged for EDDSA algorithms in ctx.cpp.
- Added comprehensive positive and negative tests in asymmetric.cpp.
- Integrated NOT_IMPLEMENTED verification for unsupported algorithms.
- Updated notimplement.cpp to reflect the new implementation.
- Removed unrelated formatting and typo fixes to minimize PR noise.

Signed-off-by: ShigrafS <shigrafsalik@proton.me>

* fix(test): resolve type mismatch in managed keypair generation test

- Use std::optional<__wasi_options_t> instead of __wasi_opt_options_t to match helper function signature.
- Pass InvaildHandle directly as an optional value.

Signed-off-by: ShigrafS <shigrafsalik@proton.me>

* style(wasi_crypto): format code using clang-format-20

Signed-off-by: SHIGRAF SALIK <140247389+ShigrafS@users.noreply.github.com>

* test(wasi_crypto): refactor managed keypair tests to follow project patterns

- Define ManagedNegativeCheck lambda for data-driven negative testing.
- Integrate managed keypair negative tests into the main Asymmetric suite.
- Replace manual EXPECT_EQ with WASI_CRYPTO_EXPECT_FAILURE macro in notimplement.cpp.

Assisted-by: Gemini (Google)
Signed-off-by: ShigrafS <shigrafsalik@proton.me>

---------

Signed-off-by: ShigrafS <shigrafsalik@proton.me>
Signed-off-by: SHIGRAF SALIK <140247389+ShigrafS@users.noreply.github.com>
Rename the constructor parameters that shadowed the inherited Env
member to HostEnv, in both the HostFunction base and its subclasses.

Assisted-by: Claude (Anthropic)
Signed-off-by: hydai <z54981220@gmail.com>
Rename parameters and locals that shadowed members, types, or
outer-scope bindings across the MLX base module, embedding, linear,
and pooling layers, the whisper tokenizer and decoder, and the VLM
cache, and give the pooling constructor parameters semantic names.

Assisted-by: Claude (Anthropic)
Signed-off-by: hydai <z54981220@gmail.com>
Implement Eddsa::PublicKey::verify() using EVP_PKEY_public_check. Part of #2669.

Signed-off-by: Parth Dagia <parth.24bcs10414@sst.scaler.com>
@0yi0

0yi0 commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator Author

Check results and options (maintainer decision needed)

(revised after reviewing the DCO app's documented mechanisms)

DCO — 95/623 flagged

Root cause: upstream squash-merges rewrite the commit author to the contributor's GitHub noreply identity while the Signed-off-by keeps their real email (plus one name-change case). These commits passed DCO upstream in their original PRs; the mismatch is a squash artifact, not something this import introduced.

Options, in increasing order of formality:

  1. Maintainer override button (recommended) — the DCO check exposes a "Set DCO to pass" button to users with write access, built for exactly this case: commits whose certification a maintainer can vouch for but the bot can't mechanically verify. One click, recorded against the overriding maintainer, no history changes.
  2. Third-party remediation commits (the formal Linux Foundation mechanism) — land a config PR adding .github/dco.yml on main:
    allowRemediationCommits:
      individual: true
      thirdParty: true
    then push additive (empty) commits where the maintainer retroactively signs off on behalf of the 95 original authors (DCO §c). Keeps the 623 imported commits byte-identical and records the certification in git history itself.
  3. git rebase --root --signoff + force-push — the bot's default suggestion; appends the maintainer's sign-off to every commit (bot accepts committer-matching sign-offs). Rewrites all 623 SHAs and alters historical messages — trades away the byte-fidelity this import deliberately preserves.
  4. Accept the red check and fast-forward merge from CLI (git push origin import/wasmedge-master:main) — works since main is unprotected, but leaves no record that the DCO state was considered.

Lint — clang-format only (commitlint passed)

Violations are confined to thirdparty/wasi_crypto/api.hpp (generated from witx), test/plugins/wasm_bpf/assets/bpf-sources/ (BPF test fixtures), and test/plugins/wasi_crypto/asymmetric.cpp — paths upstream doesn't format-enforce, plus likely clang-format version drift between upstream CI and super-linter's bundled binary.

Options:

  1. Config follow-up PR (recommended) — add FILTER_REGEX_EXCLUDE for thirdparty/.* and test/plugins/wasm_bpf/assets/.* to super-linter.yml; reformat asymmetric.cpp in feat: standalone CMake build against a released WasmEdge SDK #24 where the pinned clang-format version gets decided. Keeps this import byte-identical to upstream.
  2. Reformat now — one style: commit on top of the import; turns Lint green here but breaks byte-fidelity with upstream and muddies future re-import diffs.

Merge mechanics

Both options below produce linear history with no merge commit, and both end with GitHub marking this PR merged (a fast-forward push to main is auto-detected).

  1. CLI fast-forward (recommended)git push origin import/wasmedge-master:main. A true fast-forward: main receives the exact reviewed commits (HEAD 18ce2cd), preserving SHAs, committer info, and the --committer-date-is-author-date normalization.
  2. GitHub "Rebase and merge" button — content, authors, author dates, Signed-off-by lines, and git blame continuity all survive, but GitHub's rebase always rewrites committer info and creates new commit SHAs even when a fast-forward is possible. The 623 reviewed SHAs referenced in this PR and feat: import plugin sources and tests from WasmEdge core with git history #23 would stop resolving on main, and committer dates become merge time. Server-side rebases of this size have also been known to time out; the fallback is option 1. DCO is unaffected either way (authors unchanged; main is unprotected so the red check doesn't block the button).

Recommendation

DCO option 1 (override button) + Lint option 1 (config follow-up) + Merge option 1 (CLI fast-forward) to preserve the reviewed SHAs. DCO option 2 is the right choice instead if an in-history certification record is preferred over check metadata.


🤖 Generated by Claude Fable 5 with Claude Code

@0yi0

0yi0 commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

Review Findings

Medium: test/plugins/wasi_logging is imported without the upstream target it depends on

PR #44 imports test/plugins/wasi_logging, but the corresponding upstream implementation is no longer under plugins/.

In current WasmEdge, wasi_logging is wired as a temporary built-in plugin:

  • Implementation lives under lib/plugin/wasi_logging
  • Headers live under include/plugin/wasi_logging
  • lib/plugin/CMakeLists.txt adds wasi_logging and links wasmedgePluginWasiLogging into wasmedgePlugin
  • plugins/CMakeLists.txt only contains a placeholder under WASMEDGE_PLUGIN_WASI_LOGGING
  • test/plugins/CMakeLists.txt still unconditionally adds test/plugins/wasi_logging

This PR’s import filter includes test/plugins, but excludes lib/plugin and include/plugin, so the test is imported while the wasmedgePluginWasiLogging target is not. Once test/plugins is wired into this repository’s build, test/plugins/wasi_logging/CMakeLists.txt will fail because it references wasmedgePluginWasiLogging.

Suggested resolution:

  • Exclude or disable test/plugins/wasi_logging until wasmedgePluginWasiLogging exists in this repository, or
  • Track this explicitly as a required adjustment in the follow-up build-system PR, or
  • If this repo intends to carry built-in plugin sources too, import the matching lib/plugin/wasi_logging and include/plugin/wasi_logging sources.

Given the stated scope of this PR is plugins/, plugin tests, thirdparty/wasi_crypto, and selected utility paths, I would prefer keeping import fidelity here and ensuring the follow-up build-system PR does not enable test/plugins/wasi_logging until the missing target is addressed.

Low: several imported utility scripts use invalid bash -v checks

A few imported scripts use patterns like:

if [[ ! -v "${CMAKE_BUILD_TYPE}" ]]; then

[[ -v ... ]] expects a variable name, not the variable’s value. If the variable is set, this checks whether another variable named by that value exists and can overwrite user-provided settings with defaults.

Affected examples include:

  • utils/wasi-nn/build-wasinn-ubuntu-openvino.sh
  • utils/wasi-nn/install-openvino.sh
  • utils/wasi-nn/install-openvino-genai.sh

Suggested fix when these scripts are touched or enabled:

if [[ ! -v CMAKE_BUILD_TYPE ]]; then

or:

: "${CMAKE_BUILD_TYPE:=Release}"

Non-blocking checks

I also checked the import shape and did not find other scope/hygiene issues:

  • Changed top-level paths are limited to plugins, test, thirdparty, and utils
  • The branch is linear relative to main with 623 commits and no merge commits
  • No binary files were detected in the PR diff
  • No unusually large blobs were found beyond normal generated/test source files
  • Secret-pattern hits were limited to static crypto test vectors in test/plugins/wasi_crypto/asymmetric.cpp

Reviewed by OpenCode, model openai/gpt-5.5.

@0yi0

0yi0 commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

Re: review findings — both verified, and tracked per the suggested resolution (import fidelity kept; no change to this branch).

Medium — test/plugins/wasi_logging without its target: confirmed.

  • This branch: plugins/ contains 15 plugins, no wasi_logging; test/plugins/CMakeLists.txt:10 unconditionally add_subdirectory(wasi_logging); test/plugins/wasi_logging/CMakeLists.txt references wasmedgePluginWasiLogging at lines 9 and 15.
  • Upstream master: lib/plugin/wasi_logging exists; plugins/wasi_logging is 404 — so no filter path could have brought the target along.

Resolution: tracked as a checklist item in #24 (the issue that ports test/plugins/CMakeLists.txt) — wasi_logging tests stay disabled until the target exists in this repo. No lib/plugin / include/plugin import: built-in plugin sources are outside this repo's scope (#22 migrates the plugins/ tree only).

Low — bash [[ ! -v "${VAR}" ]]: confirmed, 7 occurrences across exactly the three named utils/wasi-nn scripts. Tracked in #31 (the issue that enables those scripts), with the suggested [[ ! -v VAR ]] / : "${VAR:=default}" fix.

The PR description's Non-goals / afterwards section now records both findings and where they're tracked.


🤖 Generated by Claude Fable 5 with Claude Code

@0yi0 0yi0 merged commit 18ce2cd into main Jun 12, 2026
1 of 3 checks passed
@0yi0 0yi0 deleted the import/wasmedge-master branch June 12, 2026 08:18
@0yi0

0yi0 commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator Author

Resolution summary (post-merge record)

Merged 2026-06-12 by CLI fast-forward. How each option set from the options comment resolved:

All leftovers from this PR, plus an import-guidelines doc capturing the verified recipe end-to-end, are consolidated in #45.


🤖 Generated by Claude Fable 5 with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: import plugin sources and tests from WasmEdge core with git history