Repository navigation
node:stream / string_decoder: multi-byte characters split across chunks are corrupted, 'drain' never fires, ReadableStream is not async-iterable #51
Description
Test I ran
node streams.ts
slice run streams.tsstreams.ts
import { StringDecoder } from "node:string_decoder";
import { Readable, Writable } from "node:stream";
const show = (k: string, v: unknown) => console.log(k.padEnd(46) + JSON.stringify(v));
const withTimeout = <T>(p: Promise<T>, ms = 1500) => Promise.race([p, new Promise<string>((r) => setTimeout(() => r("NEVER SETTLED (" + ms + " ms)"), ms))]);
(async () => {
const euro = Buffer.from("€"); // e2 82 ac
const d = new StringDecoder("utf8");
show("StringDecoder split '€' over 2 writes", [d.write(euro.subarray(0, 1)), d.write(euro.subarray(1)), d.end()]);
const r = new Readable({ read() {} }); r.setEncoding("utf8"); r.push(euro.subarray(0, 2)); r.push(euro.subarray(2)); r.push(null);
let s = ""; for await (const c of r) s += c; show("Readable.setEncoding('utf8') split '€'", s);
show("Writable 'drain' after write() returned false", await withTimeout(new Promise((res) => { const w = new Writable({ highWaterMark: 2, write(_c, _e, cb) { setTimeout(cb, 1); } }); const ret = w.write("abc"); w.once("drain", () => res("drain fired; write() returned " + ret)); })));
show("for await (chunk of ReadableStream)", await withTimeout((async () => { const rs = new ReadableStream({ start(c) { c.enqueue("a"); c.enqueue("b"); c.close(); } }); const out: string[] = []; try { for await (const ch of rs as any) out.push(ch); return out; } catch (e: any) { return "THROWS " + e.name + ": " + e.message; } })()));
show("ReadableStream.tee() destructuring", await withTimeout((async () => { try { const [x, y] = new ReadableStream({ start(c) { c.enqueue(1); c.close(); } }).tee(); return [(await x.getReader().read()).value, (await y.getReader().read()).value]; } catch (e: any) { return "THROWS " + e.name + ": " + e.message; } })()));
show("Readable.toWeb(Readable.from([buf])) text", await withTimeout((async () => { try { return await new Response(Readable.toWeb(Readable.from([Buffer.from("w")])) as any).text(); } catch (e: any) { return "THROWS " + e.name + ": " + e.message; } })()));
})();Result
Node v24.22.2:
StringDecoder split '€' over 2 writes ["","€",""]
Readable.setEncoding('utf8') split '€' "€"
Writable 'drain' after write() returned false "drain fired; write() returned false"
for await (chunk of ReadableStream) ["a","b"]
ReadableStream.tee() destructuring [1,1]
Readable.toWeb(Readable.from([buf])) text "w"
Slice v0.10.7:
StringDecoder split '€' over 2 writes ["�","��",""]
Readable.setEncoding('utf8') split '€' "��"
Writable 'drain' after write() returned false "NEVER SETTLED (1500 ms)"
for await (chunk of ReadableStream) "THROWS TypeError: value with type `object` is not iterable"
ReadableStream.tee() destructuring [1,1]
Readable.toWeb(Readable.from([buf])) text ""
What is wrong
StringDecoderdoes not buffer incomplete UTF-8 sequences.€(e2 82 ac) written as two chunks yields U+FFFD characters instead of""then"€".Readable#setEncoding("utf8")has the same defect, so any text stream whose chunk boundary falls inside a multi-byte character is corrupted (HTTP bodies, file streams, child process output).Writablenever emits'drain'afterwrite()returnedfalse. Code following the standard back-pressure pattern (if (!w.write(x)) await once(w, "drain")) waits forever; underslice runthe process then exits 0 with no output because the event loop is empty.for await (const chunk of readableStream)throwsTypeError: value with type 'object' is not iterable.COMPATIBILITY.mdlists "for await...ofasync iteration" for Streams.Readable.toWeb()produces a stream that yields no data (""instead of"w").Readable.toWebis listed as supported.
StringDecoder (write, end) and the Readable / Writable classes are listed as supported in the matrix.
Environment
- Slice
v0.10.7, commitea7c70c, built withcargo +1.96.0 build --release --locked --bin slice - Reference: Node.js
v24.22.2(same script, same machine) - Ubuntu 26.04, Linux 7.0, x86_64
- The same output is produced by the
v0.10.5tag, so this is not a regression in 0.10.6/0.10.7.
Activity
siktec-lab commentedon Nov 11, 2026
Addressed in #66 (
v0.10.10):StringDecoder(node:string_decoder) andReadable.prototype.setEncodingnow buffer incomplete multi-byte UTF-8 (2/3/4-byte codepoints), UTF-16LE (odd byte boundaries and surrogate pairs), and Base64 chunks across.write()calls and flush remaining bytes on.end().WritabletrackswritableLengthagainstwritableHighWaterMark, returnsfalsefrom.write()when the buffer reacheshighWaterMark, and emits'drain'once pending writes complete.ReadableStreamnow implements[Symbol.asyncIterator]and.values(), and queues pendingreader.read()/Response.prototype.text()consumers for async push streams (includingReadable.toWeb(Readable.from(...))).