Uncompressed streaming inserts emit one HTTP chunk per row, inflating the request body ~50%

[claude]

Description

When insert is given an object-mode stream (or a raw-format stream that pushes small pieces) and request compression is off (the default — compression.request defaults to undefined, see packages/client-common/src/config.ts:393 + normalizeRequestCompression), each row reaches the transport as its own write() on the http.ClientRequest. The body is sent with Transfer-Encoding: chunked, so every row becomes its own HTTP chunk: "<len>\r\n" + row + "\r\n", i.e. ~5 extra wire bytes and one write syscall per row.

Path:

• packages/client-node/src/utils/encoder.ts:26-32 — object-mode values are piped through mapStream(...),
• packages/client-node/src/utils/stream.ts:49-58 — that Transform pushes one string per row,
• packages/client-node/src/connection/socket_pool.ts:395-402 — pipeStream() pipes that stream straight into request when params.request_compression is unset. With compression enabled the gzip/zstd Transform sits in between and coalesces writes into large blocks, which is why this is invisible on the compressed path.

Array inputs are unaffected: encodeValues joins all rows into a single string (encoder.ts:34-37), so the body is one chunk.

Measured on a 1000-row JSONEachRow insert of {x: i} (raw wire bytes captured from a TCP sink, so chunk framing is real, not estimated):

input	request compression	payload	body wire bytes	chunks	overhead
array of 1000 objects	off	9,890 B	9,903 B	1	+0.1%
object-mode stream, 1000 rows	off	9,890 B	14,895 B	1000	+50.6%
object-mode stream, 1000 rows	gzip	9,890 B	2,000 B	2	+0.9%

The inflation scales with how small a row is: narrow rows pay the most, and a wide row still pays ~5 B plus one extra socket write. This is the same defect reported for the .NET client in ClickHouse/clickhouse-cs#524 (there it is one chunk per field on the uncompressed RowBinary path); in clickhouse-js it is one chunk per row, and unlike the .NET client the affected path is the default one, since request compression is off by default here.

ClickHouse server version

26.7.2.59 (the wire measurement itself uses a local TCP sink rather than the server, so the request bytes are captured verbatim; a real server accepts the body either way — this is purely a wire-efficiency issue, not a correctness one).

Reproduction

Self-contained script (run from the repo root with npx tsx); it points the client at a local TCP sink and counts chunked-encoding frames in the captured request body.

import net from "net";
import Stream from "stream";
import { createClient } from "./packages/client-node/src/index.js";

function analyze(raw: Buffer) {
  const sep = raw.indexOf("\r\n\r\n");
  const headers = raw.subarray(0, sep).toString();
  const body = raw.subarray(sep + 4);
  if (!/transfer-encoding: chunked/i.test(headers)) {
    return { chunks: 1, payload: body.length, wire: body.length };
  }
  let off = 0, chunks = 0, payload = 0;
  while (off < body.length) {
    const nl = body.indexOf("\r\n", off);
    if (nl < 0) break;
    const size = parseInt(body.subarray(off, nl).toString(), 16);
    if (Number.isNaN(size) || size === 0) break;
    chunks++;
    payload += size;
    off = nl + 2 + size + 2;
  }
  return { chunks, payload, wire: body.length };
}

async function run(label: string, values: () => any, port: number, gzip = false) {
  const captured: Buffer[] = [];
  const server = net.createServer((socket) => {
    socket.on("data", (d) => {
      captured.push(d);
      if (Buffer.concat(captured).includes("0\r\n\r\n")) {
        socket.write("HTTP/1.1 200 OK\r\nContent-Length: 0\r\nConnection: close\r\n\r\n");
        socket.end();
      }
    });
  });
  await new Promise<void>((r) => server.listen(port, r));
  const client = createClient({
    url: `http://localhost:${port}`,
    compression: { request: gzip },
  });
  try {
    await client.insert({ table: "t", values: values(), format: "JSONEachRow" });
  } catch {
    /* the sink returns a minimal response */
  }
  await client.close();
  server.close();
  const a = analyze(Buffer.concat(captured));
  console.log(
    `${label}: chunks=${a.chunks} payload=${a.payload} wire=${a.wire} ` +
      `overhead=${(((a.wire - a.payload) / a.payload) * 100).toFixed(1)}%`,
  );
}

async function main() {
  const rows = Array.from({ length: 1000 }, (_, i) => ({ x: i }));
  await run("array", () => rows, 18201);
  await run("stream", () => Stream.Readable.from(rows, { objectMode: true }), 18202);
  await run("stream + gzip", () => Stream.Readable.from(rows, { objectMode: true }), 18203, true);
}

main();

Expected — the streaming insert should cost roughly what the array insert costs (a handful of chunks, ~0% framing overhead):

array:         chunks=1 payload=9890 wire=9903  overhead=0.1%
stream:        chunks=~1 payload=9890 wire=~9903 overhead=~0.1%
stream + gzip: chunks=2 payload=9890 wire=2000  overhead=0.9%

Actual:

array:         chunks=1    payload=9890 wire=9903  overhead=0.1%
stream:        chunks=1000 payload=9890 wire=14895 overhead=50.6%
stream + gzip: chunks=2    payload=9890 wire=2000  overhead=0.9%

Suggested fix

Give the uncompressed path the same coalescing the compressed path gets for free — insert a buffering stage between the body stream and the request in packages/client-node/src/connection/socket_pool.ts:395-402, e.g. pipe through a Transform that accumulates into a buffer (the compressors effectively use tens of KB) and flushes on threshold and on final, so:

Stream.pipeline(bodyStream, coalesce(), request, callback);

Alternatively (or additionally) batch inside mapStream (packages/client-node/src/utils/stream.ts:49-58) so N encoded rows are pushed as one string. Buffering at the connection level is the more general fix, since it also helps raw-format streams (CSV/TSV/RowBinary) whose producer pushes small pieces. Whichever place it lands, the flush on stream end must happen before request.end() or the tail of the batch is lost.

Link

Relayed from ClickHouse/clickhouse-cs#524 — same root cause (no buffer between the row serializer and the chunked HTTP request stream on the uncompressed path).