Skip to content

[rowbinary] readRows/streamRowBatches silently drop all rows when every column is zero-width (Tuple()) #1000

Description

@claude

Description

@clickhouse/rowbinary (skills/clickhouse-js-node-rowbinary) infers "no more rows" from "no more bytes":

// skills/clickhouse-js-node-rowbinary/src/readers/rows.ts:46
while (state.pos < state.buf.length) {
  const row = readRow(state);
  ...
}

Tuple() occupies zero bytes in RowBinary. A result set whose columns are all zero-width therefore has zero-byte rows, and the cursor-vs-buffer-length probe cannot distinguish N such rows from end-of-stream. Every row is dropped silently — no exception, no warning, readRows returns []. streamRowBatches (src/readers/stream.ts) is built on readRows, so the streaming path silently yields nothing too.

Server evidence (the wire really is 0 bytes for 3 rows, so the count is not recoverable from the byte stream):

$ curl -s 'http://localhost:8123/?query=SELECT tuple() FROM numbers(3) FORMAT RowBinary' | wc -c
0
$ curl -s 'http://localhost:8123/?query=SELECT count() FROM (SELECT tuple() FROM numbers(3))'
3

Contrast case that already works and must not regress: as soon as one non-zero-width column is present (SELECT tuple(), toUInt8(5)), rows have non-zero byte length and the driver is correct.

Related, and currently masking the above on the header-driven path: Tuple() cannot be compiled at all. astToReader routes the empty tuple into dataTypeReader, which falls through to the NULLARY lookup and throws RowBinaryTypeError: unsupported RowBinary type: Tuple (src/readers/compile.ts:204) — even when a normal column sits next to it. So compileRowBinaryWithNamesAndTypes throws rather than under-reporting; the silent loss is reachable through hand-written / codegen'd row readers, which is the package's primary documented usage.

The main clients (@clickhouse/client, @clickhouse/client-web) are not affected: they read newline-framed text formats (JSONEachRow etc.), where SELECT tuple() FROM numbers(3) yields three {"()":[]} lines.

ClickHouse server version

26.7.3.19 (verified against a running server at http://localhost:8123).

Reproduction

Vitest, run from skills/clickhouse-js-node-rowbinary (uses the package's own tests/clickhouse.ts helper):

import { describe, expect, it } from "vitest";
import { query } from "./clickhouse.js";
import { Cursor } from "../src/readers/core.js";
import { readRows } from "../src/readers/rows.js";
import { readTuple } from "../src/readers/composite.js";
import { streamRowBatches } from "../src/readers/stream.js";
import { compileRowBinaryWithNamesAndTypes } from "../src/readers/rowBinaryWithNamesAndTypes.js";

describe("zero-width (Tuple()) rows", () => {
  it("readRows drops every zero-width row", async () => {
    const buf = await query("SELECT tuple() FROM numbers(3) FORMAT RowBinary");
    expect(buf.length).toBe(0);                       // 3 rows, 0 bytes on the wire
    const rows = readRows(readTuple([]))(new Cursor(buf));
    expect(rows.length).toBe(3);                      // actual: 0
  });

  it("streamRowBatches over the same response yields nothing", async () => {
    const buf = await query("SELECT tuple() FROM numbers(3) FORMAT RowBinary");
    async function* chunks() { yield buf }
    let rows = 0;
    for await (const batch of streamRowBatches(chunks(), readTuple([]), { warnOnSmallChunks: false })) {
      rows += batch.length;
    }
    expect(rows).toBe(3);                             // actual: 0
  });

  it("compiled header path throws on Tuple()", async () => {
    const buf = await query(
      "SELECT tuple() AS t, toUInt8(5) AS n FROM numbers(3) FORMAT RowBinaryWithNamesAndTypes",
    );
    const s = new Cursor(buf);
    const compiled = compileRowBinaryWithNamesAndTypes(s);  // throws
    expect(compiled.readRows(s).length).toBe(3);
  });
});

Actual output:

 ❯ tests/ZeroWidth.test.ts (3 tests | 3 failed)
   × readRows drops every zero-width row
       AssertionError: expected +0 to be 3
   × streamRowBatches over the same response yields nothing
       AssertionError: expected +0 to be 3
   × compiled header path throws on Tuple()
       RowBinaryTypeError: unsupported RowBinary type: Tuple
        ❯ dataTypeReader src/readers/compile.ts:204:13

Expected: 3 rows (each [] / { t: [], n: 5 }) in all three cases.

Suggested fix

  • src/readers/compile.ts — route an argument-less Tuple to tupleReader (or add a zero-width leaf) so Tuple() / Tuple(Tuple()) compile at all; a zero-column tuple reader consumes 0 bytes and returns [].
  • src/readers/rows.ts / src/readers/stream.ts — the row boundary cannot come from byte presence when the compiled row width is a fixed 0. Options, in rough order of intrusiveness (a maintainer design call):
    1. Detect the degenerate case (every column reader is fixed zero-width) and source the row count out of band — e.g. the response's X-ClickHouse-Summary header (read_rows), or document that callers must supply it.
    2. Throw on a zero-width row reader rather than silently returning [] — worse than (1) for users, far better than silent data loss.
    3. Use a framed format (Native block headers carry the row count) for such result sets.
  • Whatever the choice, note it in the readRows docstring, which currently states the byte-exhaustion invariant as unconditional.

Link

Reported for the .NET client as ClickHouse/clickhouse-cs#570 (same defect class: end-of-rows inferred from end-of-bytes). Found by automated cross-client analysis; verified here against ClickHouse 26.7.3.19.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions