From 8f904c794c729893f27268433ec22ee185f89a5e Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 20:11:02 +0200 Subject: [PATCH 01/46] Add design notes and plan for the WebSocket proxy Document the plan for `clickhouse-wsproxy`: a standalone WebSocket-to-native-protocol proxy that reuses `Connection` and `FormatFactory` but drops the CLI/`ClientBase` layer, deployed as an edge sidecar for format conversion and mid-query push. Records the design decisions, the build-wiring approach, measured binary size, and the step-by-step plan. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 241 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 241 insertions(+) create mode 100644 TODO.md diff --git a/TODO.md b/TODO.md new file mode 100644 index 000000000000..e70842bfbcfb --- /dev/null +++ b/TODO.md @@ -0,0 +1,241 @@ +# WebSocket proxy on top of the native-protocol client layer + +## Business rationale (the actual ROI) + +The value is **not** raw performance or a cleaner API — it is **deployment velocity**. +Getting changes into the managed cloud is slow and expensive. A thin, self-contained +proxy that lives at the edge (as a sidecar next to the app) lets us move logic that +would otherwise require a server-side/cloud change out to a place we can ship quickly. + +Concretely, the proxy earns its keep by: + +- **Edge-side format conversion** — offload the format-conversion CPU from the managed + cluster to the sidecar, and let apps speak whatever format is convenient (`JSONEachRow`, + `CSV`, `Arrow`, `Parquet`, …) without a cloud-side change. +- **Mid-query push over WebSocket** — stream `Progress`, `ProfileEvents`, and `Log` + packets to the app as they arrive, which the HTTP interface cannot stream cleanly. +- **Remote backend** — unlike the in-tree web terminal (which talks to a + `LocalConnection`), the proxy talks to a *remote* server over the native protocol. + +If (edge conversion) and (mid-query push) are not the point, ROI shrinks — at that point +the server's existing HTTP interface (`FORMAT` / `default_format`) already does +server-side conversion, so keep this justification explicit. + +## Goal + +**Decided form factor:** a **standalone binary** under `programs/wsproxy/`, deployed as an +edge sidecar. Preference: keep it a *separate* binary rather than wiring it into the +multi-call `clickhouse` dispatch (see build note in the plan). + +**Decided session model:** **strict 1:1 sidecar** — one WS session = one native +`Connection` to a fixed backend, thread-per-session blocking IO. No pooling, no replica +load-balancing. + +A new native-protocol WebSocket proxy, built **in-tree**, that: + +- accepts WebSocket connections from apps, +- opens one native-protocol `Connection` per session to a remote ClickHouse server, +- maps native protocol packets ↔ WebSocket frames, +- does format conversion at the edge in both directions, +- **drops** the entire CLI/REPL surface (`ClientBase`, Replxx, `Suggest`, progress + bars, pager, history, autocomplete). + +## Design decisions + +### Cut line — move it down a layer + +Reuse the layer *below* the client program, not the client program itself. + +- **OUT (the CLI):** `programs/client/Client.cpp`, `src/Client/ClientBase.cpp`, + Replxx line editing, terminal progress, pager, history, `Suggest.cpp`. +- **IN (the machinery):** + - `src/Client/Connection.{h,cpp}` — native protocol: handshake, revision negotiation, + `sendQuery`, `sendData`, `sendCancel`, `receivePacket`, compression, TLS. + - `src/Formats/` + `FormatFactory` — full format matrix, both directions (output + formats for results, input formats for INSERT payloads). + - `src/DataTypes`, `src/Columns`, `src/Core` (`Block`, `Protocol.h`, `Settings`), + `src/IO`, `src/Compression` — the type system and codecs underneath the formats. + +### Existence proof already in the tree — use it as the primary template + +`src/Server/WebTerminalRequestHandler.cpp` already ships a WebSocket endpoint inside +`clickhouse-server` (the "web terminal": browser → WS → embedded client). It resolves +most open questions and corrects the earlier guesses: + +- **Front door is hand-rolled RFC 6455, NOT `Poco::Net::WebSocket`.** Frame read/write, + masking, fragment reassembly, control frames, and close codes are implemented directly + on `response.getSocket()` (`WebTerminalRequestHandler.cpp:85-258`). Copy those framing + helpers rather than pulling in `Poco::Net::WebSocket`. +- **Front door is an HTTP request handler, not a bespoke server.** Registered in + `HTTPHandlerFactory.cpp:450` via `HandlingRuleHTTPHandlerFactory` + on `/webterminal`. This is the canonical way to accept and upgrade a WS connection in + this codebase. + +### Template set + +- **Upstream (WS front door):** `WebTerminalRequestHandler` + the `HTTPHandlerFactory` + registration. +- **Downstream (native protocol driving):** `clickhouse-benchmark` + (`programs/benchmark/Benchmark.cpp`) — drives `Connection` directly, multi-connection, + with no `ClientBase`. NOTE: benchmark is a *pure client* — it has no accept socket, so + it is only a template for the downstream half. + +The proxy is essentially `WebTerminalRequestHandler` with the backend swapped: keep the +WS side, delete PTY + `ClientEmbedded` (`ClientEmbedded.h:17`, a `ClientBase` subclass) + +`LocalConnection`, and substitute a **remote `Connection` + `FormatFactory`**. + +### Concurrency model — single thread, one `poll` loop per session + +Follow the web terminal, not a two-thread-per-session design. +`WebTerminalRequestHandler.cpp:685-695` multiplexes `{ws socket fd, pty fd}` in a single +`poll(fds, 2, 100)` loop. For the proxy this becomes `poll({ws socket, Connection socket})`: + +- One thread per session (consistent with the server's own thread-per-connection TCP model). +- Cancellation falls out for free: a frame arriving on the WS fd mid-query → + `Connection::sendCancel` (`Connection.h:128`), then drain. +- `Connection::poll(timeout)` (`Connection.h:136`) and `setAsyncCallback` + (`Connection.h:172`) exist precisely for this interleaving — it is how the server + watches remote connections during distributed queries. + +## Packet ↔ frame mapping (to be detailed) + +Per-session loop: accept WS → open one `Connection` → on query, `sendQuery` and loop +`receivePacket`, mapping packets to WS frames: + +- `Data` blocks → run through an output format → binary frames. +- `Progress` / `ProfileEvents` / `Log` → small control frames. +- `Exception` / `EndOfStream` → terminators. +- **INSERT dance:** server replies to INSERT with a header block describing the table + structure → instantiate an *input* format for whatever the app sent + (`JSONEachRow`, `CSV`, …) → parse into `Block`s → `sendData` natively. Mirror the + logic in `ClientBase::processInsertQuery`. +- **Cancel:** WS close frame or cancel frame → `Connection::sendCancel` → drain. + +## What we inherit for free + +Complete type coverage (`LowCardinality`, `Dynamic`, `Variant`, `JSON`, +`AggregateFunction` states — everything, as long as we rebase), the full format matrix in +both directions, native LZ4/ZSTD, TLS, and protocol revision negotiation so moderate +proxy/server version skew just works. + +## The bill / risks + +- **Marrying the ClickHouse build:** recent Clang only, ~100 submodules, serious RAM, + long builds, and CI has to carry it. +- **Large binary** — `FormatFactory` alone drags in Arrow/Parquet/ORC/capnproto/protobuf/ + Avro; expect a few hundred MB. Fine for a sidecar; not a lightweight artifact. +- **Rebase treadmill** — monthly upstream releases. Mitigation: keep the handler + self-contained in its own directory with near-zero diffs to shared code, so rebases + stay mechanical. Watch for `Connection` and `FormatFactory` API churn. + +## Rejected alternative + +`clickhouse-cpp` (standalone C++ client library): clean CMake, none of the monorepo pain, +but has its own partial type-system reimplementation and **zero** format machinery. Since +edge-side format conversion is the whole point, it defeats the purpose. Only interesting +if we ever need just two or three formats. + +## Step 2 status — DONE (branch `wsproxy-skeleton`) + +Real WebSocket endpoint implemented and validated: +- `WebSocketFrames.{h,cpp}` — reusable RFC 6455 framing (lifted/adapted from + `WebTerminalRequestHandler`): handshake accept-key, masked read / unmasked write, + fragmentation, control frames, size caps. +- `WsProxyHandler.{h,cpp}` — HTTP entry: non-WS → info page; WS upgrade → handshake then + an echo loop (the placeholder the step-3 session loop replaces). No `Origin` check + (clients are apps, not browsers). +- `WsProxy.cpp` — blocks `SIGINT`/`SIGTERM` in the main thread before the server spawns + workers, so Poco's `waitForTerminationRequest` `sigwait` catches them → **clean exit 0 + shutdown** (verified). `BaseDaemon` re-base deferred (not needed for the signal fix). + +Validated with a dependency-free raw-socket WS client (`tmp/ws_test.py`): handshake + +verified `Sec-WebSocket-Accept`, text echo, 200-byte binary echo, ping/pong, close. + +Test-hygiene note: background launches detach and orphan (PPID 1) if not killed by exact +binary PID; `pgrep -f` also matches the launch shell. Always test one instance, select the +PID whose command starts with `./build`, and confirm zero processes between runs. + +## Plan + +1. **Skeleton.** Standalone `programs/wsproxy/` binary. **DONE — builds, links, runs-to-listen.** + Implemented on branch `wsproxy-skeleton`: `programs/wsproxy/WsProxy.cpp` (a + `Poco::Util::ServerApplication` that creates the global `Context`, calls + `registerFormats`, and hosts a `DB::HTTPServer` with a stub handler on port 9010) plus + `programs/wsproxy/CMakeLists.txt` (`clickhouse_add_executable`, no multi-call wiring). + Compiled with zero warnings on our code; linked cleanly. + + **Measured binary size (arm64 dev build):** 424 MB with debug symbols, **306 MB + stripped**, of which ~241 MB is actual code (`__text`). Confirms *separate ≠ smaller*: + this is roughly the same order as the 728 MB multi-call `clickhouse` dev binary, and a + ~100 MB target is not reachable without fighting the monorepo's coupling (trimming + `registerFormats` would claw back tens of MB, not ~200, because `Connection` mandatorily + pulls in `DataTypes`/`Columns`/`IO`/`Compression`/`Core`). + + **RESOLVED — clean, not a fight (build wiring).** + + **Smoke test (runtime):** launches, binds port 9010 instantly, answers HTTP `GET` with + the stub `200` (Context + `registerFormats` + `DB::HTTPServer` all init fine at runtime). + **Known teardown gap:** `SIGTERM` kills the process hard (exit 144), our `"Shutting + down"` path never runs — because raw `Poco::Util::ServerApplication` does not mask + signals across the `HTTPServer` worker threads the way `BaseDaemon` does. Fix in step 2: + base the program on `BaseDaemon` (or block `SIGINT`/`SIGTERM` process-wide before + starting the server). + Precedent: `BUILD_STANDALONE_KEEPER` (`programs/keeper/CMakeLists.txt:26-44`) already + produces a *separate* executable via `clickhouse_add_executable(clickhouse-keeper …)` + linking real libraries — exactly the pattern we need. `clickhouse_add_executable` is a + project macro (top-level `CMakeLists.txt:545`), so it is available to us. + + Concrete recipe for a separate binary that stays OUT of multi-call dispatch: + - Create `programs/wsproxy/` with `WsProxy.cpp` (+ WS/session sources) and a + `CMakeLists.txt`. + - In that `CMakeLists.txt`, call **`clickhouse_add_executable(clickhouse-wsproxy )`** + directly and `target_link_libraries(... dbms clickhouse_common_config + clickhouse_functions daemon …)`. We can skip the `clickhouse_program_add` / + `clickhouse-wsproxy-lib` indirection that keeper keeps — that indirection only exists + to feed multi-call, which we are deliberately not joining. + - Add a single `add_subdirectory(wsproxy)` line to `programs/CMakeLists.txt`. + - **Do NOT** add a `mainEntryClickHouseWsproxy` to `programs/main.cpp` and **do NOT** + add a `clickhouse_program_install(clickhouse-wsproxy …)` line. Omitting both is the + entire mechanism that keeps the binary separate. Result: zero diff to shared dispatch + code (good for the rebase treadmill). + - Optionally guard behind `option(ENABLE_CLICKHOUSE_WSPROXY …)` mirroring the keeper + flags, so CI can skip it. + + Crib global `Context` init + `registerFormats` from `Client.cpp` / `LocalServer.cpp`. + + **Caveat — separate ≠ smaller.** `Connection`, `ClientBase`, and `FormatFactory` all + compile into the monolithic `dbms` library (`src/Client/CMakeLists.txt` only builds + examples; the sources are globbed into `dbms`). We must link `dbms`, so the wsproxy + binary will be roughly the same size as the full `clickhouse` binary (hundreds of MB). + Keeper's standalone binary is small *because it avoids `dbms`* — we cannot. Separate-ness + buys a distinct ELF and independent deployment, not a lighter artifact. Marginal build + cost over multi-call is one extra link of `dbms` (our own sources are tiny). +2. **WS front door.** Lift the RFC 6455 framing helpers from `WebTerminalRequestHandler` + (handshake, `readWebSocketFrame`, `sendWebSocketFrame`, close codes, masking, + fragmentation). Reuse its Origin / auth / size-cap hardening. +3. **Backend `Connection`.** Open one native `Connection` per session (template on + `clickhouse-benchmark`). Handshake + revision negotiation. +4. **Session loop.** Single `poll({ws fd, connection fd})` loop. Map query → `sendQuery`, + `receivePacket` → WS frames per the mapping above. +5. **Output formats.** Route `Data` blocks through `FormatFactory::getOutputFormat` into + binary frames; the app picks the format. +6. **INSERT path.** Header block → input format → `Block`s → `sendData`. Mirror + `ClientBase::processInsertQuery`. +7. **Cancellation + teardown.** WS close/cancel → `Connection::sendCancel` → drain → + clean close frame (RFC 6455 correct). +8. **Tests.** Integration test modeled on `tests/integration/test_webterminal`. + +## Resolved decisions + +- **Form factor:** standalone `programs/wsproxy/` binary; prefer a separate executable over + multi-call dispatch (build feasibility to be confirmed in step 1). +- **Session model:** strict 1:1 (one WS session ↔ one native `Connection`), thread-per-session. + +## Open questions + +- ~~Separate-binary feasibility~~ — **RESOLVED (step 1):** clean via + `clickhouse_add_executable`, precedent is `BUILD_STANDALONE_KEEPER`. See plan step 1. +- **Auth model:** in-band first-frame auth like the web terminal, or pass-through of the + app's credentials to the backend `Connection`? +- **Backend target config:** fixed host/port from config/CLI (sidecar assumption), or per-session + target from the WS handshake? From 72a9fdfd671a9a892eae0f8990ef571c17cea1ec Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 20:11:08 +0200 Subject: [PATCH 02/46] Add reusable RFC 6455 WebSocket framing module Factor the WebSocket frame primitives into a standalone module, adapted from `WebTerminalRequestHandler`: the handshake accept-key (`computeWebSocketAccept`), masked-read / unmasked-write framing (`readWebSocketFrame`, `sendWebSocketFrame`), fragmentation, control frames, and a payload-size cap. The proxy session loop will drive these primitives directly, so they live apart from the HTTP handler. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/WebSocketFrames.cpp | 225 +++++++++++++++++++++++++++ programs/wsproxy/WebSocketFrames.h | 61 ++++++++ 2 files changed, 286 insertions(+) create mode 100644 programs/wsproxy/WebSocketFrames.cpp create mode 100644 programs/wsproxy/WebSocketFrames.h diff --git a/programs/wsproxy/WebSocketFrames.cpp b/programs/wsproxy/WebSocketFrames.cpp new file mode 100644 index 000000000000..f243e2f00e32 --- /dev/null +++ b/programs/wsproxy/WebSocketFrames.cpp @@ -0,0 +1,225 @@ +#include + +#include + +#include + +#include + + +namespace DB::WsProxy +{ + +namespace +{ + +/// Monotonic clock in nanoseconds, self-contained so the framing layer has no +/// dependency beyond libc. Used only to enforce absolute read deadlines. +UInt64 monotonicNs() +{ + struct timespec ts + { + }; + clock_gettime(CLOCK_MONOTONIC, &ts); + return static_cast(ts.tv_sec) * 1'000'000'000ULL + static_cast(ts.tv_nsec); +} + +/// Send all bytes to the socket, handling partial writes. +void sendAllBytes(Poco::Net::StreamSocket & socket, const char * data, size_t len) +{ + size_t total = 0; + while (total < len) + { + int sent = socket.sendBytes(data + total, static_cast(len - total)); + if (sent <= 0) + throw Poco::IOException("Failed to send bytes to WebSocket"); + total += static_cast(sent); + } +} + +/// Read exactly `n` bytes. If `deadline_ns` is non-zero, abort when the +/// monotonic clock passes it. +bool readExact(Poco::Net::StreamSocket & socket, char * buf, size_t n, UInt64 deadline_ns) +{ + size_t total = 0; + while (total < n) + { + if (deadline_ns && monotonicNs() > deadline_ns) + return false; + int received = socket.receiveBytes(buf + total, static_cast(n - total)); + if (received <= 0) + return false; + total += static_cast(received); + } + return true; +} + +} + +String computeWebSocketAccept(const String & key) +{ + const String magic = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; + Poco::SHA1Engine sha1; + sha1.update(key + magic); + auto digest = sha1.digest(); + return base64Encode(String(reinterpret_cast(digest.data()), digest.size())); +} + +bool isValidWebSocketKey(const String & key) +{ + if (key.empty() || key.size() > 128) + return false; + try + { + return base64Decode(key).size() == 16; + } + catch (...) /// Ok: malformed base64 is just an invalid key; the caller maps it to a 400. + { + return false; + } +} + +void sendWebSocketFrame(Poco::Net::StreamSocket & socket, uint8_t opcode, const char * data, size_t len) +{ + uint8_t header[10]; + size_t header_len = 2; + + header[0] = 0x80 | opcode; /// FIN + opcode + + if (len < 126) + { + header[1] = static_cast(len); + } + else if (len < 65536) + { + header[1] = 126; + header[2] = static_cast((len >> 8) & 0xFF); + header[3] = static_cast(len & 0xFF); + header_len = 4; + } + else + { + header[1] = 127; + for (int i = 0; i < 8; ++i) + header[2 + i] = static_cast((len >> (56 - 8 * i)) & 0xFF); + header_len = 10; + } + + /// Combine header and payload into a single send to avoid partial frame writes. + String buf; + buf.reserve(header_len + len); + buf.append(reinterpret_cast(header), header_len); + if (len > 0) + buf.append(data, len); + sendAllBytes(socket, buf.data(), buf.size()); +} + +void sendWebSocketBinary(Poco::Net::StreamSocket & socket, const char * data, size_t len) +{ + sendWebSocketFrame(socket, Opcode::Binary, data, len); +} + +void sendWebSocketText(Poco::Net::StreamSocket & socket, const String & text) +{ + sendWebSocketFrame(socket, Opcode::Text, text.data(), text.size()); +} + +void sendWebSocketClose(Poco::Net::StreamSocket & socket, uint16_t code, const String & reason) +{ + String payload; + payload.push_back(static_cast((code >> 8) & 0xFF)); + payload.push_back(static_cast(code & 0xFF)); + payload.append(reason); + sendWebSocketFrame(socket, Opcode::Close, payload.data(), payload.size()); +} + +WebSocketFrame readWebSocketFrame(Poco::Net::StreamSocket & socket, UInt64 deadline_ns, uint64_t max_payload_size) +{ + WebSocketFrame frame; + uint8_t header[2]; + + if (!readExact(socket, reinterpret_cast(header), 2, deadline_ns)) + return frame; + + frame.fin = (header[0] & 0x80) != 0; + frame.opcode = header[0] & 0x0F; + + /// RSV1/RSV2/RSV3 must be zero (no extensions negotiated). + if (header[0] & 0x70) + { + frame.protocol_error = true; + return frame; + } + + /// Reject reserved opcodes per RFC 6455 section 5.2. + if ((frame.opcode >= 0x03 && frame.opcode <= 0x07) || frame.opcode >= 0x0B) + { + frame.protocol_error = true; + return frame; + } + + bool masked = (header[1] & 0x80) != 0; + uint64_t payload_len = header[1] & 0x7F; + + /// RFC 6455: client-to-server frames MUST be masked. + if (!masked) + { + frame.protocol_error = true; + return frame; + } + + /// Control frames (opcode >= 0x08) must have FIN set and payload <= 125. + if (frame.opcode >= 0x08) + { + if (!frame.fin || payload_len > 125) + { + frame.protocol_error = true; + return frame; + } + } + + if (payload_len == 126) + { + uint8_t ext[2]; + if (!readExact(socket, reinterpret_cast(ext), 2, deadline_ns)) + return frame; + payload_len = (static_cast(ext[0]) << 8) | ext[1]; + } + else if (payload_len == 127) + { + uint8_t ext[8]; + if (!readExact(socket, reinterpret_cast(ext), 8, deadline_ns)) + return frame; + payload_len = 0; + for (const auto & byte : ext) + payload_len = (payload_len << 8) | byte; + } + + /// Cap the payload before allocating, so a crafted length header cannot + /// force a huge allocation. Distinct from `protocol_error` so the caller + /// can close with the dedicated 1009 code. + if (payload_len > max_payload_size) + { + frame.message_too_big = true; + return frame; + } + + uint8_t mask_key[4] = {}; + if (!readExact(socket, reinterpret_cast(mask_key), 4, deadline_ns)) + return frame; + + frame.payload.resize(payload_len); + if (payload_len > 0) + { + if (!readExact(socket, frame.payload.data(), payload_len, deadline_ns)) + return frame; + + for (uint64_t i = 0; i < payload_len; ++i) + frame.payload[i] ^= static_cast(mask_key[i % 4]); + } + + frame.valid = true; + return frame; +} + +} diff --git a/programs/wsproxy/WebSocketFrames.h b/programs/wsproxy/WebSocketFrames.h new file mode 100644 index 000000000000..a2b8b7045aeb --- /dev/null +++ b/programs/wsproxy/WebSocketFrames.h @@ -0,0 +1,61 @@ +#pragma once + +#include + +#include + +/// Low-level RFC 6455 WebSocket framing over a raw stream socket. +/// +/// Lifted and adapted from `src/Server/WebTerminalRequestHandler.cpp`, factored +/// into a reusable module because the proxy session loop (native protocol <-> +/// WebSocket) will drive these primitives directly. Server-to-client frames are +/// never masked; client-to-server frames MUST be masked (enforced on read). + +namespace DB::WsProxy +{ + +/// WebSocket opcodes (RFC 6455 section 5.2). +namespace Opcode +{ + static constexpr uint8_t Continuation = 0x0; + static constexpr uint8_t Text = 0x1; + static constexpr uint8_t Binary = 0x2; + static constexpr uint8_t Close = 0x8; + static constexpr uint8_t Ping = 0x9; + static constexpr uint8_t Pong = 0xA; +} + +struct WebSocketFrame +{ + uint8_t opcode = 0; + bool fin = false; + String payload; + /// A fully and correctly read frame. + bool valid = false; + /// Set on an RFC 6455 protocol violation; caller closes with 1002. + bool protocol_error = false; + /// Set when the advertised payload exceeds `max_payload_size`; caller closes with 1009. + bool message_too_big = false; +}; + +/// Compute the `Sec-WebSocket-Accept` response value per RFC 6455 section 4.2.2. +String computeWebSocketAccept(const String & key); + +/// Validate that `Sec-WebSocket-Key` is a base64-encoded 16-byte nonce. +bool isValidWebSocketKey(const String & key); + +/// Send a single unmasked frame with the given opcode. +void sendWebSocketFrame(Poco::Net::StreamSocket & socket, uint8_t opcode, const char * data, size_t len); +void sendWebSocketBinary(Poco::Net::StreamSocket & socket, const char * data, size_t len); +void sendWebSocketText(Poco::Net::StreamSocket & socket, const String & text); +void sendWebSocketClose(Poco::Net::StreamSocket & socket, uint16_t code, const String & reason); + +/// Read a single frame. `deadline_ns == 0` means no absolute deadline (per-read +/// timeouts are still governed by the socket's receive timeout). `max_payload_size` +/// caps the advertised payload length before any allocation. +WebSocketFrame readWebSocketFrame( + Poco::Net::StreamSocket & socket, + UInt64 deadline_ns = 0, + uint64_t max_payload_size = 16 * 1024 * 1024); + +} From 2685274a36d80059832c18c8e6177e4688ce933b Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 20:11:17 +0200 Subject: [PATCH 03/46] Add standalone clickhouse-wsproxy binary with a WebSocket endpoint Introduce `clickhouse-wsproxy`, a separate executable deliberately kept out of the multi-call `clickhouse` dispatch: it has its own `clickhouse_add_executable`, no `mainEntryClickHouseWsProxy` in `programs/main.cpp`, and no `clickhouse_program_install` line. This keeps the diff to shared dispatch code at zero (mirrors the `BUILD_STANDALONE_KEEPER` pattern). Linking `dbms` is unavoidable, so the binary is roughly the size of the full `clickhouse` binary. At startup it creates the global `Context` and calls `registerFormats`, then serves a WebSocket endpoint through ClickHouse's own `HTTPServer`: non-upgrade requests get an info page; a WebSocket upgrade completes the RFC 6455 handshake and then echoes frames (a placeholder for the native-protocol session loop added later). `SIGINT`/`SIGTERM` are blocked in the main thread before the server spawns worker threads, so the workers inherit the mask and Poco's `waitForTerminationRequest` consumes the signal via `sigwait`, giving a clean shutdown instead of the default terminate disposition firing on a worker thread. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/CMakeLists.txt | 8 + programs/wsproxy/CMakeLists.txt | 30 ++++ programs/wsproxy/WsProxy.cpp | 135 ++++++++++++++ programs/wsproxy/WsProxyHandler.cpp | 266 ++++++++++++++++++++++++++++ programs/wsproxy/WsProxyHandler.h | 25 +++ 5 files changed, 464 insertions(+) create mode 100644 programs/wsproxy/CMakeLists.txt create mode 100644 programs/wsproxy/WsProxy.cpp create mode 100644 programs/wsproxy/WsProxyHandler.cpp create mode 100644 programs/wsproxy/WsProxyHandler.h diff --git a/programs/CMakeLists.txt b/programs/CMakeLists.txt index c44a71c1a85a..bd1f7fa9ad02 100644 --- a/programs/CMakeLists.txt +++ b/programs/CMakeLists.txt @@ -25,6 +25,8 @@ option (ENABLE_CLICKHOUSE_KEEPER_CONVERTER "Util allows to convert ZooKeeper log option (ENABLE_CLICKHOUSE_KEEPER_CLIENT "ClickHouse Keeper Client" ${ENABLE_CLICKHOUSE_ALL}) +option (ENABLE_CLICKHOUSE_WSPROXY "Standalone WebSocket proxy (native protocol <-> WebSocket, edge format conversion)" ${ENABLE_CLICKHOUSE_ALL}) + if (NOT ENABLE_NURAFT) # RECONFIGURE_MESSAGE_LEVEL should not be used here, # since ENABLE_NURAFT is set to OFF for FreeBSD and Darwin. @@ -126,6 +128,12 @@ if (ENABLE_CLICKHOUSE_KEEPER) add_subdirectory (keeper) endif() +# Standalone binary with its own `main` and `add_executable` (see wsproxy/CMakeLists.txt), +# so it is added here rather than via the multi-call `clickhouse_program_install` list below. +if (ENABLE_CLICKHOUSE_WSPROXY) + add_subdirectory (wsproxy) +endif() + if (ENABLE_CLICKHOUSE_SELF_EXTRACTING AND NOT ENABLE_DUMMY_LAUNCHERS) add_subdirectory (self-extracting) endif () diff --git a/programs/wsproxy/CMakeLists.txt b/programs/wsproxy/CMakeLists.txt new file mode 100644 index 000000000000..e9d91ad4daa1 --- /dev/null +++ b/programs/wsproxy/CMakeLists.txt @@ -0,0 +1,30 @@ +set (CLICKHOUSE_WSPROXY_SOURCES + WsProxy.cpp + WsProxyHandler.cpp + WebSocketFrames.cpp +) + +set (CLICKHOUSE_WSPROXY_LINK + PRIVATE + clickhouse_common_config + clickhouse_common_io + clickhouse_functions + dbms +) + +# A standalone binary, deliberately NOT wired into the multi-call `clickhouse` +# dispatch: there is no `mainEntryClickHouseWsProxy` in programs/main.cpp and no +# `clickhouse_program_install` line for it. Omitting both is what keeps it a +# separate executable with its own `main`, which in turn keeps the diff to +# shared dispatch code at zero. This mirrors the standalone-keeper pattern +# (`BUILD_STANDALONE_KEEPER`), except we do not also produce a multi-call lib. +# +# Note: linking `dbms` is unavoidable (it is where `Connection`, `FormatFactory` +# and the HTTP server infrastructure live), so this binary is roughly the size +# of the full `clickhouse` binary. The separate target buys an independently +# deployable artifact, not a smaller one. +clickhouse_add_executable (clickhouse-wsproxy ${CLICKHOUSE_WSPROXY_SOURCES}) +target_link_libraries (clickhouse-wsproxy ${CLICKHOUSE_WSPROXY_LINK}) +# So the program's own headers resolve as angled includes (matches the keeper +# standalone pattern). +target_include_directories (clickhouse-wsproxy PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) diff --git a/programs/wsproxy/WsProxy.cpp b/programs/wsproxy/WsProxy.cpp new file mode 100644 index 000000000000..ba852b4957e1 --- /dev/null +++ b/programs/wsproxy/WsProxy.cpp @@ -0,0 +1,135 @@ +#include +#include + +#include + +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include + +#include + +#include + +#include + + +/// Standalone WebSocket proxy. +/// +/// Deliberately a separate binary, kept out of the multi-call `clickhouse` +/// dispatch (no `mainEntryClickHouseWsProxy`, no `clickhouse_program_install`). +/// +/// Step 2 scope: bring up the global `Context` and the format machinery, then +/// serve a real WebSocket endpoint (`WsProxyHandler`) that completes the RFC +/// 6455 handshake and echoes messages. Step 3 will replace the echo loop with a +/// native-protocol `Connection` to a backend server. +/// +/// Signal handling: we block SIGINT/SIGTERM in the main thread *before* the +/// server spawns any worker threads, so the workers inherit the blocked mask. +/// A process-directed signal then stays pending until `waitForTerminationRequest` +/// consumes it via `sigwait`, giving a clean shutdown instead of the default +/// "terminate" disposition firing on a worker thread. (The heavier alternative +/// is to base this on `BaseDaemon`, which also provides config/logging/crash +/// handling; deferred until the proxy needs those.) + +namespace DB +{ + +namespace +{ + +class WsProxyHandlerFactory : public HTTPRequestHandlerFactory +{ +public: + std::unique_ptr createRequestHandler(const HTTPServerRequest &) override + { + return std::make_unique(); + } +}; + +/// Conservative HTTP limits/timeouts for the skeleton; a later revision should +/// source these from configuration. +class WsProxyHTTPContext : public IHTTPContext +{ +public: + uint64_t getMaxHstsAge() const override { return 0; } + uint64_t getMaxUriSize() const override { return 1024 * 1024; } + uint64_t getMaxFields() const override { return 1'000'000; } + uint64_t getMaxFieldNameSize() const override { return 128 * 1024; } + uint64_t getMaxFieldValueSize() const override { return 128 * 1024; } + uint64_t getMaxRequestHeaderSize() const override { return 8 * 1024 * 1024; } + Poco::Timespan getHeadersReadTimeout() const override { return {30, 0}; } + Poco::Timespan getReceiveTimeout() const override { return {30, 0}; } + Poco::Timespan getSendTimeout() const override { return {30, 0}; } +}; + +} + +class WsProxyServer : public Poco::Util::ServerApplication +{ +protected: + int main(const std::vector &) override + { + LoggerPtr log = getLogger("WsProxy"); + + /// Block termination signals before any worker threads are created, so + /// they inherit the mask and `waitForTerminationRequest` can consume the + /// signal cleanly. See the file header for the rationale. + sigset_t sigset{}; + sigemptyset(&sigset); + sigaddset(&sigset, SIGINT); + sigaddset(&sigset, SIGTERM); + pthread_sigmask(SIG_BLOCK, &sigset, nullptr); + + /// Minimal global state the format machinery needs. + shared_context = Context::createShared(); + global_context = Context::createGlobal(shared_context.get()); + global_context->makeGlobalContext(); + global_context->setApplicationType(Context::ApplicationType::SERVER); + + registerFormats(); + + static constexpr UInt16 port = 9010; + Poco::Net::ServerSocket socket(port); + Poco::ThreadPool server_pool(/* minCapacity= */ 1, /* maxCapacity= */ 16); + Poco::Net::HTTPServerParams::Ptr params(new Poco::Net::HTTPServerParams); + + HTTPServer server( + std::make_shared(), + std::make_shared(), + server_pool, + socket, + params); + + server.start(); + LOG_INFO(log, "clickhouse-wsproxy listening on port {}", port); + + waitForTerminationRequest(); + + LOG_INFO(log, "Shutting down"); + server.stop(); + return Application::EXIT_OK; + } + +private: + SharedContextHolder shared_context; + ContextMutablePtr global_context; +}; + +} + +int main(int argc, char ** argv) +{ + DB::WsProxyServer app; + return app.run(argc, argv); +} diff --git a/programs/wsproxy/WsProxyHandler.cpp b/programs/wsproxy/WsProxyHandler.cpp new file mode 100644 index 000000000000..d770be5c69b6 --- /dev/null +++ b/programs/wsproxy/WsProxyHandler.cpp @@ -0,0 +1,266 @@ +#include +#include + +#include +#include + +#include +#include + +#include +#include + +#include +#include + +#include + +#include + + +namespace DB +{ + +using namespace DB::WsProxy; + +namespace +{ + +/// Per RFC 7230 the `Connection` header is a comma-separated token list. Match +/// the exact `upgrade` token rather than a substring (which would also accept +/// unrelated values containing those letters). +bool hasUpgradeToken(const String & connection_header) +{ + String value = connection_header; + std::transform(value.begin(), value.end(), value.begin(), ::tolower); + + size_t pos = 0; + while (pos <= value.size()) + { + size_t comma = value.find(',', pos); + size_t end_pos = (comma == String::npos) ? value.size() : comma; + size_t start = value.find_first_not_of(" \t", pos); + if (start != String::npos && start < end_pos) + { + size_t last = value.find_last_not_of(" \t", end_pos - 1); + if (value.substr(start, last - start + 1) == "upgrade") + return true; + } + if (comma == String::npos) + break; + pos = comma + 1; + } + return false; +} + +} + +void WsProxyHandler::serveInfo(HTTPServerRequest & request, HTTPServerResponse & response) +{ + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_OK); + response.setContentType("text/plain; charset=UTF-8"); + *response.send() + << "clickhouse-wsproxy\n" + << "Requested: " << request.getURI() << "\n" + << "Connect with a WebSocket client to open a session. Native-protocol\n" + << "bridging to a backend server is not implemented yet (step 3); the\n" + << "current build echoes WebSocket messages back.\n"; +} + +void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResponse & response) +{ + LoggerPtr log = getLogger("WsProxyHandler"); + + if (request.getMethod() != HTTPRequest::HTTP_GET) + { + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_METHOD_NOT_ALLOWED); + *response.send() << "WebSocket upgrade requires GET method.\n"; + return; + } + + String ws_key = request.get("Sec-WebSocket-Key", ""); + if (!isValidWebSocketKey(ws_key)) + { + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_BAD_REQUEST); + *response.send() << "Invalid or missing Sec-WebSocket-Key.\n"; + return; + } + + /// RFC 6455 fixed the protocol at version 13; earlier drafts are obsolete. + if (request.get("Sec-WebSocket-Version", "") != "13") + { + response.set("Sec-WebSocket-Version", "13"); + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_BAD_REQUEST); + *response.send() << "Unsupported WebSocket version.\n"; + return; + } + + /// Complete the handshake by writing 101 directly to the socket; from here + /// on the stream is in WebSocket framing mode and we own the socket. + /// + /// Note: unlike the server's web terminal, no `Origin` check is enforced + /// here. The proxy's clients are applications, not browsers, so browser + /// cross-site protections do not apply; authentication will be handled in + /// step 3 as part of opening the backend connection. + Poco::Net::StreamSocket & socket = response.getSocket(); + const String handshake + = "HTTP/1.1 101 Switching Protocols\r\n" + "Upgrade: websocket\r\n" + "Connection: Upgrade\r\n" + "Sec-WebSocket-Accept: " + + computeWebSocketAccept(ws_key) + "\r\n\r\n"; + + /// `sendWebSocketFrame` and friends throw on write failure; reuse the frame + /// writer's guarantee by sending the raw handshake through the socket. + size_t sent_total = 0; + while (sent_total < handshake.size()) + { + int sent = socket.sendBytes(handshake.data() + sent_total, static_cast(handshake.size() - sent_total)); + if (sent <= 0) + { + LOG_DEBUG(log, "Failed to write WebSocket handshake"); + return; + } + sent_total += static_cast(sent); + } + + SCOPE_EXIT({ + try + { + socket.shutdown(); + } + catch (...) + { + tryLogCurrentException(log, "Failed to shut down WebSocket socket"); + } + }); + + LOG_DEBUG(log, "WebSocket session established (echo mode)"); + + /// Bound each blocking read so a stalled client cannot pin the handler thread. + socket.setReceiveTimeout(Poco::Timespan(30, 0)); + + bool running = true; + bool close_sent = false; + auto send_close_once = [&](uint16_t code, const String & reason) + { + if (close_sent) + return; + close_sent = true; + sendWebSocketClose(socket, code, reason); + }; + + /// Fragment reassembly state (RFC 6455 section 5.4). + String message_buffer; + uint8_t message_opcode = 0; + bool in_fragmented_message = false; + static constexpr size_t MAX_MESSAGE_SIZE = 16 * 1024 * 1024; + + while (running) + { + WebSocketFrame frame; + try + { + frame = readWebSocketFrame(socket); + } + catch (const Poco::TimeoutException &) + { + LOG_DEBUG(log, "WebSocket read timed out, closing"); + break; + } + catch (...) + { + LOG_DEBUG(log, "WebSocket read error: {}", getCurrentExceptionMessage(false)); + break; + } + + if (frame.message_too_big) + { + send_close_once(1009, "Message too big"); + break; + } + if (frame.protocol_error) + { + send_close_once(1002, "Protocol error"); + break; + } + if (!frame.valid) + break; + + /// Control frames may interleave with fragmented data frames. + if (frame.opcode >= 0x08) + { + switch (frame.opcode) + { + case Opcode::Close: + send_close_once(1000, "Bye"); + running = false; + break; + case Opcode::Ping: + sendWebSocketFrame(socket, Opcode::Pong, frame.payload.data(), frame.payload.size()); + break; + default: + break; + } + continue; + } + + /// Data frame: accumulate fragments. + if (frame.opcode != Opcode::Continuation) + { + if (in_fragmented_message) + { + send_close_once(1002, "New message during fragmentation"); + break; + } + message_opcode = frame.opcode; + message_buffer = std::move(frame.payload); + in_fragmented_message = !frame.fin; + } + else + { + if (!in_fragmented_message) + { + send_close_once(1002, "Unexpected continuation frame"); + break; + } + if (message_buffer.size() + frame.payload.size() > MAX_MESSAGE_SIZE) + { + send_close_once(1009, "Message too big"); + break; + } + message_buffer.append(frame.payload); + if (frame.fin) + in_fragmented_message = false; + } + + if (!frame.fin) + continue; /// More fragments to come. + + /// Complete message assembled: echo it back with the same opcode. + sendWebSocketFrame(socket, message_opcode, message_buffer.data(), message_buffer.size()); + message_buffer.clear(); + } + + LOG_DEBUG(log, "WebSocket session closed"); +} + +void WsProxyHandler::handleRequest(HTTPServerRequest & request, HTTPServerResponse & response, const ProfileEvents::Event &) +{ + try + { + String upgrade = request.get("Upgrade", ""); + std::transform(upgrade.begin(), upgrade.end(), upgrade.begin(), ::tolower); + + if (upgrade == "websocket" && hasUpgradeToken(request.get("Connection", ""))) + handleWebSocket(request, response); + else + serveInfo(request, response); + } + catch (...) + { + tryLogCurrentException("WsProxyHandler"); + } +} + +} diff --git a/programs/wsproxy/WsProxyHandler.h b/programs/wsproxy/WsProxyHandler.h new file mode 100644 index 000000000000..f9e17576ee38 --- /dev/null +++ b/programs/wsproxy/WsProxyHandler.h @@ -0,0 +1,25 @@ +#pragma once + +#include + +namespace DB +{ + +/// HTTP entry point for the proxy. +/// +/// Step 2 scope: a plain HTTP request serves a short info page; a WebSocket +/// upgrade request completes the RFC 6455 handshake and then runs an echo loop +/// (control frames handled, data frames reflected back). The echo loop is the +/// placeholder for step 3, where each session will instead open a native-protocol +/// `Connection` to a backend server and bridge packets <-> frames. +class WsProxyHandler : public HTTPRequestHandler +{ +public: + void handleRequest(HTTPServerRequest & request, HTTPServerResponse & response, const ProfileEvents::Event & write_event) override; + +private: + void handleWebSocket(HTTPServerRequest & request, HTTPServerResponse & response); + void serveInfo(HTTPServerRequest & request, HTTPServerResponse & response); +}; + +} From 167c251556f16c3881098beefc4ca021d60f8ebe Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 20:16:35 +0200 Subject: [PATCH 04/46] Ignore the repo-root tmp/ scratch directory Local scratch space (build/test logs, throwaway scripts) lives in `tmp/` per the project conventions; keep it out of version control. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 6f33e7e83db8..1204fd3b20d4 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,7 @@ ci/local.env .env /tests/integration/.env /ci/tmp +/tmp /tests/venv /obj-x86_64-linux-gnu/ From 0721f3ac11e81a258f73370434fd5d6f5f8aff82 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 22:06:34 +0200 Subject: [PATCH 05/46] Bridge WebSocket sessions to a native-protocol Connection Replace the echo loop with `ProxySession`, which opens one `Connection` per WebSocket session (1:1) to a backend server (from `WSPROXY_BACKEND_*` env vars, default `localhost:9000`). A query arrives as a text frame; the result streams back as binary frames encoded with the output format from the URL `?format=` parameter (default `JSONEachRow`), one flush per block via `FormatFactory::getOutputFormat` and `IOutputFormat::write`/`flush`. Completion, backend exceptions, and cancellation are reported as JSON control frames (`end` / `error` / `cancelled`). A Close frame observed in the interleaved `Connection::poll` / socket poll loop triggers `Connection::sendCancel` and drains the backend to `EndOfStream`. `WriteBufferToWebSocket` latches a `broken` flag instead of throwing from `nextImpl` when the client is gone, so the buffer is always finalized (clean result) or canceled (error/aborted) with no teardown warnings. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/CMakeLists.txt | 1 + programs/wsproxy/ProxySession.cpp | 372 ++++++++++++++++++++++++++++ programs/wsproxy/ProxySession.h | 58 +++++ programs/wsproxy/WsProxy.cpp | 39 ++- programs/wsproxy/WsProxyHandler.cpp | 154 ++++-------- programs/wsproxy/WsProxyHandler.h | 18 +- 6 files changed, 522 insertions(+), 120 deletions(-) create mode 100644 programs/wsproxy/ProxySession.cpp create mode 100644 programs/wsproxy/ProxySession.h diff --git a/programs/wsproxy/CMakeLists.txt b/programs/wsproxy/CMakeLists.txt index e9d91ad4daa1..a84bb0e8390b 100644 --- a/programs/wsproxy/CMakeLists.txt +++ b/programs/wsproxy/CMakeLists.txt @@ -2,6 +2,7 @@ set (CLICKHOUSE_WSPROXY_SOURCES WsProxy.cpp WsProxyHandler.cpp WebSocketFrames.cpp + ProxySession.cpp ) set (CLICKHOUSE_WSPROXY_LINK diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp new file mode 100644 index 000000000000..eeb369b93634 --- /dev/null +++ b/programs/wsproxy/ProxySession.cpp @@ -0,0 +1,372 @@ +#include +#include + +#include + +#include +#include +#include +#include + +#include +#include + +#include +#include + +#include +#include +#include + +#include +#include +#include + + +namespace DB +{ + +using namespace DB::WsProxy; + +namespace +{ + +/// A `WriteBuffer` whose flushes are emitted as WebSocket binary frames. The +/// output format writes into it directly, so calling `flush` after each result +/// block streams that block to the client as its own frame (mid-query push). +/// +/// Send failures are swallowed and latched into `broken` rather than thrown: +/// once the client is gone there is nothing to do but stop, and a throwing +/// `nextImpl` would otherwise leave the buffer unfinalized (and could throw +/// from `finalize`). Callers check `isBroken` to notice the client left. +class WriteBufferToWebSocket : public BufferWithOwnMemory +{ +public: + explicit WriteBufferToWebSocket(Poco::Net::StreamSocket & socket_, size_t size = DBMS_DEFAULT_BUFFER_SIZE) + : BufferWithOwnMemory(size), socket(socket_) + { + } + + ~WriteBufferToWebSocket() override = default; + + bool isBroken() const { return broken; } + +private: + void nextImpl() override + { + if (broken || offset() == 0) + return; + try + { + sendWebSocketBinary(socket, working_buffer.begin(), offset()); + } + catch (...) + { + broken = true; + } + } + + Poco::Net::StreamSocket & socket; + bool broken = false; +}; + +/// Minimal JSON string escaping for the small control-frame payloads. +String escapeJSON(const String & s) +{ + String out; + out.reserve(s.size() + 2); + for (char c : s) + { + switch (c) + { + case '"': out += R"(\")"; break; + case '\\': out += R"(\\)"; break; + case '\n': out += R"(\n)"; break; + case '\r': out += R"(\r)"; break; + case '\t': out += R"(\t)"; break; + default: + if (static_cast(c) < 0x20) + { + static const char * hex = "0123456789abcdef"; + out += "\\u00"; + out += hex[(c >> 4) & 0xF]; + out += hex[c & 0xF]; + } + else + out += c; + } + } + return out; +} + +} + +ProxySession::ProxySession(Poco::Net::StreamSocket & socket_, ContextPtr context_, BackendParams backend_, String format_) + : socket(socket_), context(std::move(context_)), backend(std::move(backend_)), format(std::move(format_)) +{ +} + +void ProxySession::sendControlEvent(const String & event, const String & message) +{ + String json = "{\"event\":\"" + event + "\""; + if (!message.empty()) + json += ",\"message\":\"" + escapeJSON(message) + "\""; + json += "}"; + sendWebSocketText(socket, json); +} + +std::optional ProxySession::readClientMessage() +{ + String buffer; + bool in_fragmented_message = false; + + while (true) + { + WebSocketFrame frame; + try + { + frame = readWebSocketFrame(socket); + } + catch (...) + { + return std::nullopt; + } + + if (!frame.valid || frame.protocol_error || frame.message_too_big) + return std::nullopt; + + /// Control frames may interleave with data frames. + if (frame.opcode >= 0x08) + { + if (frame.opcode == Opcode::Close) + return std::nullopt; + if (frame.opcode == Opcode::Ping) + sendWebSocketFrame(socket, Opcode::Pong, frame.payload.data(), frame.payload.size()); + continue; + } + + if (frame.opcode != Opcode::Continuation) + { + if (in_fragmented_message) + return std::nullopt; /// New data frame during fragmentation. + buffer = std::move(frame.payload); + in_fragmented_message = !frame.fin; + } + else + { + if (!in_fragmented_message) + return std::nullopt; /// Continuation without a start. + buffer.append(frame.payload); + if (frame.fin) + in_fragmented_message = false; + } + + if (frame.fin) + return buffer; + } +} + +bool ProxySession::executeQuery(Connection & connection, const String & query) +{ + LoggerPtr log = getLogger("ProxySession"); + + const auto & settings = context->getSettingsRef(); + auto timeouts = ConnectionTimeouts::getTCPTimeoutsWithoutFailover(settings); + + ClientInfo client_info; + client_info.query_kind = ClientInfo::QueryKind::INITIAL_QUERY; + + connection.sendQuery( + timeouts, + query, + /* query_parameters */ {}, + /* query_id */ "", + QueryProcessingStage::Complete, + &settings, + &client_info, + /* with_pending_data */ false, + /* external_roles */ {}, + /* process_progress_callback */ {}); + + WriteBufferToWebSocket out_buf(socket); + OutputFormatPtr output; + bool cancelled = false; + bool client_gone = false; + + /// Send a JSON control frame, tolerating a client that has already left. + auto try_control = [&](const String & event, const String & message) + { + if (client_gone) + return; + try + { + sendControlEvent(event, message); + } + catch (...) + { + client_gone = true; + } + }; + + auto note_client_gone = [&] + { + client_gone = true; + if (!cancelled) + { + connection.sendCancel(); + cancelled = true; + } + }; + + while (true) + { + /// While no server data is pending, watch the client socket so a Close + /// frame mid-query cancels the running query promptly. + while (!connection.poll(50'000 /* microseconds */)) + { + if (socket.poll(Poco::Timespan(0), Poco::Net::Socket::SELECT_READ)) + { + WebSocketFrame frame; + try + { + frame = readWebSocketFrame(socket); + } + catch (...) + { + frame.valid = false; + } + + /// A Close frame (or broken read) means the client is gone: + /// cancel the backend query and drain until EndOfStream. Other + /// in-query frames are ignored for now. + if (!frame.valid || frame.opcode == Opcode::Close) + note_client_gone(); + } + } + + Packet packet = connection.receivePacket(); + switch (packet.type) + { + case Protocol::Server::Data: + { + if (client_gone) + break; /// Draining after cancel; discard. + if (!output) + output = FormatFactory::instance().getOutputFormat(format, out_buf, packet.block.cloneEmpty(), context); + if (packet.block.rows() > 0) + { + output->write(packet.block); + output->flush(); /// Stream this block as its own frame(s). + if (out_buf.isBroken()) + note_client_gone(); /// Client left mid-stream. + } + break; + } + case Protocol::Server::Exception: + { + /// The result is incomplete, so discard buffered bytes rather than + /// finalizing the format (which would append a valid suffix). + output.reset(); + out_buf.cancel(); + const String message = packet.exception ? packet.exception->displayText() : "Unknown error from server"; + LOG_DEBUG(log, "Backend query exception: {}", message); + try_control("error", message); + return !client_gone; + } + case Protocol::Server::EndOfStream: + { + if (!client_gone && output) + { + output->finalize(); + out_buf.finalize(); + } + else + { + output.reset(); + out_buf.cancel(); + } + try_control(cancelled ? "cancelled" : "end", ""); + return !client_gone; + } + /// Not surfaced to the client yet (progress/log/profile push is a + /// later refinement); drain and continue. + case Protocol::Server::Progress: + case Protocol::Server::ProfileInfo: + case Protocol::Server::ProfileEvents: + case Protocol::Server::Totals: + case Protocol::Server::Extremes: + case Protocol::Server::Log: + case Protocol::Server::TableColumns: + case Protocol::Server::PartUUIDs: + case Protocol::Server::TimezoneUpdate: + break; + default: + LOG_WARNING(log, "Unexpected packet {} from backend, ending session", packet.type); + output.reset(); + out_buf.cancel(); + try_control("error", "Unexpected packet from backend"); + return false; + } + } +} + +void ProxySession::run() +{ + LoggerPtr log = getLogger("ProxySession"); + + Connection connection( + backend.host, + backend.port, + backend.database, + backend.user, + backend.password, + /* proto_send_chunked_ */ "chunked_optional", + /* proto_recv_chunked_ */ "chunked_optional", + /* ssh_private_key_ */ SSHKey{}, + /* jwt_ */ "", + /* quota_key_ */ "", + /* cluster_ */ "", + /* cluster_secret_ */ "", + /* client_name_ */ "clickhouse-wsproxy", + Protocol::Compression::Enable, + Protocol::Secure::Disable, + /* tls_sni_override_ */ "", + /* bind_host_ */ ""); + + LOG_DEBUG(log, "Proxy session started; backend {}:{}", backend.host, backend.port); + + while (true) + { + auto query = readClientMessage(); + if (!query) + break; + + if (query->empty()) + continue; + + try + { + if (!executeQuery(connection, *query)) + break; + } + catch (...) + { + const String message = getCurrentExceptionMessage(false); + LOG_DEBUG(log, "Session error: {}", message); + try + { + sendControlEvent("error", message); + } + catch (...) + { + /// The socket may already be gone; nothing more we can do. + LOG_DEBUG(log, "Failed to deliver error event to client"); + } + break; + } + } + + LOG_DEBUG(log, "Proxy session ended"); +} + +} diff --git a/programs/wsproxy/ProxySession.h b/programs/wsproxy/ProxySession.h new file mode 100644 index 000000000000..03a8080d9497 --- /dev/null +++ b/programs/wsproxy/ProxySession.h @@ -0,0 +1,58 @@ +#pragma once + +#include + +#include +#include + +#include + +namespace DB +{ + +class Connection; + +/// Where the proxy forwards native-protocol queries. +struct BackendParams +{ + String host = "localhost"; + UInt16 port = 9000; + String user = "default"; + String password; + String database; +}; + +/// Drives one WebSocket session against a backend ClickHouse server over the +/// native protocol. +/// +/// Owns a `Connection` for the lifetime of the session. For each query the +/// client sends as a WebSocket text frame, it runs `sendQuery` / `receivePacket` +/// and streams the result blocks back as binary frames (encoded with the chosen +/// output format), followed by a JSON control frame (`end` / `error` / +/// `cancelled`). A Close frame arriving mid-query triggers `Connection::sendCancel`. +class ProxySession +{ +public: + ProxySession(Poco::Net::StreamSocket & socket_, ContextPtr context_, BackendParams backend_, String format_); + + void run(); + +private: + /// Read one complete client message (reassembling fragments, answering pings). + /// Returns the message payload, or nullopt when the session should end + /// (Close frame, read error, or protocol violation). + std::optional readClientMessage(); + + /// Execute one query and stream its result. Returns false if the session + /// should end afterwards (client closed mid-query). + bool executeQuery(Connection & connection, const String & query); + + void sendControlEvent(const String & event, const String & message); + + Poco::Net::StreamSocket & socket; + ContextPtr context; + BackendParams backend; + String format; +}; + +} diff --git a/programs/wsproxy/WsProxy.cpp b/programs/wsproxy/WsProxy.cpp index ba852b4957e1..61cb1c37d304 100644 --- a/programs/wsproxy/WsProxy.cpp +++ b/programs/wsproxy/WsProxy.cpp @@ -20,9 +20,13 @@ #include #include +#include #include +#include +#include + /// Standalone WebSocket proxy. /// @@ -51,12 +55,39 @@ namespace class WsProxyHandlerFactory : public HTTPRequestHandlerFactory { public: + WsProxyHandlerFactory(ContextPtr context_, BackendParams backend_) + : context(std::move(context_)), backend(std::move(backend_)) + { + } + std::unique_ptr createRequestHandler(const HTTPServerRequest &) override { - return std::make_unique(); + return std::make_unique(context, backend); } + +private: + ContextPtr context; + BackendParams backend; }; +/// Read a string environment variable, falling back to `def` when unset/empty. +std::string envOr(const char * name, const std::string & def) +{ + const char * value = std::getenv(name); + return (value && *value) ? std::string(value) : def; +} + +BackendParams backendParamsFromEnv() +{ + BackendParams params; + params.host = envOr("WSPROXY_BACKEND_HOST", params.host); + params.port = static_cast(std::stoul(envOr("WSPROXY_BACKEND_PORT", "9000"))); + params.user = envOr("WSPROXY_BACKEND_USER", params.user); + params.password = envOr("WSPROXY_BACKEND_PASSWORD", params.password); + params.database = envOr("WSPROXY_BACKEND_DATABASE", params.database); + return params; +} + /// Conservative HTTP limits/timeouts for the skeleton; a later revision should /// source these from configuration. class WsProxyHTTPContext : public IHTTPContext @@ -99,6 +130,8 @@ class WsProxyServer : public Poco::Util::ServerApplication registerFormats(); + const BackendParams backend = backendParamsFromEnv(); + static constexpr UInt16 port = 9010; Poco::Net::ServerSocket socket(port); Poco::ThreadPool server_pool(/* minCapacity= */ 1, /* maxCapacity= */ 16); @@ -106,13 +139,13 @@ class WsProxyServer : public Poco::Util::ServerApplication HTTPServer server( std::make_shared(), - std::make_shared(), + std::make_shared(global_context, backend), server_pool, socket, params); server.start(); - LOG_INFO(log, "clickhouse-wsproxy listening on port {}", port); + LOG_INFO(log, "clickhouse-wsproxy listening on port {}; backend {}:{}", port, backend.host, backend.port); waitForTerminationRequest(); diff --git a/programs/wsproxy/WsProxyHandler.cpp b/programs/wsproxy/WsProxyHandler.cpp index d770be5c69b6..2510e18bbd64 100644 --- a/programs/wsproxy/WsProxyHandler.cpp +++ b/programs/wsproxy/WsProxyHandler.cpp @@ -1,5 +1,6 @@ #include #include +#include #include #include @@ -12,10 +13,12 @@ #include #include +#include #include #include +#include namespace DB @@ -53,6 +56,31 @@ bool hasUpgradeToken(const String & connection_header) return false; } +/// Output format from the `format` query parameter of the WebSocket URL. +String outputFormatFromURI(const String & uri_string, LoggerPtr log) +{ + String out_format = "JSONEachRow"; + try + { + Poco::URI uri(uri_string); + for (const auto & param : uri.getQueryParameters()) + { + if (param.first == "format" && !param.second.empty()) + out_format = param.second; + } + } + catch (...) + { + LOG_DEBUG(log, "Could not parse request URI for the format parameter; using default"); + } + return out_format; +} + +} + +WsProxyHandler::WsProxyHandler(ContextPtr context_, BackendParams backend_) + : context(std::move(context_)), backend(std::move(backend_)) +{ } void WsProxyHandler::serveInfo(HTTPServerRequest & request, HTTPServerResponse & response) @@ -62,9 +90,10 @@ void WsProxyHandler::serveInfo(HTTPServerRequest & request, HTTPServerResponse & *response.send() << "clickhouse-wsproxy\n" << "Requested: " << request.getURI() << "\n" - << "Connect with a WebSocket client to open a session. Native-protocol\n" - << "bridging to a backend server is not implemented yet (step 3); the\n" - << "current build echoes WebSocket messages back.\n"; + << "Open a WebSocket connection to run queries. Send a query as a text\n" + << "frame; results stream back as binary frames in the output format\n" + << "(set via ?format=..., default JSONEachRow), followed by a JSON\n" + << "control frame ({\"event\":\"end\"} / \"error\" / \"cancelled\").\n"; } void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResponse & response) @@ -100,8 +129,7 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp /// /// Note: unlike the server's web terminal, no `Origin` check is enforced /// here. The proxy's clients are applications, not browsers, so browser - /// cross-site protections do not apply; authentication will be handled in - /// step 3 as part of opening the backend connection. + /// cross-site protections do not apply; authentication is a later concern. Poco::Net::StreamSocket & socket = response.getSocket(); const String handshake = "HTTP/1.1 101 Switching Protocols\r\n" @@ -110,8 +138,6 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp "Sec-WebSocket-Accept: " + computeWebSocketAccept(ws_key) + "\r\n\r\n"; - /// `sendWebSocketFrame` and friends throw on write failure; reuse the frame - /// writer's guarantee by sending the raw handshake through the socket. size_t sent_total = 0; while (sent_total < handshake.size()) { @@ -131,116 +157,20 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp } catch (...) { - tryLogCurrentException(log, "Failed to shut down WebSocket socket"); + /// The client may have already closed the socket; expected, not worth a stack trace. + LOG_DEBUG(log, "WebSocket socket shutdown: {}", getCurrentExceptionMessage(false)); } }); - LOG_DEBUG(log, "WebSocket session established (echo mode)"); - - /// Bound each blocking read so a stalled client cannot pin the handler thread. - socket.setReceiveTimeout(Poco::Timespan(30, 0)); - - bool running = true; - bool close_sent = false; - auto send_close_once = [&](uint16_t code, const String & reason) - { - if (close_sent) - return; - close_sent = true; - sendWebSocketClose(socket, code, reason); - }; + const String out_format = outputFormatFromURI(request.getURI(), log); + LOG_DEBUG(log, "WebSocket session established; output format {}", out_format); - /// Fragment reassembly state (RFC 6455 section 5.4). - String message_buffer; - uint8_t message_opcode = 0; - bool in_fragmented_message = false; - static constexpr size_t MAX_MESSAGE_SIZE = 16 * 1024 * 1024; + /// Bound each blocking WebSocket read so a stalled client cannot pin the + /// handler thread indefinitely between queries. + socket.setReceiveTimeout(Poco::Timespan(300, 0)); - while (running) - { - WebSocketFrame frame; - try - { - frame = readWebSocketFrame(socket); - } - catch (const Poco::TimeoutException &) - { - LOG_DEBUG(log, "WebSocket read timed out, closing"); - break; - } - catch (...) - { - LOG_DEBUG(log, "WebSocket read error: {}", getCurrentExceptionMessage(false)); - break; - } - - if (frame.message_too_big) - { - send_close_once(1009, "Message too big"); - break; - } - if (frame.protocol_error) - { - send_close_once(1002, "Protocol error"); - break; - } - if (!frame.valid) - break; - - /// Control frames may interleave with fragmented data frames. - if (frame.opcode >= 0x08) - { - switch (frame.opcode) - { - case Opcode::Close: - send_close_once(1000, "Bye"); - running = false; - break; - case Opcode::Ping: - sendWebSocketFrame(socket, Opcode::Pong, frame.payload.data(), frame.payload.size()); - break; - default: - break; - } - continue; - } - - /// Data frame: accumulate fragments. - if (frame.opcode != Opcode::Continuation) - { - if (in_fragmented_message) - { - send_close_once(1002, "New message during fragmentation"); - break; - } - message_opcode = frame.opcode; - message_buffer = std::move(frame.payload); - in_fragmented_message = !frame.fin; - } - else - { - if (!in_fragmented_message) - { - send_close_once(1002, "Unexpected continuation frame"); - break; - } - if (message_buffer.size() + frame.payload.size() > MAX_MESSAGE_SIZE) - { - send_close_once(1009, "Message too big"); - break; - } - message_buffer.append(frame.payload); - if (frame.fin) - in_fragmented_message = false; - } - - if (!frame.fin) - continue; /// More fragments to come. - - /// Complete message assembled: echo it back with the same opcode. - sendWebSocketFrame(socket, message_opcode, message_buffer.data(), message_buffer.size()); - message_buffer.clear(); - } + ProxySession session(socket, context, backend, out_format); + session.run(); LOG_DEBUG(log, "WebSocket session closed"); } diff --git a/programs/wsproxy/WsProxyHandler.h b/programs/wsproxy/WsProxyHandler.h index f9e17576ee38..8d9ee303c047 100644 --- a/programs/wsproxy/WsProxyHandler.h +++ b/programs/wsproxy/WsProxyHandler.h @@ -1,5 +1,8 @@ #pragma once +#include + +#include #include namespace DB @@ -7,19 +10,24 @@ namespace DB /// HTTP entry point for the proxy. /// -/// Step 2 scope: a plain HTTP request serves a short info page; a WebSocket -/// upgrade request completes the RFC 6455 handshake and then runs an echo loop -/// (control frames handled, data frames reflected back). The echo loop is the -/// placeholder for step 3, where each session will instead open a native-protocol -/// `Connection` to a backend server and bridge packets <-> frames. +/// A plain HTTP request serves a short info page. A WebSocket upgrade completes +/// the RFC 6455 handshake and then hands the socket to a `ProxySession`, which +/// bridges the WebSocket to a native-protocol `Connection` against the backend. +/// The desired output format is taken from the `format` query parameter of the +/// WebSocket URL (default `JSONEachRow`). class WsProxyHandler : public HTTPRequestHandler { public: + WsProxyHandler(ContextPtr context_, BackendParams backend_); + void handleRequest(HTTPServerRequest & request, HTTPServerResponse & response, const ProfileEvents::Event & write_event) override; private: void handleWebSocket(HTTPServerRequest & request, HTTPServerResponse & response); void serveInfo(HTTPServerRequest & request, HTTPServerResponse & response); + + ContextPtr context; + BackendParams backend; }; } From ee4ce5aa0520475e8e017cc43bada5123d1056f5 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 22:06:35 +0200 Subject: [PATCH 06/46] Record step 3 (native-protocol bridging) in the design notes Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/TODO.md b/TODO.md index e70842bfbcfb..be2b8a4a673b 100644 --- a/TODO.md +++ b/TODO.md @@ -155,6 +155,36 @@ Test-hygiene note: background launches detach and orphan (PPID 1) if not killed binary PID; `pgrep -f` also matches the launch shell. Always test one instance, select the PID whose command starts with `./build`, and confirm zero processes between runs. +## Step 3 status — DONE (branch `wsproxy-skeleton`) + +The echo loop is replaced by a real native-protocol bridge (`ProxySession.{h,cpp}`): + +- Opens one `Connection` per WebSocket session (1:1) to a backend from env + (`WSPROXY_BACKEND_HOST`/`PORT`/`USER`/`PASSWORD`/`DATABASE`; defaults `localhost:9000`). +- Client sends a query as a text frame; output format from the WS URL `?format=` + (default `JSONEachRow`). Results stream back as **binary frames**, one flush per block, + via `FormatFactory::getOutputFormat` → `output->write` → `output->flush`. +- Control frames (JSON text): `{"event":"end"}`, `{"event":"error","message":...}`, + `{"event":"cancelled"}`. +- Mid-query cancel: a Close frame detected in the interleaved `connection.poll` / + `socket.poll` loop triggers `Connection::sendCancel`, then drains to `EndOfStream`. + +Two bugs found by running end-to-end and fixed: +1. **Wasn't actually streaming** — `WriteBufferToWebSocket` only flushed on a full 1 MB + buffer, so nothing went out until `finalize`. Fix: `output->flush()` per block. +2. **`WriteBuffer` not finalized on all paths / `nextImpl` threw on a gone client** — gave + a "neither finalized nor canceled" warning and teardown exceptions. Fix: `nextImpl` + latches a `broken` flag instead of throwing; every exit path finalizes (clean) or + cancels (error/aborted). + +Validated against a real ClickHouse server (installed 25.8.1) with `tmp/ws_test3.py` and +`tmp/ws_cancel_test.py`: scalar/`numbers(5)` selects, error propagation, `?format=TSV`, +multi-query sessions, per-block streaming, and mid-query cancel confirmed via +`system.processes` (query stopped server-side). Clean SIGTERM shutdown still holds. + +Not yet done (next): INSERT path (plan step 6), progress/log/profile push as control +frames, config-file/auth, `BaseDaemon` hardening. + ## Plan 1. **Skeleton.** Standalone `programs/wsproxy/` binary. **DONE — builds, links, runs-to-listen.** From 47cd9115a6ef75c291c5eac7af82f27d5e1ecbda Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 22:56:03 +0200 Subject: [PATCH 07/46] Add INSERT support: stream client data to the backend via an input format `ProxySession::executeQuery` now best-effort parses the query with `ParserQuery` and dispatches: an `ASTInsertQuery` goes to `executeInsert`, everything else to the existing SELECT path. For an INSERT the app sends `INSERT INTO t [FORMAT X]` as a text frame and then streams data as binary frames terminated by a zero-length binary frame; the proxy parses it with the input format (`FORMAT` clause, else `Values` for inline data, else the URL `?format=`) over a `ReadBufferFromWebSocket`, pulls blocks with a `PullingPipelineExecutor`, and `sendData`s them, then sends an empty block to finish. Inline `INSERT ... VALUES (...)` data is handled from the query text. The native INSERT handshake requires `with_pending_data=true` on `sendQuery` and an (empty) `sendExternalTablesData` before the server returns the sample block; omitting either deadlocks. A Close frame mid-stream aborts via `sendCancel` instead of committing partial data. `ReadBufferFromWebSocket` latches a `finished` flag because `ReadBuffer::next` may call `nextImpl` again after EOF, which would otherwise block on a frame that never arrives. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/ProxySession.cpp | 291 ++++++++++++++++++++++++++++-- programs/wsproxy/ProxySession.h | 8 +- 2 files changed, 279 insertions(+), 20 deletions(-) diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp index eeb369b93634..3a3edefb3c91 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/programs/wsproxy/ProxySession.cpp @@ -13,9 +13,18 @@ #include #include +#include +#include +#include + +#include +#include +#include #include #include +#include +#include #include #include @@ -31,6 +40,11 @@ using namespace DB::WsProxy; namespace { +/// Generous parser limits for the best-effort INSERT-detection parse. +constexpr size_t MAX_QUERY_SIZE = 16 << 20; +constexpr size_t MAX_PARSER_DEPTH = 1000; +constexpr size_t MAX_PARSER_BACKTRACKS = 1'000'000; + /// A `WriteBuffer` whose flushes are emitted as WebSocket binary frames. The /// output format writes into it directly, so calling `flush` after each result /// block streams that block to the client as its own frame (mid-query push). @@ -70,6 +84,80 @@ class WriteBufferToWebSocket : public BufferWithOwnMemory bool broken = false; }; +/// A `ReadBuffer` that yields the payloads of incoming WebSocket binary frames, +/// used to feed the input format when parsing client-supplied INSERT data. +/// +/// End of data is a zero-length binary frame or a text frame (clean). A Close +/// frame or read error is an abort (client disconnected mid-insert): `wasAborted` +/// lets the caller cancel the backend query instead of committing partial data. +class ReadBufferFromWebSocket : public ReadBuffer +{ +public: + explicit ReadBufferFromWebSocket(Poco::Net::StreamSocket & socket_) + : ReadBuffer(nullptr, 0), socket(socket_) + { + } + + bool wasAborted() const { return aborted; } + +private: + bool nextImpl() override + { + /// `ReadBuffer::next()` may call `nextImpl` again after a false return + /// (there is no permanent EOF latch in the base class), and some input + /// formats do a trailing read. Latch the end so we never block on a + /// frame that will not arrive. + if (finished || aborted) + return false; + + while (true) + { + WebSocketFrame frame; + try + { + frame = readWebSocketFrame(socket); + } + catch (...) + { + aborted = true; + return false; + } + + if (!frame.valid || frame.opcode == Opcode::Close) + { + aborted = true; + return false; + } + if (frame.opcode == Opcode::Ping) + { + sendWebSocketFrame(socket, Opcode::Pong, frame.payload.data(), frame.payload.size()); + continue; + } + if (frame.opcode == Opcode::Text) + { + finished = true; /// Clean end-of-data control frame. + return false; + } + + /// Binary (or continuation) frame: an empty one is the clean end marker. + if (frame.payload.empty()) + { + finished = true; + return false; + } + + current_frame = std::move(frame.payload); + BufferBase::set(current_frame.data(), current_frame.size(), 0); + return true; + } + } + + Poco::Net::StreamSocket & socket; + String current_frame; + bool aborted = false; + bool finished = false; +}; + /// Minimal JSON string escaping for the small control-frame payloads. String escapeJSON(const String & s) { @@ -88,7 +176,7 @@ String escapeJSON(const String & s) if (static_cast(c) < 0x20) { static const char * hex = "0123456789abcdef"; - out += "\\u00"; + out += R"(\u00)"; out += hex[(c >> 4) & 0xF]; out += hex[c & 0xF]; } @@ -108,13 +196,54 @@ ProxySession::ProxySession(Poco::Net::StreamSocket & socket_, ContextPtr context void ProxySession::sendControlEvent(const String & event, const String & message) { - String json = "{\"event\":\"" + event + "\""; + String json = R"({"event":")" + event + "\""; if (!message.empty()) - json += ",\"message\":\"" + escapeJSON(message) + "\""; + json += R"(,"message":")" + escapeJSON(message) + "\""; json += "}"; sendWebSocketText(socket, json); } +void ProxySession::sendBackendQuery(Connection & connection, const String & query, bool with_pending_data) +{ + const auto & settings = context->getSettingsRef(); + auto timeouts = ConnectionTimeouts::getTCPTimeoutsWithoutFailover(settings); + + ClientInfo client_info; + client_info.query_kind = ClientInfo::QueryKind::INITIAL_QUERY; + + /// `with_pending_data` must be true for INSERTs so the server enters the + /// send-data handshake and replies with the sample/header block. + connection.sendQuery( + timeouts, + query, + /* query_parameters */ {}, + /* query_id */ "", + QueryProcessingStage::Complete, + &settings, + &client_info, + with_pending_data, + /* external_roles */ {}, + /* process_progress_callback */ {}); +} + +void ProxySession::drainUntilEndOfStream(Connection & connection) +{ + try + { + while (true) + { + Packet packet = connection.receivePacket(); + if (packet.type == Protocol::Server::EndOfStream || packet.type == Protocol::Server::Exception) + return; + } + } + catch (...) + { + /// The connection may be broken after a cancel; nothing more to drain. + LOG_DEBUG(getLogger("ProxySession"), "drain after cancel: {}", getCurrentExceptionMessage(false)); + } +} + std::optional ProxySession::readClientMessage() { String buffer; @@ -168,25 +297,34 @@ std::optional ProxySession::readClientMessage() bool ProxySession::executeQuery(Connection & connection, const String & query) { - LoggerPtr log = getLogger("ProxySession"); + /// Best-effort parse to detect an INSERT (with or without inline data). + /// Anything that does not parse as an INSERT takes the generic path, where + /// the backend reports real syntax/semantic errors. + ASTPtr ast; + try + { + ParserQuery parser(query.data() + query.size()); + ast = parseQuery(parser, query, MAX_QUERY_SIZE, MAX_PARSER_DEPTH, MAX_PARSER_BACKTRACKS); + } + catch (...) + { + ast = nullptr; + } - const auto & settings = context->getSettingsRef(); - auto timeouts = ConnectionTimeouts::getTCPTimeoutsWithoutFailover(settings); + if (ast) + { + if (const auto * insert = ast->as()) + return executeInsert(connection, query, *insert); + } - ClientInfo client_info; - client_info.query_kind = ClientInfo::QueryKind::INITIAL_QUERY; + return executeSelect(connection, query); +} - connection.sendQuery( - timeouts, - query, - /* query_parameters */ {}, - /* query_id */ "", - QueryProcessingStage::Complete, - &settings, - &client_info, - /* with_pending_data */ false, - /* external_roles */ {}, - /* process_progress_callback */ {}); +bool ProxySession::executeSelect(Connection & connection, const String & query) +{ + LoggerPtr log = getLogger("ProxySession"); + + sendBackendQuery(connection, query); WriteBufferToWebSocket out_buf(socket); OutputFormatPtr output; @@ -310,6 +448,121 @@ bool ProxySession::executeQuery(Connection & connection, const String & query) } } +bool ProxySession::executeInsert(Connection & connection, const String & query, const ASTInsertQuery & insert) +{ + LoggerPtr log = getLogger("ProxySession"); + + const char * query_end = query.data() + query.size(); + const bool has_inline_data = insert.data && insert.data < query_end; + /// Format precedence: an explicit `FORMAT` clause; else `Values` for inline + /// `INSERT ... VALUES (...)` data; else the session's `?format=` for streamed data. + const String insert_format = !insert.format.empty() ? insert.format : (has_inline_data ? "Values" : format); + /// Send the query without the inline data section, keeping the FORMAT clause. + const String query_to_send = insert.data ? String(query.data(), insert.data) : query; + + sendBackendQuery(connection, query_to_send, /* with_pending_data */ true); + + /// The server waits for external-tables data before replying with the sample + /// block; we have none, so send an empty set to unblock the handshake. + ExternalTablesData external_tables_data; + connection.sendExternalTablesData(external_tables_data); + + /// The server replies with the sample block describing the target structure. + Block sample; + while (true) + { + Packet packet = connection.receivePacket(); + if (packet.type == Protocol::Server::Data) + { + sample = packet.block; + break; + } + if (packet.type == Protocol::Server::Exception) + { + const String message = packet.exception ? packet.exception->displayText() : "Unknown error from server"; + sendControlEvent("error", message); + return true; + } + if (packet.type == Protocol::Server::EndOfStream) + { + /// No structure to send data with (e.g. nothing was expected); we are done. + sendControlEvent("end", ""); + return true; + } + /// Ignore Log / Progress / TableColumns / TimezoneUpdate while waiting. + } + + /// Data source: inline data from the query if present, else streamed from the client. + std::unique_ptr data_in; + ReadBufferFromWebSocket * ws_in = nullptr; + if (has_inline_data) + { + data_in = std::make_unique(insert.data, query_end - insert.data); + } + else + { + auto ws = std::make_unique(socket); + ws_in = ws.get(); + data_in = std::move(ws); + } + + bool aborted = false; + try + { + auto source = context->getInputFormat(insert_format, *data_in, sample, DEFAULT_BLOCK_SIZE); + Pipe pipe(source); + QueryPipeline pipeline(std::move(pipe)); + PullingPipelineExecutor executor(pipeline); + + Block block; + while (executor.pull(block)) + { + if (!block.empty()) + connection.sendData(block, /* name */ "", /* scalar */ false); + } + aborted = ws_in && ws_in->wasAborted(); + } + catch (...) + { + /// Error parsing the client-supplied data: abort the insert. + const String message = getCurrentExceptionMessage(false); + LOG_DEBUG(log, "INSERT data error: {}", message); + connection.sendCancel(); + drainUntilEndOfStream(connection); + sendControlEvent("error", message); + return true; + } + + if (aborted) + { + /// Client disconnected mid-insert: cancel rather than commit partial data. + connection.sendCancel(); + drainUntilEndOfStream(connection); + return false; + } + + connection.sendData({}, "", false); /// Empty block signals end of data. + + while (true) + { + Packet packet = connection.receivePacket(); + switch (packet.type) + { + case Protocol::Server::EndOfStream: + sendControlEvent("end", ""); + return true; + case Protocol::Server::Exception: + { + const String message = packet.exception ? packet.exception->displayText() : "Unknown error from server"; + sendControlEvent("error", message); + return true; + } + default: + break; /// Ignore progress / log / profile events. + } + } +} + void ProxySession::run() { LoggerPtr log = getLogger("ProxySession"); diff --git a/programs/wsproxy/ProxySession.h b/programs/wsproxy/ProxySession.h index 03a8080d9497..64589c189617 100644 --- a/programs/wsproxy/ProxySession.h +++ b/programs/wsproxy/ProxySession.h @@ -11,6 +11,7 @@ namespace DB { class Connection; +class ASTInsertQuery; /// Where the proxy forwards native-protocol queries. struct BackendParams @@ -44,9 +45,14 @@ class ProxySession std::optional readClientMessage(); /// Execute one query and stream its result. Returns false if the session - /// should end afterwards (client closed mid-query). + /// should end afterwards (client closed mid-query). Dispatches to the + /// INSERT or SELECT path depending on the parsed query kind. bool executeQuery(Connection & connection, const String & query); + bool executeSelect(Connection & connection, const String & query); + bool executeInsert(Connection & connection, const String & query, const ASTInsertQuery & insert); + void sendBackendQuery(Connection & connection, const String & query, bool with_pending_data = false); + void drainUntilEndOfStream(Connection & connection); void sendControlEvent(const String & event, const String & message); Poco::Net::StreamSocket & socket; From 8593910ffed7b8b24879a056cec8df0445cc4d0c Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 22:56:03 +0200 Subject: [PATCH 08/46] Record the INSERT path in the design notes Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/TODO.md b/TODO.md index be2b8a4a673b..d660ef09a508 100644 --- a/TODO.md +++ b/TODO.md @@ -185,6 +185,37 @@ multi-query sessions, per-block streaming, and mid-query cancel confirmed via Not yet done (next): INSERT path (plan step 6), progress/log/profile push as control frames, config-file/auth, `BaseDaemon` hardening. +## Step 4 status (plan step 6) — INSERT path DONE (branch `wsproxy-skeleton`) + +Reverse-direction bridging in `ProxySession::executeInsert`: the app sends `INSERT INTO t +[FORMAT X]` as a text frame, then streams data as **binary frames** ending with a +zero-length binary frame (clean) — a Close mid-stream aborts (→ `sendCancel`, no partial +commit). Inline `INSERT ... VALUES (...)` is handled too (data taken from the query, format +inferred as `Values`). `executeQuery` best-effort parses with `ParserQuery` and dispatches +INSERT vs SELECT. + +Flow: `sendQuery(with_pending_data=true)` → `sendExternalTablesData({})` → receive sample +block → `getInputFormat` over a `ReadBufferFromWebSocket` → `PullingPipelineExecutor` pulls +blocks → `Connection::sendData` each → `sendData({})` to finish → await `EndOfStream`. + +Four bugs found by running end-to-end (all fixed): +1. **`with_pending_data` must be true** for INSERT or the server never enters the send-data + handshake. +2. **Must send empty external-tables data** (`sendExternalTablesData({})`) after the query, + else server waits for external tables while the proxy waits for the sample → deadlock. +3. **`ReadBuffer` has no permanent EOF latch** — `next()` re-invokes `nextImpl` after a false + return, and the input format does a trailing read; without a `finished` latch the WS read + buffer blocked on a frame that never arrives (this was the actual hang). Fixed with a latch. +4. Used the **synchronous `PullingPipelineExecutor`** (not the async one) — runs in the + calling thread, no query-context/thread-group requirement for a bare global `Context`. + +Validated against installed ClickHouse 25.8.1 with `tmp/ws_insert_test.py`: streamed +JSONEachRow insert, inline VALUES insert, format inference, server-side row/sum verification, +INSERT-into-missing-table error. No regressions in SELECT (`tmp/ws_test3.py`) or cancel +(`tmp/ws_cancel_test.py`); clean SIGTERM shutdown holds. + +Still next: progress/log/profile push as control frames, config-file/auth, `BaseDaemon`. + ## Plan 1. **Skeleton.** Standalone `programs/wsproxy/` binary. **DONE — builds, links, runs-to-listen.** From a6c6e43932026a2c6e0fddf24e985db554372392 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 23:01:41 +0200 Subject: [PATCH 09/46] Push mid-query progress to the client as control frames Each server `Progress` packet in the SELECT path is forwarded as a JSON text frame (`{"event":"progress","read_rows":...,"read_bytes":...,"total_rows_to_read":...}`). Server progress is incremental, so reads are accumulated and the latest total estimate is tracked, and running totals are pushed. Text frames do not disturb the binary result stream, so clients skip `progress` events until the terminal `end`/`error`/`cancelled`. This delivers the mid-query push that motivates the proxy over the plain HTTP interface. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/ProxySession.cpp | 35 +++++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp index 3a3edefb3c91..659868c6c04c 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/programs/wsproxy/ProxySession.cpp @@ -356,6 +356,12 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) } }; + /// Server progress packets are incremental; accumulate the reads and track + /// the latest total estimate so each pushed event carries running totals. + UInt64 total_read_rows = 0; + UInt64 total_read_bytes = 0; + UInt64 total_rows_to_read = 0; + while (true) { /// While no server data is pending, watch the client socket so a Close @@ -426,9 +432,34 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) try_control(cancelled ? "cancelled" : "end", ""); return !client_gone; } - /// Not surfaced to the client yet (progress/log/profile push is a - /// later refinement); drain and continue. case Protocol::Server::Progress: + { + /// Mid-query push: forward running progress as a JSON text frame. + /// Text frames do not disturb the binary result stream. + if (!client_gone) + { + const auto values = packet.progress.getValues(); + total_read_rows += values.read_rows; + total_read_bytes += values.read_bytes; + if (values.total_rows_to_read) + total_rows_to_read = values.total_rows_to_read; + + const String json = R"({"event":"progress","read_rows":)" + std::to_string(total_read_rows) + + R"(,"read_bytes":)" + std::to_string(total_read_bytes) + + R"(,"total_rows_to_read":)" + std::to_string(total_rows_to_read) + "}"; + try + { + sendWebSocketText(socket, json); + } + catch (...) + { + client_gone = true; + } + } + break; + } + /// Not surfaced to the client yet (profile events / logs / totals push + /// is a later refinement); drain and continue. case Protocol::Server::ProfileInfo: case Protocol::Server::ProfileEvents: case Protocol::Server::Totals: From 29fa745fe9c031f1101d9da1d3166c978cc0227c Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 23:01:41 +0200 Subject: [PATCH 10/46] Record progress push in the design notes Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/TODO.md b/TODO.md index d660ef09a508..0eb5b75d77bb 100644 --- a/TODO.md +++ b/TODO.md @@ -216,6 +216,20 @@ INSERT-into-missing-table error. No regressions in SELECT (`tmp/ws_test3.py`) or Still next: progress/log/profile push as control frames, config-file/auth, `BaseDaemon`. +## Step 5 status — progress push DONE (branch `wsproxy-skeleton`) + +Mid-query push implemented in `executeSelect`: each server `Progress` packet is forwarded as +a `{"event":"progress","read_rows":N,"read_bytes":M,"total_rows_to_read":T}` text frame. +Server progress is incremental, so reads are accumulated and the latest total estimate is +tracked; running totals are pushed. Text frames don't disturb the binary result stream, so +clients just skip `progress` events when reading to the terminal `end`/`error`/`cancelled`. + +Validated with `tmp/ws_progress_test.py` (a ~3s `sleepEachRow` query): 32 progress events, +monotonic `read_rows` 1→30, `total_rows_to_read=30`. No regressions (SELECT/INSERT/cancel +suites updated to skip progress frames and still green). + +Still next: profile-events / log push, config-file/auth, `BaseDaemon` hardening. + ## Plan 1. **Skeleton.** Standalone `programs/wsproxy/` binary. **DONE — builds, links, runs-to-listen.** From 2e7522149b0d8dec9020c1fa5108099720b67b27 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 8 Jul 2026 23:10:49 +0200 Subject: [PATCH 11/46] Add Node.js/Vitest integration tests for the WebSocket proxy A self-contained JavaScript test suite (target audience: JS engineers consuming the WebSocket API) under `programs/wsproxy/tests`, using Node's built-in global `WebSocket` and Vitest. `test/globalSetup.mjs` launches the backend and the proxy if not already running, creates the shared `default.wsp_test` table, and tears down what it started; `test/helpers.mjs` wraps the WS protocol as a small `Session` client. Covers SELECT (formats, errors, large streamed results, multi-query sessions), INSERT (streamed JSONEachRow/TSV, inline VALUES, format inference, errors, verified against server-side counts), mid-query progress push, and cancellation (closing the socket stops the query server-side). 15 tests across 4 files. `node_modules` is git-ignored. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/tests/.gitignore | 1 + programs/wsproxy/tests/README.md | 43 + programs/wsproxy/tests/package-lock.json | 1460 +++++++++++++++++ programs/wsproxy/tests/package.json | 14 + programs/wsproxy/tests/test/cancel.test.mjs | 68 + programs/wsproxy/tests/test/globalSetup.mjs | 79 + programs/wsproxy/tests/test/helpers.mjs | 168 ++ programs/wsproxy/tests/test/insert.test.mjs | 88 + programs/wsproxy/tests/test/progress.test.mjs | 46 + programs/wsproxy/tests/test/select.test.mjs | 64 + programs/wsproxy/tests/vitest.config.mjs | 13 + 11 files changed, 2044 insertions(+) create mode 100644 programs/wsproxy/tests/.gitignore create mode 100644 programs/wsproxy/tests/README.md create mode 100644 programs/wsproxy/tests/package-lock.json create mode 100644 programs/wsproxy/tests/package.json create mode 100644 programs/wsproxy/tests/test/cancel.test.mjs create mode 100644 programs/wsproxy/tests/test/globalSetup.mjs create mode 100644 programs/wsproxy/tests/test/helpers.mjs create mode 100644 programs/wsproxy/tests/test/insert.test.mjs create mode 100644 programs/wsproxy/tests/test/progress.test.mjs create mode 100644 programs/wsproxy/tests/test/select.test.mjs create mode 100644 programs/wsproxy/tests/vitest.config.mjs diff --git a/programs/wsproxy/tests/.gitignore b/programs/wsproxy/tests/.gitignore new file mode 100644 index 000000000000..c2658d7d1b31 --- /dev/null +++ b/programs/wsproxy/tests/.gitignore @@ -0,0 +1 @@ +node_modules/ diff --git a/programs/wsproxy/tests/README.md b/programs/wsproxy/tests/README.md new file mode 100644 index 000000000000..486189940edc --- /dev/null +++ b/programs/wsproxy/tests/README.md @@ -0,0 +1,43 @@ +# clickhouse-wsproxy integration tests + +Node.js + [Vitest](https://vitest.dev/) integration tests for `clickhouse-wsproxy`, +written from the perspective of a JavaScript client using the WebSocket API. + +They exercise the proxy end-to-end against a real ClickHouse backend: + +- `select.test.mjs` — SELECT queries, output-format switching (`JSONEachRow` / `TSV` / + `CSV`), error propagation, large streamed results, multi-query sessions. +- `insert.test.mjs` — streamed INSERT (`JSONEachRow` / `TSV`), inline `VALUES`, format + inference, and error handling, verified against server-side row counts. +- `progress.test.mjs` — mid-query progress push (`{"event":"progress",...}` frames). +- `cancel.test.mjs` — closing the socket mid-query cancels the running query server-side. + +## Requirements + +- Node.js >= 22 (uses the built-in global `WebSocket`; no `ws` dependency). +- A built proxy binary at `build/programs/wsproxy/clickhouse-wsproxy`. +- A `clickhouse-server` binary (default `/usr/local/bin/clickhouse-server`) and the + throwaway config at `tmp/ch/config.xml` (see the project setup). + +## Running + +```bash +cd programs/wsproxy/tests +npm install +npm test +``` + +`test/globalSetup.mjs` launches the backend (`:9000`) and the proxy (`:9010`) if they are +not already listening, creates the shared `default.wsp_test` table, and tears down whatever +it started. Override binary/config locations with the `CLICKHOUSE_SERVER`, `CH_CONFIG`, +`WSPROXY_BIN`, and `WSPROXY_URL` environment variables. + +## Protocol under test + +The client sends a SQL query as a **text** frame. SELECT results stream back as **binary** +frames encoded in the output format chosen via the WS URL `?format=` parameter. Mid-query +progress and the terminal outcome (`end` / `error` / `cancelled`) arrive as JSON **text** +frames. For INSERT, the client sends `INSERT INTO t [FORMAT X]` then streams data as binary +frames ending with a zero-length binary frame; closing the socket mid-query cancels it. + +See `test/helpers.mjs` for the small client wrapper (`Session`, `runQuery`, `backendScalar`). diff --git a/programs/wsproxy/tests/package-lock.json b/programs/wsproxy/tests/package-lock.json new file mode 100644 index 000000000000..7fdfcf87f116 --- /dev/null +++ b/programs/wsproxy/tests/package-lock.json @@ -0,0 +1,1460 @@ +{ + "name": "wsproxy-tests", + "version": "0.1.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "wsproxy-tests", + "version": "0.1.0", + "devDependencies": { + "vitest": "^2.1.9" + } + }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.21.5.tgz", + "integrity": "sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.21.5.tgz", + "integrity": "sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.21.5.tgz", + "integrity": "sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.21.5.tgz", + "integrity": "sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.21.5.tgz", + "integrity": "sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.21.5.tgz", + "integrity": "sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.21.5.tgz", + "integrity": "sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.21.5.tgz", + "integrity": "sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.21.5.tgz", + "integrity": "sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.21.5.tgz", + "integrity": "sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.21.5.tgz", + "integrity": "sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.21.5.tgz", + "integrity": "sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.21.5.tgz", + "integrity": "sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.21.5.tgz", + "integrity": "sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.21.5.tgz", + "integrity": "sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.21.5.tgz", + "integrity": "sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.21.5.tgz", + "integrity": "sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.21.5.tgz", + "integrity": "sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.21.5.tgz", + "integrity": "sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.21.5.tgz", + "integrity": "sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.21.5.tgz", + "integrity": "sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.21.5.tgz", + "integrity": "sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.21.5.tgz", + "integrity": "sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@jridgewell/sourcemap-codec": { + "version": "1.5.5", + "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz", + "integrity": "sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==", + "dev": true, + "license": "MIT" + }, + "node_modules/@rollup/rollup-android-arm-eabi": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.62.2.tgz", + "integrity": "sha512-6o7ZLZK+BeenkZCFNDXqpbjw9bD6nuWonvS/lwQJp7NoVVxm6p3qE7qQ5jGuBjiFsgvqjD8mZAU5oWxTmbOeOg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-android-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.62.2.tgz", + "integrity": "sha512-BaH7BllCACHoH1LguOU56UItGfUWjujlO65kS9LAodViaN4bwIKd7oeW/ZHJ/4ljr/7MIiENnNy3HJ0zXv8Zkw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-darwin-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.62.2.tgz", + "integrity": "sha512-v39RCCvj4He82I9sFmk+M1VZ0PLM9sfsLVikjfx2hYBNALhrrOR2D3JjQA6AhlaSOgcR+RzrKY7e1+bT6SUO/A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-darwin-x64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.62.2.tgz", + "integrity": "sha512-yl0y2vq3S3lHeuXhEdss6TWfKW8vkujImO12tn4ZkG/4oghr09LvdYm2RElVjokTQiUvDUGXLGsYeLqUMCKpGA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-freebsd-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.62.2.tgz", + "integrity": "sha512-tT4pvt4qXD+vEoezupCWi+a1F0vvDiksiHc+PxRlYTOH1I6/X4id9jPxTP+Fg+545euaFT1jJVs4CEdHZAU1vw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-freebsd-x64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.62.2.tgz", + "integrity": "sha512-6nU5F2wCW+qvCBhTn1pdIU3bzsIoF7EUwsCDRxilWGprQR6yd508YnH9+OKFCwpfS8pjZqDUmnCAr7exax0XCg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-linux-arm-gnueabihf": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.62.2.tgz", + "integrity": "sha512-n1GJHPOvpIfhi3TmrCeh6S6URt9BFCt0KQE3qvexyGCTAKpR4Lg+eWvNZEqu7epxwus/8ElT3hacYEucm49SZg==", + "cpu": [ + "arm" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm-musleabihf": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.62.2.tgz", + "integrity": "sha512-JqgflS8wEB+UXV/vS1RpRbifGBeN4D5lz8D8oOFbFZw4vedvdOgCFAjfBmIMdW3yL10XpQQ0Ambepw6MXrhOnA==", + "cpu": [ + "arm" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.62.2.tgz", + "integrity": "sha512-wnFJkogWvN4jm/hQRF2UBaeUmk20j5+DmHvoyWii2b8HJDyvz1MF2OU/6ynXt2KR63rbZLWkFpoytpdc/yBuSA==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.62.2.tgz", + "integrity": "sha512-HVu2bp0zhvJ8xHEV9+UUs7S90VadmBSY3LcIMvozbPo4AuMGDWlz3ymHLHZPX4hR67TKTt8Qp5PJ5RBg/i+RMQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-gnu/-/rollup-linux-loong64-gnu-4.62.2.tgz", + "integrity": "sha512-mQqqAV8QaoSgr9I2fKDLY2BAVvmKjWoGiu/cSYQonsLvtqwEn1E4QYfnCOcp5zoEqNhsDYin1s6jx/VJmrxlZg==", + "cpu": [ + "loong64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-musl/-/rollup-linux-loong64-musl-4.62.2.tgz", + "integrity": "sha512-IxKLoxCQ2IWi6bT2akyDUBGsOImDKB+sPp4EsTmwFQ/fMwpCKm8uLSSgP/Kx/QYUgKis6SEZ5/Nlhup0DIA0PQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-gnu/-/rollup-linux-ppc64-gnu-4.62.2.tgz", + "integrity": "sha512-Mk5ha2RQSgyFfmYYLkBpPnUk8D8FriBxesO1u9O75X0mHgXL1UQcH5Itl2lurWL2tj0RxV9b9tJgipac0hRY9A==", + "cpu": [ + "ppc64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-musl/-/rollup-linux-ppc64-musl-4.62.2.tgz", + "integrity": "sha512-CjvEnqJL/0/TQ3TXX3OPIJ/kmBellrWd4heXUmHeJlTnmwjKpSJzoehLaL6Xk0ZnMHBu9dZuFADNOrtjF4v+2w==", + "cpu": [ + "ppc64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.62.2.tgz", + "integrity": "sha512-1SiZbzwdkaDURsew/tSOrooKiYy7EQGT6m8ufavAi9NEyQb/6VuIxFXAL1fqa4iZe3g4NbNk4P7J32z2tw5Mgg==", + "cpu": [ + "riscv64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.62.2.tgz", + "integrity": "sha512-nQts12zJ3NQRoE6uYljOH89v7szzLDvG2JD/vsX+vGXU8w/At1GowTZ5/7qeFQ8m7L55rpR8Okugnuo5bgjy2Q==", + "cpu": [ + "riscv64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-s390x-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.62.2.tgz", + "integrity": "sha512-E9/ll019jhPIJgpzfZoIkBGhcz+kKNgVWYRY0zr9srBdPPFVpvOKW8VaJKUbeK+eZXyQF9ltME+Kk6affeaPgg==", + "cpu": [ + "s390x" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.62.2.tgz", + "integrity": "sha512-5BqxR/pshjey51iliyzTD5Xi3EN0aLmQ2lZ3lvefVV9c82BvrLo2/6OT55iifpWBufs6kdwWbuOKS841DrmK9A==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.62.2.tgz", + "integrity": "sha512-uNN83XxQrRAh/w0/pmAfibcwyb6YWt4gP+dpnQKPVJshAloQ785ii8CT8ZCIxkGg9opVsvAlGhFitSm6D1Jjpg==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-openbsd-x64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openbsd-x64/-/rollup-openbsd-x64-4.62.2.tgz", + "integrity": "sha512-srjEIxSH3LRnJN6THczDHWQplqEMFiAJrTab0msUryh9kwNpkICf3Ea6q6MN/2cZwRFUNx5w+h6Hpi4QuHS6Zg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ] + }, + "node_modules/@rollup/rollup-openharmony-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openharmony-arm64/-/rollup-openharmony-arm64-4.62.2.tgz", + "integrity": "sha512-8hOJnxgbyObnCm5AlRA3A931xX19xq80RjVTKgJOvEKWqJruP/Uf12IbAOaDjjEXYRewwHLfmF0YRIdK3OwKWA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ] + }, + "node_modules/@rollup/rollup-win32-arm64-msvc": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.62.2.tgz", + "integrity": "sha512-mmF4AY1i0hG/bLWUctUq59gtmgaSIRa3cu/A3JFRp/sCNEme2bgDEiDS22P9FbnJB8NJNF4jPJiSP5RHQpUTDg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-ia32-msvc": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.62.2.tgz", + "integrity": "sha512-DZgkknc6jhHrk46V25vbAM0zZkyP0nSDkJB8/dRkLTxv470dOmWDqGoEJl/9A0dFfS7yE3REOwNDxpHwSLSt0Q==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-gnu/-/rollup-win32-x64-gnu-4.62.2.tgz", + "integrity": "sha512-T6xr6ucWSFto+VGajA8YH26LdpHRuP4YLHEKAtCWvJDOlnmWcDZVCI2Jmjr+IFHDlt2zRaTAKE4tfjTaWLgJBg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-msvc": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.62.2.tgz", + "integrity": "sha512-BfzEnDJOt9T8M989/lA37EcJgat01wLRnoi5dQf3QzOH7jzpqTAzdDbVfRljVr5r+jzKqpbHeyOfAaXxAd0PAA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@types/estree": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz", + "integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/@vitest/expect": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-2.1.9.tgz", + "integrity": "sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/spy": "2.1.9", + "@vitest/utils": "2.1.9", + "chai": "^5.1.2", + "tinyrainbow": "^1.2.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/mocker": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-2.1.9.tgz", + "integrity": "sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/spy": "2.1.9", + "estree-walker": "^3.0.3", + "magic-string": "^0.30.12" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "msw": "^2.4.9", + "vite": "^5.0.0" + }, + "peerDependenciesMeta": { + "msw": { + "optional": true + }, + "vite": { + "optional": true + } + } + }, + "node_modules/@vitest/pretty-format": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-2.1.9.tgz", + "integrity": "sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyrainbow": "^1.2.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/runner": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-2.1.9.tgz", + "integrity": "sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/utils": "2.1.9", + "pathe": "^1.1.2" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/snapshot": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-2.1.9.tgz", + "integrity": "sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "2.1.9", + "magic-string": "^0.30.12", + "pathe": "^1.1.2" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/spy": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-2.1.9.tgz", + "integrity": "sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyspy": "^3.0.2" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/utils": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-2.1.9.tgz", + "integrity": "sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "2.1.9", + "loupe": "^3.1.2", + "tinyrainbow": "^1.2.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/assertion-error": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz", + "integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + } + }, + "node_modules/cac": { + "version": "6.7.14", + "resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz", + "integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/chai": { + "version": "5.3.3", + "resolved": "https://registry.npmjs.org/chai/-/chai-5.3.3.tgz", + "integrity": "sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==", + "dev": true, + "license": "MIT", + "dependencies": { + "assertion-error": "^2.0.1", + "check-error": "^2.1.1", + "deep-eql": "^5.0.1", + "loupe": "^3.1.0", + "pathval": "^2.0.0" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/check-error": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/check-error/-/check-error-2.1.3.tgz", + "integrity": "sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 16" + } + }, + "node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "dev": true, + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/deep-eql": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/deep-eql/-/deep-eql-5.0.2.tgz", + "integrity": "sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/es-module-lexer": { + "version": "1.7.0", + "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz", + "integrity": "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==", + "dev": true, + "license": "MIT" + }, + "node_modules/esbuild": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.21.5.tgz", + "integrity": "sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=12" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.21.5", + "@esbuild/android-arm": "0.21.5", + "@esbuild/android-arm64": "0.21.5", + "@esbuild/android-x64": "0.21.5", + "@esbuild/darwin-arm64": "0.21.5", + "@esbuild/darwin-x64": "0.21.5", + "@esbuild/freebsd-arm64": "0.21.5", + "@esbuild/freebsd-x64": "0.21.5", + "@esbuild/linux-arm": "0.21.5", + "@esbuild/linux-arm64": "0.21.5", + "@esbuild/linux-ia32": "0.21.5", + "@esbuild/linux-loong64": "0.21.5", + "@esbuild/linux-mips64el": "0.21.5", + "@esbuild/linux-ppc64": "0.21.5", + "@esbuild/linux-riscv64": "0.21.5", + "@esbuild/linux-s390x": "0.21.5", + "@esbuild/linux-x64": "0.21.5", + "@esbuild/netbsd-x64": "0.21.5", + "@esbuild/openbsd-x64": "0.21.5", + "@esbuild/sunos-x64": "0.21.5", + "@esbuild/win32-arm64": "0.21.5", + "@esbuild/win32-ia32": "0.21.5", + "@esbuild/win32-x64": "0.21.5" + } + }, + "node_modules/estree-walker": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz", + "integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0" + } + }, + "node_modules/expect-type": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz", + "integrity": "sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=12.0.0" + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/loupe": { + "version": "3.2.1", + "resolved": "https://registry.npmjs.org/loupe/-/loupe-3.2.1.tgz", + "integrity": "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/magic-string": { + "version": "0.30.21", + "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", + "integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.5" + } + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "dev": true, + "license": "MIT" + }, + "node_modules/nanoid": { + "version": "3.3.15", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.15.tgz", + "integrity": "sha512-y7Wygv/7mEOvxTuEQDB8StXdMRBWf1kR/tlhAzBRUFkB2jfcLOAxO/SHmOO2zgz1pVgK29/kyupn059/bCHdjA==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "bin": { + "nanoid": "bin/nanoid.cjs" + }, + "engines": { + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" + } + }, + "node_modules/pathe": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/pathe/-/pathe-1.1.2.tgz", + "integrity": "sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/pathval": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/pathval/-/pathval-2.0.1.tgz", + "integrity": "sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 14.16" + } + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/postcss": { + "version": "8.5.16", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.16.tgz", + "integrity": "sha512-vuwillviilfKZsg0VGj5R/YwwcHx4SLsIOI/7K6mQkWx+l5cUHTjj5g0AasTBcyXsbfTgrwsUNmVUb5xVwyPwg==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "nanoid": "^3.3.12", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/rollup": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.62.2.tgz", + "integrity": "sha512-RFnrW4lhXA3s3eqHDZvN654g8OTjzRfqpIRJYczCGB6HzphckVAi/Qh4tbPUbRuDi7s1Llv8g/NspLkttY3gTA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "1.0.9" + }, + "bin": { + "rollup": "dist/bin/rollup" + }, + "engines": { + "node": ">=18.0.0", + "npm": ">=8.0.0" + }, + "optionalDependencies": { + "@rollup/rollup-android-arm-eabi": "4.62.2", + "@rollup/rollup-android-arm64": "4.62.2", + "@rollup/rollup-darwin-arm64": "4.62.2", + "@rollup/rollup-darwin-x64": "4.62.2", + "@rollup/rollup-freebsd-arm64": "4.62.2", + "@rollup/rollup-freebsd-x64": "4.62.2", + "@rollup/rollup-linux-arm-gnueabihf": "4.62.2", + "@rollup/rollup-linux-arm-musleabihf": "4.62.2", + "@rollup/rollup-linux-arm64-gnu": "4.62.2", + "@rollup/rollup-linux-arm64-musl": "4.62.2", + "@rollup/rollup-linux-loong64-gnu": "4.62.2", + "@rollup/rollup-linux-loong64-musl": "4.62.2", + "@rollup/rollup-linux-ppc64-gnu": "4.62.2", + "@rollup/rollup-linux-ppc64-musl": "4.62.2", + "@rollup/rollup-linux-riscv64-gnu": "4.62.2", + "@rollup/rollup-linux-riscv64-musl": "4.62.2", + "@rollup/rollup-linux-s390x-gnu": "4.62.2", + "@rollup/rollup-linux-x64-gnu": "4.62.2", + "@rollup/rollup-linux-x64-musl": "4.62.2", + "@rollup/rollup-openbsd-x64": "4.62.2", + "@rollup/rollup-openharmony-arm64": "4.62.2", + "@rollup/rollup-win32-arm64-msvc": "4.62.2", + "@rollup/rollup-win32-ia32-msvc": "4.62.2", + "@rollup/rollup-win32-x64-gnu": "4.62.2", + "@rollup/rollup-win32-x64-msvc": "4.62.2", + "fsevents": "~2.3.2" + } + }, + "node_modules/siginfo": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz", + "integrity": "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==", + "dev": true, + "license": "ISC" + }, + "node_modules/source-map-js": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", + "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/stackback": { + "version": "0.0.2", + "resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz", + "integrity": "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==", + "dev": true, + "license": "MIT" + }, + "node_modules/std-env": { + "version": "3.10.0", + "resolved": "https://registry.npmjs.org/std-env/-/std-env-3.10.0.tgz", + "integrity": "sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinybench": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz", + "integrity": "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinyexec": { + "version": "0.3.2", + "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-0.3.2.tgz", + "integrity": "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinypool": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/tinypool/-/tinypool-1.1.1.tgz", + "integrity": "sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^18.0.0 || >=20.0.0" + } + }, + "node_modules/tinyrainbow": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-1.2.0.tgz", + "integrity": "sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/tinyspy": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/tinyspy/-/tinyspy-3.0.2.tgz", + "integrity": "sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/vite": { + "version": "5.4.21", + "resolved": "https://registry.npmjs.org/vite/-/vite-5.4.21.tgz", + "integrity": "sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==", + "dev": true, + "license": "MIT", + "dependencies": { + "esbuild": "^0.21.3", + "postcss": "^8.4.43", + "rollup": "^4.20.0" + }, + "bin": { + "vite": "bin/vite.js" + }, + "engines": { + "node": "^18.0.0 || >=20.0.0" + }, + "funding": { + "url": "https://github.com/vitejs/vite?sponsor=1" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + }, + "peerDependencies": { + "@types/node": "^18.0.0 || >=20.0.0", + "less": "*", + "lightningcss": "^1.21.0", + "sass": "*", + "sass-embedded": "*", + "stylus": "*", + "sugarss": "*", + "terser": "^5.4.0" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "less": { + "optional": true + }, + "lightningcss": { + "optional": true + }, + "sass": { + "optional": true + }, + "sass-embedded": { + "optional": true + }, + "stylus": { + "optional": true + }, + "sugarss": { + "optional": true + }, + "terser": { + "optional": true + } + } + }, + "node_modules/vite-node": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/vite-node/-/vite-node-2.1.9.tgz", + "integrity": "sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==", + "dev": true, + "license": "MIT", + "dependencies": { + "cac": "^6.7.14", + "debug": "^4.3.7", + "es-module-lexer": "^1.5.4", + "pathe": "^1.1.2", + "vite": "^5.0.0" + }, + "bin": { + "vite-node": "vite-node.mjs" + }, + "engines": { + "node": "^18.0.0 || >=20.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/vitest": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/vitest/-/vitest-2.1.9.tgz", + "integrity": "sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/expect": "2.1.9", + "@vitest/mocker": "2.1.9", + "@vitest/pretty-format": "^2.1.9", + "@vitest/runner": "2.1.9", + "@vitest/snapshot": "2.1.9", + "@vitest/spy": "2.1.9", + "@vitest/utils": "2.1.9", + "chai": "^5.1.2", + "debug": "^4.3.7", + "expect-type": "^1.1.0", + "magic-string": "^0.30.12", + "pathe": "^1.1.2", + "std-env": "^3.8.0", + "tinybench": "^2.9.0", + "tinyexec": "^0.3.1", + "tinypool": "^1.0.1", + "tinyrainbow": "^1.2.0", + "vite": "^5.0.0", + "vite-node": "2.1.9", + "why-is-node-running": "^2.3.0" + }, + "bin": { + "vitest": "vitest.mjs" + }, + "engines": { + "node": "^18.0.0 || >=20.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "@edge-runtime/vm": "*", + "@types/node": "^18.0.0 || >=20.0.0", + "@vitest/browser": "2.1.9", + "@vitest/ui": "2.1.9", + "happy-dom": "*", + "jsdom": "*" + }, + "peerDependenciesMeta": { + "@edge-runtime/vm": { + "optional": true + }, + "@types/node": { + "optional": true + }, + "@vitest/browser": { + "optional": true + }, + "@vitest/ui": { + "optional": true + }, + "happy-dom": { + "optional": true + }, + "jsdom": { + "optional": true + } + } + }, + "node_modules/why-is-node-running": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz", + "integrity": "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==", + "dev": true, + "license": "MIT", + "dependencies": { + "siginfo": "^2.0.0", + "stackback": "0.0.2" + }, + "bin": { + "why-is-node-running": "cli.js" + }, + "engines": { + "node": ">=8" + } + } + } +} diff --git a/programs/wsproxy/tests/package.json b/programs/wsproxy/tests/package.json new file mode 100644 index 000000000000..de514a0e5b89 --- /dev/null +++ b/programs/wsproxy/tests/package.json @@ -0,0 +1,14 @@ +{ + "name": "wsproxy-tests", + "version": "0.1.0", + "private": true, + "type": "module", + "description": "Integration tests for clickhouse-wsproxy (WebSocket -> native protocol proxy)", + "scripts": { + "test": "vitest run", + "watch": "vitest" + }, + "devDependencies": { + "vitest": "^2.1.9" + } +} diff --git a/programs/wsproxy/tests/test/cancel.test.mjs b/programs/wsproxy/tests/test/cancel.test.mjs new file mode 100644 index 000000000000..0cee09e5ac9a --- /dev/null +++ b/programs/wsproxy/tests/test/cancel.test.mjs @@ -0,0 +1,68 @@ +import { describe, it, expect } from "vitest"; +import { Session, backendScalar, sleep } from "./helpers.mjs"; + +describe("cancel", () => { + it( + "cancels a running query server-side", + async () => { + // Unique marker so we can find exactly this query in system.processes. + const marker = "CANCEL_MARKER_" + Date.now(); + + // A long (~10s) query that streams slowly, one row per block, with the + // marker in a trailing comment so it shows up in system.processes. + const sql = `SELECT sleepEachRow(0.2), number FROM numbers(50) SETTINGS max_block_size = 1 -- ${marker}`; + + // Query that counts how many instances of our marker query are running, + // excluding the counting query itself. + const countSql = `SELECT count() FROM system.processes WHERE query LIKE '%${marker}%' AND query NOT LIKE '%system.processes%'`; + + const s = new Session("JSONEachRow"); + await s.ready(); + s.sendQuery(sql); + + // Wait until the query is actually streaming: the first BINARY frame + // proves a block arrived and the query is running on the backend. + let streaming = false; + for (let i = 0; i < 50 && !streaming; i++) { + const frame = await s.nextFrame(); + if (frame.type === "binary") streaming = true; + else if (frame.type === "close") break; // socket died unexpectedly + } + expect(streaming).toBe(true); + + // Sanity check: the query really is visible on the backend. + const running = await backendScalar(countSql); + expect(running).not.toBe("0"); + + // Closing the socket cancels the running query. + const startedAt = Date.now(); + s.close(); + + // Poll for up to ~5s until the marker query is gone from the backend. + let finalCount = running; + for (let i = 0; i < 20; i++) { + finalCount = await backendScalar(countSql); + if (finalCount === "0") break; + await sleep(250); + } + const elapsed = Date.now() - startedAt; + + // The query was cancelled promptly, not left to finish naturally. + expect(finalCount).toBe("0"); + // ~10s natural completion; cancellation must be much faster. + expect(elapsed).toBeLessThan(6000); + }, + 20000, + ); + + it("a normal query is unaffected", async () => { + // The proxy should be perfectly healthy after a cancel. + const s = new Session("JSONEachRow"); + try { + const { control } = await s.run("SELECT 1"); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + }); +}); diff --git a/programs/wsproxy/tests/test/globalSetup.mjs b/programs/wsproxy/tests/test/globalSetup.mjs new file mode 100644 index 000000000000..792d44f50f70 --- /dev/null +++ b/programs/wsproxy/tests/test/globalSetup.mjs @@ -0,0 +1,79 @@ +// Vitest global setup: ensure a ClickHouse backend (:9000) and the proxy (:9010) +// are running, create the shared test table, and tear down whatever we started. +// +// Reuses anything already listening (so you can run against a stack you started +// by hand); only processes we spawn here are killed on teardown. + +import { spawn } from "node:child_process"; +import net from "node:net"; +import { dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const testDir = dirname(fileURLToPath(import.meta.url)); +const repoRoot = resolve(testDir, "../../../.."); // programs/wsproxy/tests/test -> repo root + +const CLICKHOUSE_SERVER = process.env.CLICKHOUSE_SERVER ?? "/usr/local/bin/clickhouse-server"; +const CH_CONFIG = process.env.CH_CONFIG ?? resolve(repoRoot, "tmp/ch/config.xml"); +const WSPROXY_BIN = + process.env.WSPROXY_BIN ?? resolve(repoRoot, "build/programs/wsproxy/clickhouse-wsproxy"); + +const BACKEND_PORT = 9000; +const PROXY_PORT = 9010; + +function portOpen(port, host = "127.0.0.1") { + return new Promise((resolve) => { + const socket = net.connect({ port, host }, () => { + socket.destroy(); + resolve(true); + }); + socket.on("error", () => resolve(false)); + }); +} + +async function waitForPort(port, timeoutMs = 30_000) { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (await portOpen(port)) return; + await new Promise((r) => setTimeout(r, 200)); + } + throw new Error(`Timed out waiting for port ${port}`); +} + +export default async function setup() { + const spawned = []; + + if (!(await portOpen(BACKEND_PORT))) { + const proc = spawn(CLICKHOUSE_SERVER, ["--config-file", CH_CONFIG], { + stdio: "ignore", + detached: false, + }); + spawned.push(proc); + await waitForPort(BACKEND_PORT); + } + + if (!(await portOpen(PROXY_PORT))) { + const proc = spawn(WSPROXY_BIN, [], { stdio: "ignore", detached: false }); + spawned.push(proc); + await waitForPort(PROXY_PORT); + } + + // Create the shared test table via the proxy (DDL goes through the SELECT path). + const { runQuery } = await import("./helpers.mjs"); + const ddl = await runQuery( + "CREATE TABLE IF NOT EXISTS default.wsp_test (a UInt32, b String) ENGINE = Memory", + ); + if (ddl.control.event !== "end") { + throw new Error(`failed to create test table: ${JSON.stringify(ddl.control)}`); + } + + // Teardown: kill only what we started. + return () => { + for (const proc of spawned) { + try { + proc.kill("SIGTERM"); + } catch { + /* ignore */ + } + } + }; +} diff --git a/programs/wsproxy/tests/test/helpers.mjs b/programs/wsproxy/tests/test/helpers.mjs new file mode 100644 index 000000000000..eab9ada88dbf --- /dev/null +++ b/programs/wsproxy/tests/test/helpers.mjs @@ -0,0 +1,168 @@ +// WebSocket client helpers for the clickhouse-wsproxy integration tests. +// +// Protocol recap (proxy <-> client over one WebSocket): +// - The client sends a SQL query as a TEXT frame. +// - SELECT results stream back as BINARY frames (bytes of the chosen output +// format, set via the `?format=` query parameter of the WS URL). +// - Mid-query progress is pushed as TEXT frames: {"event":"progress",...}. +// - The query terminates with a TEXT control frame: +// {"event":"end"} | {"event":"error","message":...} | {"event":"cancelled"} +// - INSERT: send `INSERT INTO t [FORMAT X]` as a TEXT frame, then stream data +// as BINARY frames, then a zero-length BINARY frame to signal end of data. +// - Cancel a running query by CLOSING the socket. +// +// Uses Node's built-in global WebSocket (Node >= 22), so no dependencies. + +export const PROXY_URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; + +/** Build the WS URL for a given output format and optional path. */ +export function urlFor(format = "JSONEachRow", path = "/") { + const sep = path.includes("?") ? "&" : "?"; + return `${PROXY_URL}${path}${sep}format=${encodeURIComponent(format)}`; +} + +/** + * A single proxy session over one WebSocket. Supports multiple sequential + * queries. Incoming frames are buffered into an async queue so callers can + * `await` them regardless of arrival timing. + */ +export class Session { + constructor(format = "JSONEachRow", path = "/") { + this.ws = new WebSocket(urlFor(format, path)); + this.ws.binaryType = "arraybuffer"; + this._queue = []; + this._waiters = []; + this._closed = false; + + const push = (frame) => { + const waiter = this._waiters.shift(); + if (waiter) waiter(frame); + else this._queue.push(frame); + }; + + this.ws.addEventListener("message", (ev) => { + if (typeof ev.data === "string") push({ type: "text", data: ev.data }); + else push({ type: "binary", data: Buffer.from(ev.data) }); + }); + this.ws.addEventListener("close", () => { + this._closed = true; + push({ type: "close" }); + }); + + this.opened = new Promise((resolve, reject) => { + this.ws.addEventListener("open", () => resolve()); + this.ws.addEventListener("error", (ev) => + reject(new Error(`WebSocket error: ${ev?.message ?? ev}`)), + ); + }); + } + + /** Resolve once the socket is open. */ + ready() { + return this.opened; + } + + /** Next incoming frame: {type:'text',data:string} | {type:'binary',data:Buffer} | {type:'close'}. */ + nextFrame() { + if (this._queue.length) return Promise.resolve(this._queue.shift()); + return new Promise((resolve) => this._waiters.push(resolve)); + } + + /** Send a SQL query (TEXT frame). */ + sendQuery(sql) { + this.ws.send(sql); + } + + /** Send INSERT data (BINARY frame). Accepts a string or a Buffer/Uint8Array. */ + sendData(chunk) { + this.ws.send(typeof chunk === "string" ? Buffer.from(chunk) : chunk); + } + + /** Signal end of INSERT data (zero-length BINARY frame). */ + endData() { + this.ws.send(new Uint8Array(0)); + } + + /** + * Consume frames until the terminal control event. + * Returns { data: Buffer, text: string, progress: object[], control: object }. + * Binary frames are concatenated into `data`; progress events collected. + */ + async collect() { + const chunks = []; + const progress = []; + for (;;) { + const frame = await this.nextFrame(); + if (frame.type === "binary") { + chunks.push(frame.data); + } else if (frame.type === "text") { + const msg = JSON.parse(frame.data); + if (msg.event === "progress") progress.push(msg); + else { + const data = Buffer.concat(chunks); + return { data, text: data.toString("utf8"), progress, control: msg }; + } + } else { + // Socket closed without a terminal control frame. + const data = Buffer.concat(chunks); + return { data, text: data.toString("utf8"), progress, control: { event: "closed" } }; + } + } + } + + /** Send a query and collect its full result. */ + async run(sql) { + await this.ready(); + this.sendQuery(sql); + return this.collect(); + } + + /** + * Run an INSERT: send the query, stream the data chunks, end, and collect + * the terminal control event. `chunks` is an array of strings/Buffers. + */ + async insert(query, chunks = []) { + await this.ready(); + this.sendQuery(query); + for (const c of chunks) this.sendData(c); + this.endData(); + return this.collect(); + } + + /** Close the socket (also used to cancel a running query). */ + close() { + try { + this.ws.close(); + } catch { + /* already closing */ + } + } +} + +/** One-shot: open a session, run a query, close, return the result object. */ +export async function runQuery(sql, { format = "JSONEachRow" } = {}) { + const s = new Session(format); + try { + return await s.run(sql); + } finally { + s.close(); + } +} + +/** + * Query the backend through the proxy and return the single scalar/text result, + * trimmed. Handy for verifying server-side state (uses TSV so there is no JSON + * quoting to strip). + */ +export async function backendScalar(sql) { + const { text, control } = await runQuery(sql, { format: "TSV" }); + if (control.event !== "end") { + throw new Error(`backend query failed: ${control.event} ${control.message ?? ""}`); + } + return text.trim(); +} + +/** Sleep helper. */ +export function sleep(ms) { + return new Promise((r) => setTimeout(r, ms)); +} diff --git a/programs/wsproxy/tests/test/insert.test.mjs b/programs/wsproxy/tests/test/insert.test.mjs new file mode 100644 index 000000000000..7394a8911a2d --- /dev/null +++ b/programs/wsproxy/tests/test/insert.test.mjs @@ -0,0 +1,88 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { Session, runQuery, backendScalar } from "./helpers.mjs"; + +describe("INSERT", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it("streams a JSONEachRow insert across multiple chunks", async () => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + "INSERT INTO default.wsp_test FORMAT JSONEachRow", + ['{"a":1,"b":"x"}\n{"a":2,"b":"y"}\n', '{"a":3,"b":"z"}\n'], + ); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("3"); + expect(await backendScalar("SELECT sum(a) FROM default.wsp_test")).toBe("6"); + }); + + it("handles an inline VALUES insert with an inferred format", async () => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + "INSERT INTO default.wsp_test VALUES (10,'v'),(11,'w')", + [], + ); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + expect(await backendScalar("SELECT sum(a) FROM default.wsp_test")).toBe("21"); + }); + + it("streams an insert in a different input format (TSV)", async () => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert("INSERT INTO default.wsp_test FORMAT TSV", [ + "5\tp\n6\tq\n", + ]); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + }); + + it("reports an error when inserting into a missing table", async () => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + "INSERT INTO default.no_such_table_zz FORMAT JSONEachRow", + ['{"a":1,"b":"x"}\n'], + ); + expect(control.event).toBe("error"); + expect(control.message).toBeTruthy(); + expect(control.message).toMatch(/table/i); + } finally { + s.close(); + } + }); + + it("keeps the connection healthy for a SELECT after inserts", async () => { + const s = new Session("JSONEachRow"); + try { + const insertResult = await s.insert( + "INSERT INTO default.wsp_test FORMAT JSONEachRow", + ['{"a":1,"b":"x"}\n{"a":2,"b":"y"}\n'], + ); + expect(insertResult.control.event).toBe("end"); + } finally { + s.close(); + } + + const { text, control } = await runQuery( + "SELECT count() FROM default.wsp_test", + ); + expect(control.event).toBe("end"); + expect(text).toContain("2"); + }); +}); diff --git a/programs/wsproxy/tests/test/progress.test.mjs b/programs/wsproxy/tests/test/progress.test.mjs new file mode 100644 index 000000000000..f78e2848846f --- /dev/null +++ b/programs/wsproxy/tests/test/progress.test.mjs @@ -0,0 +1,46 @@ +import { describe, it, expect } from "vitest"; +import { runQuery } from "./helpers.mjs"; + +/** True if the numbers are in non-decreasing order (compare against a sorted copy). */ +function isMonotonicNonDecreasing(values) { + const sorted = [...values].sort((a, b) => a - b); + return values.every((v, i) => v === sorted[i]); +} + +describe("progress", () => { + it( + "pushes progress events during a slow query", + async () => { + // ~3s query producing many small blocks so the server emits several + // progress updates while it runs. + const sql = + "SELECT sleepEachRow(0.1), number FROM numbers(30) SETTINGS max_block_size = 1"; + const { progress, control } = await runQuery(sql); + + expect(control.event).toBe("end"); + + // Multiple mid-query pushes, not just a single terminal one. + expect(progress.length).toBeGreaterThanOrEqual(2); + + for (const ev of progress) { + expect(typeof ev.read_rows).toBe("number"); + expect(typeof ev.read_bytes).toBe("number"); + expect(typeof ev.total_rows_to_read).toBe("number"); + } + + // Running totals never go backwards. + const readRows = progress.map((ev) => ev.read_rows); + expect(isMonotonicNonDecreasing(readRows)).toBe(true); + + const last = progress[progress.length - 1]; + expect(last.read_rows).toBe(30); + expect(last.total_rows_to_read).toBe(30); + }, + 20000, + ); + + it("still ends cleanly for an instant query", async () => { + const { control } = await runQuery("SELECT 1"); + expect(control.event).toBe("end"); + }); +}); diff --git a/programs/wsproxy/tests/test/select.test.mjs b/programs/wsproxy/tests/test/select.test.mjs new file mode 100644 index 000000000000..1c64106d2ac6 --- /dev/null +++ b/programs/wsproxy/tests/test/select.test.mjs @@ -0,0 +1,64 @@ +import { describe, it, expect } from "vitest"; +import { Session, runQuery } from "./helpers.mjs"; + +describe("SELECT", () => { + it("returns a scalar SELECT as JSONEachRow", async () => { + const { text, control } = await runQuery("SELECT 1 AS a, 'hello' AS b"); + expect(control.event).toBe("end"); + expect(text).toContain('{"a":1,"b":"hello"}'); + }); + + it("streams a small result set as one line per row", async () => { + const { text, control } = await runQuery("SELECT number FROM numbers(5)"); + expect(control.event).toBe("end"); + const lines = text.split("\n").filter((l) => l.length > 0); + expect(lines).toHaveLength(5); + }); + + it("respects the output format chosen via ?format=", async () => { + const sql = "SELECT 42, 'x'"; + + const tsv = await runQuery(sql, { format: "TSV" }); + expect(tsv.control.event).toBe("end"); + expect(tsv.text.trim()).toBe("42\tx"); + + const csv = await runQuery(sql, { format: "CSV" }); + expect(csv.control.event).toBe("end"); + expect(csv.text).toContain('42,"x"'); + + const json = await runQuery(sql, { format: "JSONEachRow" }); + expect(json.control.event).toBe("end"); + const line = json.text.split("\n").find((l) => l.length > 0); + expect(() => JSON.parse(line)).not.toThrow(); + }); + + it("propagates a backend error as an error control event", async () => { + const { control } = await runQuery("SELECT * FROM default.a_missing_table_zz"); + expect(control.event).toBe("error"); + expect(typeof control.message).toBe("string"); + expect(control.message.length).toBeGreaterThan(0); + expect(control.message).toMatch(/table|UNKNOWN_TABLE/i); + }); + + it("streams a large result set without dropping rows", async () => { + const { text, control } = await runQuery("SELECT number FROM numbers(10000)"); + expect(control.event).toBe("end"); + const lines = text.split("\n").filter((l) => l.length > 0); + expect(lines).toHaveLength(10000); + }); + + it("runs multiple queries over one persistent session", async () => { + const s = new Session("JSONEachRow"); + try { + const first = await s.run("SELECT 1 AS n"); + expect(first.control.event).toBe("end"); + expect(first.text).toContain('{"n":1}'); + + const second = await s.run("SELECT 2 AS n"); + expect(second.control.event).toBe("end"); + expect(second.text).toContain('{"n":2}'); + } finally { + s.close(); + } + }); +}); diff --git a/programs/wsproxy/tests/vitest.config.mjs b/programs/wsproxy/tests/vitest.config.mjs new file mode 100644 index 000000000000..775bd666550a --- /dev/null +++ b/programs/wsproxy/tests/vitest.config.mjs @@ -0,0 +1,13 @@ +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { + // Launches the backend ClickHouse server and the proxy (see test/globalSetup.mjs). + globalSetup: "./test/globalSetup.mjs", + // These are integration tests against a single shared backend, so run test + // files one at a time to avoid cross-file interference on shared tables. + fileParallelism: false, + testTimeout: 30_000, + hookTimeout: 60_000, + }, +}); From 72cd4b11166efefd4769fe115f68a50108adfbd9 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 9 Jul 2026 18:00:22 +0200 Subject: [PATCH 12/46] Push server logs and profile events to the client as control frames `executeSelect` forwards `Log` and `ProfileEvents` packets via `sendBlockEvent`, which serializes the block to JSONEachRow and splices the rows into `{"event":"log"|"profile_events","rows":[...]}`. Reusing the output format keeps this correct for whatever columns/types the server sends. `ProfileEvents` arrive automatically; `Log` delivery needs `send_logs_level`, which the backend reads from the query packet's settings (a query-text SETTINGS clause is too late). So a new session parameter `?logs=` is threaded through the handler into `ProxySession` and applied to a mutable `Settings` copy in `sendBackendQuery`. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/ProxySession.cpp | 98 ++++++++++++++++++++++++++--- programs/wsproxy/ProxySession.h | 10 ++- programs/wsproxy/WsProxyHandler.cpp | 21 ++++--- 3 files changed, 112 insertions(+), 17 deletions(-) diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp index 659868c6c04c..e6f94038bf4c 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/programs/wsproxy/ProxySession.cpp @@ -26,6 +26,7 @@ #include #include #include +#include #include #include @@ -189,8 +190,17 @@ String escapeJSON(const String & s) } -ProxySession::ProxySession(Poco::Net::StreamSocket & socket_, ContextPtr context_, BackendParams backend_, String format_) - : socket(socket_), context(std::move(context_)), backend(std::move(backend_)), format(std::move(format_)) +ProxySession::ProxySession( + Poco::Net::StreamSocket & socket_, + ContextPtr context_, + BackendParams backend_, + String format_, + String logs_level_) + : socket(socket_) + , context(std::move(context_)) + , backend(std::move(backend_)) + , format(std::move(format_)) + , logs_level(std::move(logs_level_)) { } @@ -203,9 +213,52 @@ void ProxySession::sendControlEvent(const String & event, const String & message sendWebSocketText(socket, json); } +void ProxySession::sendBlockEvent(const String & event, const Block & block) +{ + if (block.rows() == 0) + return; + + /// Serialize the block to JSONEachRow (one JSON object per row), then splice + /// the rows into a JSON array so the whole batch is one valid control frame: + /// {"event":"log","rows":[{...},{...}]}. Reusing the output format keeps this + /// correct for whatever columns/types the server sends. + WriteBufferFromOwnString buf; + auto out = FormatFactory::instance().getOutputFormat("JSONEachRow", buf, block.cloneEmpty(), context); + out->write(block); + out->finalize(); + const String & rows_text = buf.str(); + + String rows_array; + size_t start = 0; + bool first = true; + while (start < rows_text.size()) + { + const size_t newline = rows_text.find('\n', start); + const size_t end = (newline == String::npos) ? rows_text.size() : newline; + if (end > start) + { + if (!first) + rows_array += ','; + rows_array.append(rows_text, start, end - start); + first = false; + } + if (newline == String::npos) + break; + start = newline + 1; + } + + sendWebSocketText(socket, R"({"event":")" + event + R"(","rows":[)" + rows_array + "]}"); +} + void ProxySession::sendBackendQuery(Connection & connection, const String & query, bool with_pending_data) { - const auto & settings = context->getSettingsRef(); + /// Copy the settings so we can opt into server log delivery. The backend + /// attaches its log queue based on `send_logs_level` in the query packet's + /// settings (a query-text SETTINGS clause is too late), so it must be set here. + Settings settings = context->getSettingsRef(); + if (!logs_level.empty()) + settings.set("send_logs_level", logs_level); + auto timeouts = ConnectionTimeouts::getTCPTimeoutsWithoutFailover(settings); ClientInfo client_info; @@ -458,13 +511,44 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) } break; } - /// Not surfaced to the client yet (profile events / logs / totals push - /// is a later refinement); drain and continue. - case Protocol::Server::ProfileInfo: + case Protocol::Server::Log: + { + /// Server-side log lines for this query (client opts in with + /// `SETTINGS send_logs_level=...`). Push as a control frame. + if (!client_gone) + { + try + { + sendBlockEvent("log", packet.block); + } + catch (...) + { + client_gone = true; + } + } + break; + } case Protocol::Server::ProfileEvents: + { + /// Periodic profile-event counters emitted during execution. + if (!client_gone) + { + try + { + sendBlockEvent("profile_events", packet.block); + } + catch (...) + { + client_gone = true; + } + } + break; + } + /// Not surfaced to the client yet (totals / extremes / profile info); + /// drain and continue. + case Protocol::Server::ProfileInfo: case Protocol::Server::Totals: case Protocol::Server::Extremes: - case Protocol::Server::Log: case Protocol::Server::TableColumns: case Protocol::Server::PartUUIDs: case Protocol::Server::TimezoneUpdate: diff --git a/programs/wsproxy/ProxySession.h b/programs/wsproxy/ProxySession.h index 64589c189617..1583ef09fa9a 100644 --- a/programs/wsproxy/ProxySession.h +++ b/programs/wsproxy/ProxySession.h @@ -34,7 +34,12 @@ struct BackendParams class ProxySession { public: - ProxySession(Poco::Net::StreamSocket & socket_, ContextPtr context_, BackendParams backend_, String format_); + ProxySession( + Poco::Net::StreamSocket & socket_, + ContextPtr context_, + BackendParams backend_, + String format_, + String logs_level_ = ""); void run(); @@ -54,11 +59,14 @@ class ProxySession void sendBackendQuery(Connection & connection, const String & query, bool with_pending_data = false); void drainUntilEndOfStream(Connection & connection); void sendControlEvent(const String & event, const String & message); + /// Serialize a Log / ProfileEvents block to a `{"event":...,"rows":[...]}` text frame. + void sendBlockEvent(const String & event, const Block & block); Poco::Net::StreamSocket & socket; ContextPtr context; BackendParams backend; String format; + String logs_level; /// If set, sent as `send_logs_level` so the backend pushes Log packets. }; } diff --git a/programs/wsproxy/WsProxyHandler.cpp b/programs/wsproxy/WsProxyHandler.cpp index 2510e18bbd64..33d9e1f11075 100644 --- a/programs/wsproxy/WsProxyHandler.cpp +++ b/programs/wsproxy/WsProxyHandler.cpp @@ -56,24 +56,23 @@ bool hasUpgradeToken(const String & connection_header) return false; } -/// Output format from the `format` query parameter of the WebSocket URL. -String outputFormatFromURI(const String & uri_string, LoggerPtr log) +/// Read a query parameter from the WebSocket URL, returning `fallback` if absent. +String queryParam(const String & uri_string, const String & name, const String & fallback, LoggerPtr log) { - String out_format = "JSONEachRow"; try { Poco::URI uri(uri_string); for (const auto & param : uri.getQueryParameters()) { - if (param.first == "format" && !param.second.empty()) - out_format = param.second; + if (param.first == name && !param.second.empty()) + return param.second; } } catch (...) { - LOG_DEBUG(log, "Could not parse request URI for the format parameter; using default"); + LOG_DEBUG(log, "Could not parse request URI for the {} parameter; using default", name); } - return out_format; + return fallback; } } @@ -162,14 +161,18 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp } }); - const String out_format = outputFormatFromURI(request.getURI(), log); + const String & uri = request.getURI(); + const String out_format = queryParam(uri, "format", "JSONEachRow", log); + /// Optional: `?logs=` (e.g. information, trace) makes the backend push + /// server-side log lines for the session's queries. + const String logs_level = queryParam(uri, "logs", "", log); LOG_DEBUG(log, "WebSocket session established; output format {}", out_format); /// Bound each blocking WebSocket read so a stalled client cannot pin the /// handler thread indefinitely between queries. socket.setReceiveTimeout(Poco::Timespan(300, 0)); - ProxySession session(socket, context, backend, out_format); + ProxySession session(socket, context, backend, out_format, logs_level); session.run(); LOG_DEBUG(log, "WebSocket session closed"); From e3885ad30e94d53a40ec54349d3619ede3097d45 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 9 Jul 2026 18:00:22 +0200 Subject: [PATCH 13/46] Test log/profile-events push; collect non-terminal events in the JS helper The helper's `collect()` now returns { data, text, progress, logs, profileEvents, control } and treats only end/error/cancelled as terminal; `Session`/`runQuery`/`urlFor` accept a `logs` option (`?logs=`). New `logs.test.mjs` covers log push via `?logs=trace`, no logs at the default level, and profile events during a slow query. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 19 +++++++++++ programs/wsproxy/tests/test/helpers.mjs | 37 +++++++++++++-------- programs/wsproxy/tests/test/logs.test.mjs | 39 +++++++++++++++++++++++ 3 files changed, 81 insertions(+), 14 deletions(-) create mode 100644 programs/wsproxy/tests/test/logs.test.mjs diff --git a/TODO.md b/TODO.md index 0eb5b75d77bb..a73bf420ea23 100644 --- a/TODO.md +++ b/TODO.md @@ -230,6 +230,25 @@ suites updated to skip progress frames and still green). Still next: profile-events / log push, config-file/auth, `BaseDaemon` hardening. +## Step 6 status — profile-events / log push DONE (branch `wsproxy-skeleton`) + +`executeSelect` now forwards `Log` and `ProfileEvents` packets as JSON control frames via +`sendBlockEvent`: each block is serialized to `JSONEachRow` and spliced into +`{"event":"log"|"profile_events","rows":[{...},{...}]}` (reusing the output format keeps it +correct for any columns/types). `ProfileEvents` are emitted by the server automatically; `Log` +delivery requires `send_logs_level` — which the backend sets from the *query packet* settings, +not query-text SETTINGS, so the proxy applies it from a new session param `?logs=` +(threaded handler → `ProxySession` → a mutable `Settings` copy in `sendBackendQuery`). + +Client protocol note: `log`/`profile_events` (like `progress`) are non-terminal text frames; +clients accumulate them until the terminal `end`/`error`/`cancelled`. The JS helper's +`collect()` now returns `{ data, text, progress, logs, profileEvents, control }`. + +Validated with `tmp/.../tests/test/logs.test.mjs` (18 tests total, all green): log push via +`?logs=trace`, no logs at the default level, and profile events during a slow query. + +Still next: config-file/auth, `BaseDaemon` hardening. (This completes the mid-query-push story.) + ## Plan 1. **Skeleton.** Standalone `programs/wsproxy/` binary. **DONE — builds, links, runs-to-listen.** diff --git a/programs/wsproxy/tests/test/helpers.mjs b/programs/wsproxy/tests/test/helpers.mjs index eab9ada88dbf..c5eb662350bd 100644 --- a/programs/wsproxy/tests/test/helpers.mjs +++ b/programs/wsproxy/tests/test/helpers.mjs @@ -15,10 +15,11 @@ export const PROXY_URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; -/** Build the WS URL for a given output format and optional path. */ -export function urlFor(format = "JSONEachRow", path = "/") { - const sep = path.includes("?") ? "&" : "?"; - return `${PROXY_URL}${path}${sep}format=${encodeURIComponent(format)}`; +/** Build the WS URL for a given output format and options ({ logs, path }). */ +export function urlFor(format = "JSONEachRow", { logs = "", path = "/" } = {}) { + const params = new URLSearchParams({ format }); + if (logs) params.set("logs", logs); + return `${PROXY_URL}${path}?${params}`; } /** @@ -27,8 +28,8 @@ export function urlFor(format = "JSONEachRow", path = "/") { * `await` them regardless of arrival timing. */ export class Session { - constructor(format = "JSONEachRow", path = "/") { - this.ws = new WebSocket(urlFor(format, path)); + constructor(format = "JSONEachRow", { logs = "", path = "/" } = {}) { + this.ws = new WebSocket(urlFor(format, { logs, path })); this.ws.binaryType = "arraybuffer"; this._queue = []; this._waiters = []; @@ -91,21 +92,29 @@ export class Session { async collect() { const chunks = []; const progress = []; + const logs = []; + const profileEvents = []; + const done = (control) => { + const data = Buffer.concat(chunks); + return { data, text: data.toString("utf8"), progress, logs, profileEvents, control }; + }; for (;;) { const frame = await this.nextFrame(); if (frame.type === "binary") { chunks.push(frame.data); } else if (frame.type === "text") { const msg = JSON.parse(frame.data); - if (msg.event === "progress") progress.push(msg); - else { - const data = Buffer.concat(chunks); - return { data, text: data.toString("utf8"), progress, control: msg }; + // Terminal events end the query; everything else is a mid-query push. + if (msg.event === "end" || msg.event === "error" || msg.event === "cancelled") { + return done(msg); } + if (msg.event === "progress") progress.push(msg); + else if (msg.event === "log") logs.push(msg); + else if (msg.event === "profile_events") profileEvents.push(msg); + // Unknown non-terminal events are ignored. } else { // Socket closed without a terminal control frame. - const data = Buffer.concat(chunks); - return { data, text: data.toString("utf8"), progress, control: { event: "closed" } }; + return done({ event: "closed" }); } } } @@ -140,8 +149,8 @@ export class Session { } /** One-shot: open a session, run a query, close, return the result object. */ -export async function runQuery(sql, { format = "JSONEachRow" } = {}) { - const s = new Session(format); +export async function runQuery(sql, { format = "JSONEachRow", logs = "" } = {}) { + const s = new Session(format, { logs }); try { return await s.run(sql); } finally { diff --git a/programs/wsproxy/tests/test/logs.test.mjs b/programs/wsproxy/tests/test/logs.test.mjs new file mode 100644 index 000000000000..6a1a8fe6ce61 --- /dev/null +++ b/programs/wsproxy/tests/test/logs.test.mjs @@ -0,0 +1,39 @@ +import { describe, it, expect } from "vitest"; +import { runQuery } from "./helpers.mjs"; + +describe("logs and profile events", () => { + it("pushes server log lines when the client opts in via ?logs=", async () => { + const { logs, control } = await runQuery( + "SELECT count() FROM numbers(1000000)", + { logs: "trace" }, + ); + expect(control.event).toBe("end"); + expect(logs.length).toBeGreaterThanOrEqual(1); + + // Each log event carries an array of row objects with a `text` column. + const rows = logs.flatMap((e) => e.rows); + expect(rows.length).toBeGreaterThanOrEqual(1); + expect(typeof rows[0].text).toBe("string"); + expect(rows[0].text.length).toBeGreaterThan(0); + }, 20000); + + it("does not push logs at the default level", async () => { + const { logs, control } = await runQuery("SELECT 1"); + expect(control.event).toBe("end"); + expect(logs.length).toBe(0); + }); + + it("pushes profile events during a query", async () => { + // A slow query gives the server time to emit ProfileEvents packets. + const { profileEvents, control } = await runQuery( + "SELECT sleepEachRow(0.1), number FROM numbers(30) SETTINGS max_block_size = 1", + ); + expect(control.event).toBe("end"); + expect(profileEvents.length).toBeGreaterThanOrEqual(1); + + const rows = profileEvents.flatMap((e) => e.rows); + expect(rows.length).toBeGreaterThanOrEqual(1); + // ProfileEvents rows have `name` and `value` columns. + expect(rows.every((r) => typeof r.name === "string")).toBe(true); + }, 20000); +}); From 261748b4c1594a8e4cd523a398df65b8c1558f36 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 9 Jul 2026 18:32:01 +0200 Subject: [PATCH 14/46] Register aggregate functions for full type coverage; add WSPROXY_PORT MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Integration tests over the type matrix revealed that `AggregateFunction`-typed columns fail with "Unknown aggregate function … while receiving packet": the proxy called only `registerFormats()`, but deserializing an `AggregateFunction` off the native wire needs the aggregate-function registry. Call `registerFunctions()` and `registerAggregateFunctions()` at startup (as `clickhouse-client` does) and link `clickhouse_aggregate_functions`. Also make the listen port configurable via `WSPROXY_PORT` (default 9010) so multiple proxy instances can run, e.g. to point one at a dead backend in resilience tests. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/CMakeLists.txt | 1 + programs/wsproxy/WsProxy.cpp | 10 +++++++++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/programs/wsproxy/CMakeLists.txt b/programs/wsproxy/CMakeLists.txt index a84bb0e8390b..2cdfc5005507 100644 --- a/programs/wsproxy/CMakeLists.txt +++ b/programs/wsproxy/CMakeLists.txt @@ -7,6 +7,7 @@ set (CLICKHOUSE_WSPROXY_SOURCES set (CLICKHOUSE_WSPROXY_LINK PRIVATE + clickhouse_aggregate_functions clickhouse_common_config clickhouse_common_io clickhouse_functions diff --git a/programs/wsproxy/WsProxy.cpp b/programs/wsproxy/WsProxy.cpp index 61cb1c37d304..d8d6c18c2b65 100644 --- a/programs/wsproxy/WsProxy.cpp +++ b/programs/wsproxy/WsProxy.cpp @@ -9,7 +9,9 @@ #include #include +#include #include +#include #include #include @@ -128,11 +130,17 @@ class WsProxyServer : public Poco::Util::ServerApplication global_context->makeGlobalContext(); global_context->setApplicationType(Context::ApplicationType::SERVER); + /// The same registrations clickhouse-client performs: aggregate functions + /// are required to (de)serialize `AggregateFunction`-typed columns off the + /// native wire, and functions back defaults/codecs. `registerFormats` alone + /// is not enough for full type coverage. + registerFunctions(); + registerAggregateFunctions(); registerFormats(); const BackendParams backend = backendParamsFromEnv(); - static constexpr UInt16 port = 9010; + const UInt16 port = static_cast(std::stoul(envOr("WSPROXY_PORT", "9010"))); Poco::Net::ServerSocket socket(port); Poco::ThreadPool server_pool(/* minCapacity= */ 1, /* maxCapacity= */ 16); Poco::Net::HTTPServerParams::Ptr params(new Poco::Net::HTTPServerParams); From 2bb54e060470b59a3c089e5fea3a57e08acb360a Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 9 Jul 2026 18:32:02 +0200 Subject: [PATCH 15/46] Expand integration tests: type matrix, formats, concurrency, resilience Adds vitest suites covering the data-type matrix (Nullable/LowCardinality/Array/Map/Tuple/Decimal/DateTime64/Enum/UUID/IP/JSON/Dynamic/Variant/AggregateFunction, SELECT + INSERT round-trips), the output-format matrix (RowBinary exact bytes, JSON/JSONCompact, Values, Pretty, Native, Parquet/Arrow magic, unknown-format error), concurrency (40 parallel sessions, slow-query fan-out, 100-query session longevity, 1M-row streaming), and resilience (session reuse after error, health after cancel, unreachable-backend error via a second proxy instance). 59 tests total, all green. Helpers gain a `baseUrl` option and a `spawnProxy` utility (test/proc.mjs). Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 29 +++ .../wsproxy/tests/test/concurrency.test.mjs | 72 +++++++ programs/wsproxy/tests/test/formats.test.mjs | 118 +++++++++++ programs/wsproxy/tests/test/helpers.mjs | 14 +- programs/wsproxy/tests/test/proc.mjs | 58 ++++++ .../wsproxy/tests/test/resilience.test.mjs | 71 +++++++ programs/wsproxy/tests/test/types.test.mjs | 195 ++++++++++++++++++ 7 files changed, 550 insertions(+), 7 deletions(-) create mode 100644 programs/wsproxy/tests/test/concurrency.test.mjs create mode 100644 programs/wsproxy/tests/test/formats.test.mjs create mode 100644 programs/wsproxy/tests/test/proc.mjs create mode 100644 programs/wsproxy/tests/test/resilience.test.mjs create mode 100644 programs/wsproxy/tests/test/types.test.mjs diff --git a/TODO.md b/TODO.md index a73bf420ea23..c3659f214066 100644 --- a/TODO.md +++ b/TODO.md @@ -249,6 +249,35 @@ Validated with `tmp/.../tests/test/logs.test.mjs` (18 tests total, all green): l Still next: config-file/auth, `BaseDaemon` hardening. (This completes the mid-query-push story.) +## Production hardening — test push #1 (branch `wsproxy-skeleton`) + +Expanded the integration suite across four areas (type matrix, format matrix, concurrency, +resilience) — 59 tests, all green. This push found and fixed a real bug and surfaced findings: + +**Bug found & fixed:** `AggregateFunction`-typed columns failed with `Unknown aggregate +function sum: while receiving packet` — the proxy called only `registerFormats()`, but +deserializing `AggregateFunction` off the native wire needs the aggregate-function registry. +Fixed by calling `registerFunctions()` + `registerAggregateFunctions()` at startup (matching +`clickhouse-client`) and linking `clickhouse_aggregate_functions`. Directly threatened the +"complete type coverage" claim. + +**Finding (deferred to productionization):** UInt64 renders **unquoted** in JSONEachRow through +the proxy, whereas `clickhouse-client` quotes by default — a format-settings faithfulness gap. +Matters for JS clients (unquoted 64-bit ints lose precision on `JSON.parse`). Root: the proxy +derives `FormatSettings` from a bare `Context`; clients can't control format settings. Fix later: +apply faithful defaults and/or let clients pass settings (URL params / query settings). + +**Confirmed working:** session reuse after a backend error/syntax error (reused `Connection` +survives); proxy stays healthy after a client cancels mid-query; unreachable backend → +`error` event, no hang/crash; 40 concurrent sessions + 1M-row streamed result + 100-query +session longevity. Also added `WSPROXY_PORT` (configurable listen port) to enable multi-instance +resilience tests. + +Coverage still thin / next test pushes: adversarial input (oversized/malformed frames, +slow-loris), backend-drops-mid-query, TLS, large INSERT, protocol-revision skew across server +versions. Productionization track (separate): auth, TLS, config file, resource limits, graceful +drain, `BaseDaemon`, CI wiring. + ## Plan 1. **Skeleton.** Standalone `programs/wsproxy/` binary. **DONE — builds, links, runs-to-listen.** diff --git a/programs/wsproxy/tests/test/concurrency.test.mjs b/programs/wsproxy/tests/test/concurrency.test.mjs new file mode 100644 index 000000000000..e237c7712447 --- /dev/null +++ b/programs/wsproxy/tests/test/concurrency.test.mjs @@ -0,0 +1,72 @@ +import { describe, it, expect } from "vitest"; +import { Session, runQuery } from "./helpers.mjs"; + +// These tests stress the proxy with many simultaneous sessions, long-lived +// sessions, and large streamed results. The proxy uses one native Connection +// per WebSocket session and a Poco thread pool (capacity ~16), so this +// exercises thread-per-session behaviour under load and result streaming. + +const countLines = (text) => text.split("\n").filter((l) => l.length > 0).length; + +describe("concurrency", () => { + it("handles many concurrent independent sessions", async () => { + const N = 40; + const results = await Promise.all( + Array.from({ length: N }, (_, i) => runQuery(`SELECT ${i} AS n`)), + ); + + expect(results).toHaveLength(N); + results.forEach((res, i) => { + expect(res.control.event).toBe("end"); + expect(JSON.parse(res.text.trim()).n).toBe(i); + }); + }, 30000); + + it("handles concurrent slow queries through the thread pool", async () => { + const N = 12; + const sql = + "SELECT sleepEachRow(0.05), number FROM numbers(10) SETTINGS max_block_size = 1"; + const results = await Promise.all( + Array.from({ length: N }, () => runQuery(sql)), + ); + + expect(results).toHaveLength(N); + for (const res of results) { + expect(res.control.event).toBe("end"); + } + }, 30000); + + it("survives many sequential queries on one long-lived session", async () => { + const N = 100; + const s = new Session("JSONEachRow"); + try { + let last; + for (let i = 0; i < N; i++) { + last = await s.run(`SELECT ${i} AS n`); + expect(last.control.event).toBe("end"); + } + expect(JSON.parse(last.text.trim()).n).toBe(N - 1); + } finally { + s.close(); + } + }, 30000); + + it("streams a large result set without dropping rows", async () => { + const { text, control } = await runQuery("SELECT number FROM numbers(1000000)"); + expect(control.event).toBe("end"); + expect(countLines(text)).toBe(1000000); + }, 30000); + + it("handles concurrent large-ish result sets", async () => { + const N = 5; + const results = await Promise.all( + Array.from({ length: N }, () => runQuery("SELECT number FROM numbers(100000)")), + ); + + expect(results).toHaveLength(N); + for (const res of results) { + expect(res.control.event).toBe("end"); + expect(countLines(res.text)).toBe(100000); + } + }, 30000); +}); diff --git a/programs/wsproxy/tests/test/formats.test.mjs b/programs/wsproxy/tests/test/formats.test.mjs new file mode 100644 index 000000000000..754afaf23fed --- /dev/null +++ b/programs/wsproxy/tests/test/formats.test.mjs @@ -0,0 +1,118 @@ +import { describe, it, expect } from "vitest"; +import { runQuery } from "./helpers.mjs"; + +// Converting between ClickHouse output formats is the proxy's whole purpose, +// so exercise a broad spread of them: fixed-byte binary formats, text formats +// with and without headers, structured JSON, and container formats with magic +// bytes (Parquet, Arrow). The row-per-line formats are covered elsewhere. +describe("output formats", () => { + it("RowBinary emits exact little-endian bytes", async () => { + const u32 = await runQuery("SELECT toUInt32(42) AS v", { format: "RowBinary" }); + expect(u32.control.event).toBe("end"); + expect(u32.data.equals(Buffer.from([42, 0, 0, 0]))).toBe(true); + + const two = await runQuery("SELECT toUInt8(1), toUInt8(2)", { format: "RowBinary" }); + expect(two.control.event).toBe("end"); + expect(two.data.equals(Buffer.from([1, 2]))).toBe(true); + }); + + it("TabSeparated / TSV separates columns with tabs", async () => { + const tsv = await runQuery("SELECT 1 AS a, 'x' AS b", { format: "TabSeparated" }); + expect(tsv.control.event).toBe("end"); + expect(tsv.text.trim()).toBe("1\tx"); + + const tsvAlias = await runQuery("SELECT 1 AS a, 'x' AS b", { format: "TSV" }); + expect(tsvAlias.control.event).toBe("end"); + expect(tsvAlias.text.trim()).toBe("1\tx"); + }); + + it("TSVWithNames prepends a header row", async () => { + const { text, control } = await runQuery("SELECT 1 AS a, 'x' AS b", { + format: "TSVWithNames", + }); + expect(control.event).toBe("end"); + const lines = text.split("\n"); + expect(lines[0]).toBe("a\tb"); + expect(lines[1]).toBe("1\tx"); + }); + + it("CSV quotes strings", async () => { + const { text, control } = await runQuery("SELECT 1, 'x'", { format: "CSV" }); + expect(control.event).toBe("end"); + expect(text).toContain('1,"x"'); + }); + + it("CSVWithNames prepends a header row", async () => { + const { text, control } = await runQuery("SELECT 1 AS a, 'x' AS b", { + format: "CSVWithNames", + }); + expect(control.event).toBe("end"); + const lines = text.split("\n").filter((l) => l.length > 0); + expect(lines[0]).toContain('"a"'); + expect(lines[0]).toContain('"b"'); + }); + + it("JSON produces a full object with meta and data", async () => { + const { text, control } = await runQuery("SELECT 1 AS a", { format: "JSON" }); + expect(control.event).toBe("end"); + const obj = JSON.parse(text); + expect(Array.isArray(obj.data)).toBe(true); + expect(Array.isArray(obj.meta)).toBe(true); + expect(obj.data[0].a).toBeDefined(); + }); + + it("JSONCompactEachRow emits an array per row", async () => { + const { text, control } = await runQuery("SELECT 1 AS a, 'x' AS b", { + format: "JSONCompactEachRow", + }); + expect(control.event).toBe("end"); + const line = text.split("\n").find((l) => l.length > 0); + expect(JSON.parse(line)).toEqual([1, "x"]); + }); + + it("Values renders a tuple literal", async () => { + const { text, control } = await runQuery("SELECT 1, 'x'", { format: "Values" }); + expect(control.event).toBe("end"); + expect(text.trim()).toBe("(1,'x')"); + }); + + it("Pretty renders a table (smoke)", async () => { + const { text, control } = await runQuery("SELECT 1 AS a", { format: "Pretty" }); + expect(control.event).toBe("end"); + expect(text.length).toBeGreaterThan(0); + // Box-drawing characters and/or the column name should be present. + expect(text.includes("┏") || text.includes("─") || text.includes("a")).toBe(true); + }); + + it("Native returns non-empty binary (smoke)", async () => { + const { data, control } = await runQuery("SELECT number FROM numbers(3)", { + format: "Native", + }); + expect(control.event).toBe("end"); + expect(data.length).toBeGreaterThan(0); + }); + + it("Parquet is framed by PAR1 magic bytes", async () => { + const { data, control } = await runQuery("SELECT number FROM numbers(10)", { + format: "Parquet", + }); + expect(control.event).toBe("end"); + expect(data.length).toBeGreaterThan(8); + expect(data.subarray(0, 4).toString("latin1")).toBe("PAR1"); + expect(data.subarray(-4).toString("latin1")).toBe("PAR1"); + }); + + it("Arrow starts with the ARROW1 file magic", async () => { + const { data, control } = await runQuery("SELECT number FROM numbers(10)", { + format: "Arrow", + }); + expect(control.event).toBe("end"); + expect(data.length).toBeGreaterThan(0); + expect(data.subarray(0, 6).toString("latin1")).toBe("ARROW1"); + }); + + it("an unknown format surfaces an error cleanly", async () => { + const { control } = await runQuery("SELECT 1", { format: "NoSuchFormat" }); + expect(control.event).toBe("error"); + }); +}); diff --git a/programs/wsproxy/tests/test/helpers.mjs b/programs/wsproxy/tests/test/helpers.mjs index c5eb662350bd..d849f84de9b5 100644 --- a/programs/wsproxy/tests/test/helpers.mjs +++ b/programs/wsproxy/tests/test/helpers.mjs @@ -15,11 +15,11 @@ export const PROXY_URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; -/** Build the WS URL for a given output format and options ({ logs, path }). */ -export function urlFor(format = "JSONEachRow", { logs = "", path = "/" } = {}) { +/** Build the WS URL for a given output format and options ({ logs, path, baseUrl }). */ +export function urlFor(format = "JSONEachRow", { logs = "", path = "/", baseUrl = PROXY_URL } = {}) { const params = new URLSearchParams({ format }); if (logs) params.set("logs", logs); - return `${PROXY_URL}${path}?${params}`; + return `${baseUrl}${path}?${params}`; } /** @@ -28,8 +28,8 @@ export function urlFor(format = "JSONEachRow", { logs = "", path = "/" } = {}) { * `await` them regardless of arrival timing. */ export class Session { - constructor(format = "JSONEachRow", { logs = "", path = "/" } = {}) { - this.ws = new WebSocket(urlFor(format, { logs, path })); + constructor(format = "JSONEachRow", { logs = "", path = "/", baseUrl = PROXY_URL } = {}) { + this.ws = new WebSocket(urlFor(format, { logs, path, baseUrl })); this.ws.binaryType = "arraybuffer"; this._queue = []; this._waiters = []; @@ -149,8 +149,8 @@ export class Session { } /** One-shot: open a session, run a query, close, return the result object. */ -export async function runQuery(sql, { format = "JSONEachRow", logs = "" } = {}) { - const s = new Session(format, { logs }); +export async function runQuery(sql, { format = "JSONEachRow", logs = "", baseUrl = PROXY_URL } = {}) { + const s = new Session(format, { logs, baseUrl }); try { return await s.run(sql); } finally { diff --git a/programs/wsproxy/tests/test/proc.mjs b/programs/wsproxy/tests/test/proc.mjs new file mode 100644 index 000000000000..177a2ec2af17 --- /dev/null +++ b/programs/wsproxy/tests/test/proc.mjs @@ -0,0 +1,58 @@ +// Helper to spawn an extra proxy instance with a custom listen port and backend, +// used by resilience tests (e.g. pointing the proxy at a dead backend). + +import { spawn } from "node:child_process"; +import net from "node:net"; +import { dirname, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const testDir = dirname(fileURLToPath(import.meta.url)); +const repoRoot = resolve(testDir, "../../../.."); +const WSPROXY_BIN = + process.env.WSPROXY_BIN ?? resolve(repoRoot, "build/programs/wsproxy/clickhouse-wsproxy"); + +function portOpen(port, host = "127.0.0.1") { + return new Promise((res) => { + const socket = net.connect({ port, host }, () => { + socket.destroy(); + res(true); + }); + socket.on("error", () => res(false)); + }); +} + +async function waitForPort(port, timeoutMs = 15_000) { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (await portOpen(port)) return; + await new Promise((r) => setTimeout(r, 100)); + } + throw new Error(`Timed out waiting for proxy port ${port}`); +} + +/** + * Spawn a proxy on `listenPort` pointing at `backendHost:backendPort`. + * Returns { url, stop } where stop() kills the process. + */ +export async function spawnProxy({ listenPort, backendHost = "127.0.0.1", backendPort }) { + const proc = spawn(WSPROXY_BIN, [], { + stdio: "ignore", + env: { + ...process.env, + WSPROXY_PORT: String(listenPort), + WSPROXY_BACKEND_HOST: backendHost, + WSPROXY_BACKEND_PORT: String(backendPort), + }, + }); + await waitForPort(listenPort); + return { + url: `ws://127.0.0.1:${listenPort}`, + stop() { + try { + proc.kill("SIGKILL"); + } catch { + /* already gone */ + } + }, + }; +} diff --git a/programs/wsproxy/tests/test/resilience.test.mjs b/programs/wsproxy/tests/test/resilience.test.mjs new file mode 100644 index 000000000000..22cd9fabd0f7 --- /dev/null +++ b/programs/wsproxy/tests/test/resilience.test.mjs @@ -0,0 +1,71 @@ +import { describe, it, expect } from "vitest"; +import { runQuery, Session } from "./helpers.mjs"; +import { spawnProxy } from "./proc.mjs"; + +describe("resilience", () => { + it("reuses the session connection after a backend (semantic) error", async () => { + const s = new Session("JSONEachRow"); + const bad = await s.run("SELECT * FROM default.no_such_table_zz"); + expect(bad.control.event).toBe("error"); + + // The reused per-session Connection must still work for the next query. + const good = await s.run("SELECT 1 AS n"); + expect(good.control.event).toBe("end"); + expect(JSON.parse(good.text.trim()).n).toBe(1); + s.close(); + }); + + it("reuses the session after a syntax error", async () => { + const s = new Session("JSONEachRow"); + const bad = await s.run("SELCT 1"); // typo -> backend syntax error + expect(bad.control.event).toBe("error"); + + const good = await s.run("SELECT 2 AS n"); + expect(good.control.event).toBe("end"); + expect(JSON.parse(good.text.trim()).n).toBe(2); + s.close(); + }); + + it("stays healthy after a client cancels a query mid-stream", async () => { + // Session A: start a slow query, then cancel by closing the socket. + const a = new Session("JSONEachRow"); + await a.ready(); + a.sendQuery("SELECT sleepEachRow(0.2), number FROM numbers(50) SETTINGS max_block_size = 1"); + for (let i = 0; i < 50; i++) { + const f = await a.nextFrame(); + if (f.type === "binary" || f.type === "close") break; + } + a.close(); + + // Session B: the proxy still serves new sessions. + const b = await runQuery("SELECT 42 AS n"); + expect(b.control.event).toBe("end"); + expect(JSON.parse(b.text.trim()).n).toBe(42); + }, 20000); + + it("returns an error (no hang or crash) when the backend is unreachable", async () => { + // A second proxy instance pointed at a closed backend port. + const proxy = await spawnProxy({ listenPort: 9011, backendPort: 59999 }); + try { + const { control } = await runQuery("SELECT 1", { baseUrl: proxy.url }); + // Either a delivered error event or a clean socket close is acceptable; + // what matters is that it neither hangs nor takes down the proxy. + expect(["error", "closed"]).toContain(control.event); + } finally { + proxy.stop(); + } + }, 20000); + + it("keeps serving after an unreachable-backend session", async () => { + const proxy = await spawnProxy({ listenPort: 9012, backendPort: 59999 }); + try { + await runQuery("SELECT 1", { baseUrl: proxy.url }); // fails, but must not crash the proxy + const ok = await runQuery("SELECT 7 AS n", { baseUrl: proxy.url }).catch(() => null); + // The second connection still gets a response (another error), proving the + // proxy process survived the first failure. + expect(ok === null || ["error", "closed"].includes(ok.control.event)).toBe(true); + } finally { + proxy.stop(); + } + }, 20000); +}); diff --git a/programs/wsproxy/tests/test/types.test.mjs b/programs/wsproxy/tests/test/types.test.mjs new file mode 100644 index 000000000000..abff499f0f21 --- /dev/null +++ b/programs/wsproxy/tests/test/types.test.mjs @@ -0,0 +1,195 @@ +import { describe, it, expect } from "vitest"; +import { runQuery, backendScalar, Session } from "./helpers.mjs"; + +// Round-trip a typed literal through the proxy's format layer, rendering it as +// JSONEachRow, and return the parsed `v` value. This validates that the proxy +// faithfully carries ClickHouse's data types through the output format. +async function selectJSON(expr) { + const { text, control } = await runQuery(`SELECT ${expr} AS v`, { + format: "JSONEachRow", + }); + expect(control.event).toBe("end"); + return JSON.parse(text.trim()).v; +} + +describe("type coverage", () => { + it("round-trips integers and floats", async () => { + expect(await selectJSON("toInt64(-5)")).toBe(-5); + // 64-bit integers must survive intact. NOTE: the proxy currently renders + // UInt64 UNQUOTED in JSONEachRow, whereas clickhouse-client quotes by + // default (output_format_json_quote_64bit_integers) — a format-settings + // faithfulness gap for the productionization track (unquoted big ints lose + // precision when JSON.parsed in JS). Assert on the raw digits so this tests + // data integrity independent of quoting. + const u64 = await runQuery("SELECT toUInt64(18446744073709551615) AS v", { + format: "JSONEachRow", + }); + expect(u64.control.event).toBe("end"); + expect(u64.text).toContain("18446744073709551615"); + expect(await selectJSON("toFloat64(1.5)")).toBe(1.5); + }); + + it("round-trips a Decimal", async () => { + // Decimals render as JSON numbers in JSONEachRow. + expect(await selectJSON("toDecimal64(3.14, 2)")).toBe(3.14); + }); + + it("round-trips strings with unicode and escapes", async () => { + expect(await selectJSON("'a\\tb\\n\"c\"'")).toBe('a\tb\n"c"'); + expect(await selectJSON("'héllo'")).toBe("héllo"); + }); + + it("round-trips date and time types", async () => { + expect(await selectJSON("toDate('2020-01-02')")).toBe("2020-01-02"); + expect(await selectJSON("toDateTime('2020-01-02 03:04:05')")).toBe( + "2020-01-02 03:04:05", + ); + expect(await selectJSON("toDateTime64('2020-01-02 03:04:05.123', 3)")).toBe( + "2020-01-02 03:04:05.123", + ); + }); + + it("round-trips a UUID", async () => { + expect( + await selectJSON("toUUID('00000000-0000-0000-0000-000000000001')"), + ).toBe("00000000-0000-0000-0000-000000000001"); + }); + + it("round-trips IPv4 and IPv6", async () => { + expect(await selectJSON("toIPv4('1.2.3.4')")).toBe("1.2.3.4"); + expect(await selectJSON("toIPv6('::1')")).toBe("::1"); + }); + + it("round-trips an Enum", async () => { + expect(await selectJSON("CAST('a', 'Enum8(\\'a\\'=1,\\'b\\'=2)')")).toBe("a"); + }); + + it("round-trips Nullable (null and non-null)", async () => { + expect(await selectJSON("CAST(NULL, 'Nullable(String)')")).toBeNull(); + expect(await selectJSON("CAST('present', 'Nullable(String)')")).toBe( + "present", + ); + }); + + it("round-trips LowCardinality", async () => { + expect(await selectJSON("CAST('x', 'LowCardinality(String)')")).toBe("x"); + }); + + it("round-trips an Array", async () => { + expect(await selectJSON("[1,2,3]::Array(UInt32)")).toEqual([1, 2, 3]); + }); + + it("round-trips a Map (rendered as a JSON object)", async () => { + // UInt8 values fit in JS numbers, so no 64-bit quoting here. + expect(await selectJSON("map('k',1,'j',2)")).toEqual({ k: 1, j: 2 }); + }); + + it("round-trips a Tuple", async () => { + // JSONEachRow renders unnamed tuples as an object keyed by position + // ({"1":1,"2":"x"}); assert loosely that the values are present. + const v = await selectJSON("tuple(1,'x')"); + const values = Array.isArray(v) ? v : Object.values(v); + expect(values).toContain(1); + expect(values).toContain("x"); + }); +}); + +describe("advanced types", () => { + it("handles the JSON type", async () => { + const { text, control } = await runQuery( + `SELECT '{"a":1,"b":"x"}'::JSON AS v`, + { format: "JSONEachRow" }, + ); + expect(control.event).toBe("end"); + expect(text).toContain("a"); + expect(text).toContain("1"); + }); + + it("handles the Dynamic type", async () => { + const v = await selectJSON("42::Dynamic"); + // The value survives; may arrive as a number or a string depending on + // Dynamic's JSON rendering, so just assert it is present and equals 42. + expect(v).not.toBeUndefined(); + expect(v).not.toBeNull(); + expect(String(v)).toBe("42"); + }); + + it("handles the Variant type", async () => { + // Cast from a type that IS in the Variant (UInt64), not a bare UInt8 literal. + const { control, text } = await runQuery( + "SELECT CAST(toUInt64(42), 'Variant(UInt64, String)') AS v", + { format: "JSONEachRow" }, + ); + expect(control.event).toBe("end"); + expect(text).toContain("42"); + }); + + it("serializes an AggregateFunction state via a binary format", async () => { + // JSONEachRow cannot represent an aggregate state; RowBinary can. This + // proves the proxy streams aggregate-state bytes without error. + const { data, control } = await runQuery( + "SELECT sumState(number) AS v FROM numbers(10)", + { format: "RowBinary" }, + ); + expect(control.event).toBe("end"); + expect(data.length).toBeGreaterThan(0); + }); +}); + +describe("INSERT round-trip", () => { + // Prove the INPUT path handles complex types end-to-end: create a Memory + // table, insert one row of a complex type via JSONEachRow, then read it back. + async function withTable(columnDef, run) { + const table = `default.wsp_types_${Date.now()}_${Math.floor(Math.random() * 1e6)}`; + const create = await runQuery( + `CREATE TABLE ${table} (${columnDef}) ENGINE = Memory`, + ); + expect(create.control.event).toBe("end"); + try { + await run(table); + } finally { + await runQuery(`DROP TABLE IF EXISTS ${table}`); + } + } + + it("round-trips an Array(UInt32) column", async () => { + await withTable("v Array(UInt32)", async (table) => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + `INSERT INTO ${table} FORMAT JSONEachRow`, + ['{"v":[1,2,3]}\n'], + ); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + expect(await backendScalar(`SELECT v FROM ${table}`)).toBe("[1,2,3]"); + }); + }); + + it("round-trips a Nullable(String) column", async () => { + await withTable("v Nullable(String)", async (table) => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + `INSERT INTO ${table} FORMAT JSONEachRow`, + ['{"v":null}\n{"v":"hi"}\n'], + ); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + // TSV renders SQL NULL as \N; the non-null value round-trips verbatim. + expect(await backendScalar(`SELECT count() FROM ${table}`)).toBe("2"); + expect( + await backendScalar(`SELECT v FROM ${table} WHERE v IS NOT NULL`), + ).toBe("hi"); + expect( + await backendScalar( + `SELECT count() FROM ${table} WHERE v IS NULL`, + ), + ).toBe("1"); + }); + }); +}); From 3dda91b031feadebbbfea5a6840869a349212656 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 9 Jul 2026 19:12:58 +0200 Subject: [PATCH 16/46] Add adversarial, backend-failure, and large-INSERT integration tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit adversarial.test.mjs uses a raw-TCP WebSocket client (test/raw.mjs) — the native global WebSocket only emits well-formed frames — to verify the proxy rejects unmasked frames, reserved opcodes, RSV bits, and oversized-advertised frames with a clean close, answers pings with pongs, and survives malformed input (fresh clients keep working). backend-failure.test.mjs spawns its own backend+proxy (test/proc.mjs `spawnBackend`) and kills the backend mid-query: the proxy delivers a prompt error/close (not a 20s hang) and the process survives. large-insert.test.mjs round-trips 500k rows (JSONEachRow, plus single-frame and TSV) against server-side count/sum. Killing clickhouse-server in tests requires `lsof -ti tcp:` — its watchdog fork survives signalling the spawned pid, the process group, and pkill on the config path. 72 tests total, all green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 23 ++- .../wsproxy/tests/test/adversarial.test.mjs | 171 ++++++++++++++++++ .../tests/test/backend-failure.test.mjs | 67 +++++++ .../wsproxy/tests/test/large-insert.test.mjs | 109 +++++++++++ programs/wsproxy/tests/test/proc.mjs | 78 +++++++- programs/wsproxy/tests/test/raw.mjs | 148 +++++++++++++++ 6 files changed, 590 insertions(+), 6 deletions(-) create mode 100644 programs/wsproxy/tests/test/adversarial.test.mjs create mode 100644 programs/wsproxy/tests/test/backend-failure.test.mjs create mode 100644 programs/wsproxy/tests/test/large-insert.test.mjs create mode 100644 programs/wsproxy/tests/test/raw.mjs diff --git a/TODO.md b/TODO.md index c3659f214066..ac5d0304263d 100644 --- a/TODO.md +++ b/TODO.md @@ -273,10 +273,25 @@ survives); proxy stays healthy after a client cancels mid-query; unreachable bac session longevity. Also added `WSPROXY_PORT` (configurable listen port) to enable multi-instance resilience tests. -Coverage still thin / next test pushes: adversarial input (oversized/malformed frames, -slow-loris), backend-drops-mid-query, TLS, large INSERT, protocol-revision skew across server -versions. Productionization track (separate): auth, TLS, config file, resource limits, graceful -drain, `BaseDaemon`, CI wiring. +### Test push #2 — adversarial / backend-failure / large INSERT (all green, 72 tests) + +- **adversarial.test.mjs** (raw-TCP client `test/raw.mjs`, since native `WebSocket` only + emits well-formed frames): unmasked frame, reserved opcode, RSV bit, and oversized-advertised + frame are all rejected with a clean close; ping→pong; and the proxy survives malformed input + (fresh clients still work). RFC 6455 hardening confirmed — no proxy bugs. +- **backend-failure.test.mjs** (spawns its own backend+proxy on non-default ports): killing the + backend mid-query → the proxy delivers a clean `error`/close **promptly** (~1s, not the 20s + natural query length) and the proxy process survives. Good resilience result. +- **large-insert.test.mjs**: 500k-row streamed JSONEachRow INSERT + 100k single-frame + 100k TSV, + verified against server-side count/sum. Streaming INSERT holds under volume. + +Test-infra learnings: `spawnBackend`/`spawnProxy` in `test/proc.mjs`; killing `clickhouse-server` +requires `lsof -ti tcp:` (its watchdog fork evades signalling the spawned pid, its process +group, and `pkill -f` on the config path). + +Coverage still thin / future pushes: slow-loris/partial-frame timeouts, TLS, protocol-revision +skew across older server versions (need old server binaries). Productionization track (separate): +auth, TLS, config file, resource limits, graceful drain, `BaseDaemon`, CI wiring. ## Plan diff --git a/programs/wsproxy/tests/test/adversarial.test.mjs b/programs/wsproxy/tests/test/adversarial.test.mjs new file mode 100644 index 000000000000..350b30f8a7b1 --- /dev/null +++ b/programs/wsproxy/tests/test/adversarial.test.mjs @@ -0,0 +1,171 @@ +// Adversarial WebSocket-frame tests for the clickhouse-wsproxy. +// +// These exercise the proxy's RFC 6455 hardening: client->server frames MUST be +// masked, reserved opcodes and RSV bits are protocol errors, over-large frames +// are rejected before allocation, control frames (ping) are answered with a +// pong, and — most importantly — none of the above takes down the proxy for +// other clients. We use the raw-TCP client because the native WebSocket cannot +// emit malformed frames. + +import { RawClient } from "./raw.mjs"; +import { runQuery } from "./helpers.mjs"; +import { describe, it, expect } from "vitest"; + +// A protocol violation should make the proxy tear the connection down. The spec +// allows either an outright TCP close or a close (0x8) control frame first, so +// we accept both: read frames until we either see a close-opcode frame or the +// socket goes away. +async function closedOrCloseFrame(c, timeoutMs = 10000) { + const deadline = Date.now() + timeoutMs; + for (;;) { + const remaining = deadline - Date.now(); + if (remaining <= 0) break; + if (await c.waitClose(0)) return true; + const f = await Promise.race([ + c.readFrame(), + new Promise((r) => setTimeout(() => r(null), Math.min(remaining, 300))), + ]); + if (f === null) continue; // timed out waiting for a frame; loop and re-check + if (f.closed) return true; + if (f.opcode === 0x8) return true; // close frame + // Anything else (e.g. a stray pong) is ignored; keep waiting for the close. + } + return c.closed; +} + +describe("adversarial frames", () => { + it( + "rejects an unmasked client frame", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + // Client->server frames must be masked; an unmasked one is a violation. + c.sendFrame({ opcode: 0x1, payload: "SELECT 1", masked: false }); + expect(await closedOrCloseFrame(c)).toBe(true); + c.close(); + }, + 10000, + ); + + it( + "rejects a reserved opcode (0x3)", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + // 0x3-0x7 are reserved non-control opcodes. + c.sendFrame({ opcode: 0x3, payload: "x" }); + expect(await closedOrCloseFrame(c)).toBe(true); + c.close(); + }, + 10000, + ); + + it( + "rejects a frame with an RSV bit set", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + // No extension was negotiated, so any RSV bit is a protocol error. + c.sendFrame({ opcode: 0x1, payload: "SELECT 1", rsv: 1 }); + expect(await closedOrCloseFrame(c)).toBe(true); + c.close(); + }, + 10000, + ); + + it( + "rejects an oversized frame before allocating", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + // Advertise 100 MiB in the header but send no payload bytes. The proxy + // must cap on the advertised length (> 16 MiB) and never allocate it. + c.sendFrame({ opcode: 0x1, payload: Buffer.alloc(0), advertisedLen: 100 * 1024 * 1024 }); + expect(await closedOrCloseFrame(c)).toBe(true); + c.close(); + }, + 10000, + ); + + it( + "answers a ping with a matching pong", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ opcode: 0x9, payload: "hb" }); + const f = await c.readFrame(); + expect(f.opcode).toBe(0xa); + expect(f.payload.toString()).toBe("hb"); + c.close(); + }, + 10000, + ); + + it( + "serves a valid masked query through the raw client", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ opcode: 0x1, payload: "SELECT 1 AS n" }); + + const binary = []; + let sawEnd = false; + for (;;) { + const f = await c.readFrame(); + if (f.closed) break; + if (f.opcode === 0x2) { + // Binary frame: query result bytes. + binary.push(f.payload); + continue; + } + if (f.opcode === 0x1) { + // Text frame: a control event. + const msg = JSON.parse(f.payload.toString()); + // Non-terminal mid-query pushes: keep reading. + if (msg.event === "progress" || msg.event === "log" || msg.event === "profile_events") + continue; + if (msg.event === "end") { + sawEnd = true; + break; + } + // Any other control event (e.g. error) is unexpected here. + throw new Error(`unexpected control event: ${f.payload.toString()}`); + } + // Ignore stray control frames (ping/pong/close-with-data), keep reading. + } + + expect(sawEnd).toBe(true); + expect(Buffer.concat(binary).toString()).toContain('{"n":1}'); + c.close(); + }, + 10000, + ); + + it( + "does not throw on an idle connect-then-disconnect", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + // Never send a query; just drop the connection. + c.close(); + + // The proxy must still serve new clients afterwards. + const r = await runQuery("SELECT 1 AS n"); + expect(r.control.event).toBe("end"); + expect(JSON.parse(r.text.trim()).n).toBe(1); + }, + 10000, + ); + + // Runs last: after all the malformed traffic above, a fresh normal client + // must still work, proving the proxy process survived every violation. + it( + "survives malformed input and keeps serving other clients", + async () => { + const r = await runQuery("SELECT 42 AS n"); + expect(r.control.event).toBe("end"); + expect(JSON.parse(r.text.trim()).n).toBe(42); + }, + 10000, + ); +}); diff --git a/programs/wsproxy/tests/test/backend-failure.test.mjs b/programs/wsproxy/tests/test/backend-failure.test.mjs new file mode 100644 index 000000000000..e72ed7c10354 --- /dev/null +++ b/programs/wsproxy/tests/test/backend-failure.test.mjs @@ -0,0 +1,67 @@ +import { describe, it, expect } from "vitest"; +import { Session, runQuery } from "./helpers.mjs"; +import { spawnBackend, spawnProxy } from "./proc.mjs"; + +// These tests run their OWN backend + proxy pair (on non-default ports) so they +// can kill the backend without disturbing the shared stack used by other files. +describe("backend failure", () => { + it("delivers an error (no hang or crash) when the backend drops mid-query", async () => { + const backend = await spawnBackend({ tcpPort: 9001 }); + const proxy = await spawnProxy({ listenPort: 9013, backendPort: 9001 }); + try { + const s = new Session("JSONEachRow", { baseUrl: proxy.url }); + await s.ready(); + // A long, slowly-streaming query so we can kill the backend mid-flight. + s.sendQuery("SELECT sleepEachRow(0.2), number FROM numbers(100) SETTINGS max_block_size = 1"); + + // Wait until the query is actually streaming (first binary frame). Skip + // non-terminal text frames (progress/log/profile_events); only stop early + // on a close or a terminal control event. + let streaming = false; + for (let i = 0; i < 200; i++) { + const f = await s.nextFrame(); + if (f.type === "binary") { + streaming = true; + break; + } + if (f.type === "close") break; + if (f.type === "text") { + const ev = JSON.parse(f.data).event; + if (ev === "end" || ev === "error" || ev === "cancelled") break; + } + } + expect(streaming).toBe(true); + + // Kill the backend out from under the running query. + backend.stop(); + + // The proxy must terminate the session cleanly (error or close), not hang. + const rest = await s.collect(); + expect(["error", "closed"]).toContain(rest.control.event); + s.close(); + } finally { + proxy.stop(); + backend.stop(); + } + }, 30000); + + it("keeps the proxy process alive after a backend failure", async () => { + const backend = await spawnBackend({ tcpPort: 9002 }); + const proxy = await spawnProxy({ listenPort: 9014, backendPort: 9002 }); + try { + // Sanity: works before the failure. + const before = await runQuery("SELECT 1 AS n", { baseUrl: proxy.url }); + expect(before.control.event).toBe("end"); + + backend.stop(); + + // After the backend is gone, new sessions get a clean error rather than a + // crash/hang — proving the proxy process survived. + const after = await runQuery("SELECT 1 AS n", { baseUrl: proxy.url }).catch(() => null); + expect(after === null || ["error", "closed"].includes(after.control.event)).toBe(true); + } finally { + proxy.stop(); + backend.stop(); + } + }, 30000); +}); diff --git a/programs/wsproxy/tests/test/large-insert.test.mjs b/programs/wsproxy/tests/test/large-insert.test.mjs new file mode 100644 index 000000000000..3f941e41876c --- /dev/null +++ b/programs/wsproxy/tests/test/large-insert.test.mjs @@ -0,0 +1,109 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { Session, runQuery, backendScalar } from "./helpers.mjs"; + +describe("large INSERT", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it( + "streams a JSONEachRow insert of 500,000 rows across multiple frames", + async () => { + const total = 500000; + const rowsPerChunk = 50000; + const chunks = []; + for (let start = 0; start < total; start += rowsPerChunk) { + const rows = []; + for (let i = start; i < start + rowsPerChunk; i++) { + rows.push(`{"a":${i},"b":"r${i}"}\n`); + } + chunks.push(rows.join("")); + } + expect(chunks.length).toBe(10); + + const s = new Session("JSONEachRow"); + let control; + try { + ({ control } = await s.insert( + "INSERT INTO default.wsp_test FORMAT JSONEachRow", + chunks, + )); + } finally { + s.close(); + } + expect(control.event).toBe("end"); + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe( + "500000", + ); + // Exact sum of 0..499999, computed with BigInt to avoid float precision issues. + const expectedSum = ((499999n * 500000n) / 2n).toString(); + expect(expectedSum).toBe("124999750000"); + expect(await backendScalar("SELECT sum(a) FROM default.wsp_test")).toBe( + expectedSum, + ); + }, + 60000, + ); + + it( + "handles a large INSERT in a single big frame", + async () => { + const total = 100000; + const rows = []; + for (let i = 0; i < total; i++) { + rows.push(`{"a":${i},"b":"r${i}"}\n`); + } + const chunk = rows.join(""); + + const s = new Session("JSONEachRow"); + let control; + try { + ({ control } = await s.insert( + "INSERT INTO default.wsp_test FORMAT JSONEachRow", + [chunk], + )); + } finally { + s.close(); + } + expect(control.event).toBe("end"); + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe( + "100000", + ); + }, + 60000, + ); + + it( + "round-trips integrity via a different input format (TSV)", + async () => { + const total = 100000; + const rows = []; + for (let i = 0; i < total; i++) { + rows.push(`${i}\tr${i}\n`); + } + const chunk = rows.join(""); + + const s = new Session("JSONEachRow"); + let control; + try { + ({ control } = await s.insert( + "INSERT INTO default.wsp_test FORMAT TSV", + [chunk], + )); + } finally { + s.close(); + } + expect(control.event).toBe("end"); + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe( + "100000", + ); + expect(await backendScalar("SELECT max(a) FROM default.wsp_test")).toBe( + "99999", + ); + }, + 60000, + ); +}); diff --git a/programs/wsproxy/tests/test/proc.mjs b/programs/wsproxy/tests/test/proc.mjs index 177a2ec2af17..6961d8380851 100644 --- a/programs/wsproxy/tests/test/proc.mjs +++ b/programs/wsproxy/tests/test/proc.mjs @@ -1,15 +1,18 @@ // Helper to spawn an extra proxy instance with a custom listen port and backend, // used by resilience tests (e.g. pointing the proxy at a dead backend). -import { spawn } from "node:child_process"; +import { spawn, spawnSync } from "node:child_process"; +import fs from "node:fs"; import net from "node:net"; -import { dirname, resolve } from "node:path"; +import os from "node:os"; +import { dirname, join, resolve } from "node:path"; import { fileURLToPath } from "node:url"; const testDir = dirname(fileURLToPath(import.meta.url)); const repoRoot = resolve(testDir, "../../../.."); const WSPROXY_BIN = process.env.WSPROXY_BIN ?? resolve(repoRoot, "build/programs/wsproxy/clickhouse-wsproxy"); +const CLICKHOUSE_SERVER = process.env.CLICKHOUSE_SERVER ?? "/usr/local/bin/clickhouse-server"; function portOpen(port, host = "127.0.0.1") { return new Promise((res) => { @@ -30,6 +33,77 @@ async function waitForPort(port, timeoutMs = 15_000) { throw new Error(`Timed out waiting for proxy port ${port}`); } +/** + * Spawn a disposable ClickHouse backend on `tcpPort` with a fresh temp data + * dir, so a test can kill it (e.g. backend-drops-mid-query). Returns + * { tcpPort, stop } where stop() kills the process and removes the temp dir. + */ +export async function spawnBackend({ tcpPort }) { + const dir = fs.mkdtempSync(join(os.tmpdir(), "wsproxy-ch-")); + const configPath = join(dir, "config.xml"); + fs.writeFileSync( + configPath, + ` + none0 + ${dir}/server.log${dir}/server.err.log + ${tcpPort} + 127.0.0.1 + ${dir}/data/ + ${dir}/tmp/ + ${dir}/user_files/ + 536870912 + false + users.xml + default + default +`, + ); + fs.writeFileSync( + join(dir, "users.xml"), + ` + + + ::/0 + defaultdefault + + +`, + ); + + // clickhouse-server forks a watchdog whose child re-parents into its own + // session, so signalling the spawned pid (or its group) leaves the real + // server running. Kill by matching the unique temp config path in argv, which + // reliably hits the actual server process (and the watchdog). + const proc = spawn(CLICKHOUSE_SERVER, ["--config-file", configPath], { stdio: "ignore" }); + await waitForPort(tcpPort, 30_000); + return { + tcpPort, + stop() { + // Kill whatever holds the TCP port — reliably the real server process, + // regardless of the watchdog fork/rename/re-parent shenanigans. + const r = spawnSync("lsof", ["-ti", `tcp:${tcpPort}`, "-sTCP:LISTEN"], { encoding: "utf8" }); + for (const pid of (r.stdout || "").split("\n").map((s) => s.trim()).filter(Boolean)) { + try { + process.kill(Number(pid), "SIGKILL"); + } catch { + /* already gone */ + } + } + spawnSync("pkill", ["-9", "-f", configPath]); + try { + proc.kill("SIGKILL"); + } catch { + /* already gone */ + } + try { + fs.rmSync(dir, { recursive: true, force: true }); + } catch { + /* ignore */ + } + }, + }; +} + /** * Spawn a proxy on `listenPort` pointing at `backendHost:backendPort`. * Returns { url, stop } where stop() kills the process. diff --git a/programs/wsproxy/tests/test/raw.mjs b/programs/wsproxy/tests/test/raw.mjs new file mode 100644 index 000000000000..f808e7c3d2e2 --- /dev/null +++ b/programs/wsproxy/tests/test/raw.mjs @@ -0,0 +1,148 @@ +// A raw-TCP WebSocket client for adversarial tests. Unlike the native global +// WebSocket, this lets us craft malformed frames (unmasked, reserved opcodes, +// oversized advertised lengths, bad fragmentation) to exercise the proxy's RFC +// 6455 hardening. + +import net from "node:net"; +import crypto from "node:crypto"; + +const GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; + +export class RawClient { + constructor({ host = "127.0.0.1", port = 9010 } = {}) { + this.socket = net.connect({ host, port }); + this.socket.setNoDelay(true); + this.buf = Buffer.alloc(0); + this.closed = false; + this._wake = null; + this.socket.on("data", (chunk) => { + this.buf = Buffer.concat([this.buf, chunk]); + this._signal(); + }); + this.socket.on("close", () => { + this.closed = true; + this._signal(); + }); + this.socket.on("error", () => { + this.closed = true; + this._signal(); + }); + } + + _signal() { + if (this._wake) { + const w = this._wake; + this._wake = null; + w(); + } + } + + _waitData() { + return new Promise((resolve) => (this._wake = resolve)); + } + + connected() { + return new Promise((resolve, reject) => { + this.socket.once("connect", resolve); + this.socket.once("error", reject); + }); + } + + /** Perform the WebSocket upgrade handshake; resolves when the 101 is received. */ + async handshake(path = "/?format=JSONEachRow") { + await this.connected(); + const key = crypto.randomBytes(16).toString("base64"); + this.socket.write( + `GET ${path} HTTP/1.1\r\nHost: 127.0.0.1\r\n` + + "Upgrade: websocket\r\nConnection: Upgrade\r\n" + + `Sec-WebSocket-Key: ${key}\r\nSec-WebSocket-Version: 13\r\n\r\n`, + ); + while (this.buf.indexOf("\r\n\r\n") === -1 && !this.closed) await this._waitData(); + const idx = this.buf.indexOf("\r\n\r\n"); + if (idx === -1) throw new Error("connection closed during handshake"); + const head = this.buf.subarray(0, idx).toString("latin1"); + this.buf = this.buf.subarray(idx + 4); + if (!head.includes("101 Switching Protocols")) throw new Error(`no 101:\n${head}`); + const expect = crypto.createHash("sha1").update(key + GUID).digest("base64"); + if (!head.includes(`Sec-WebSocket-Accept: ${expect}`)) throw new Error("bad accept"); + return head; + } + + /** + * Send a WebSocket frame with full control over the header. + * opts: { opcode, payload (Buffer|string), fin=true, masked=true, rsv=0, + * advertisedLen (override the length field to lie about payload size) } + */ + sendFrame({ opcode, payload = Buffer.alloc(0), fin = true, masked = true, rsv = 0, advertisedLen }) { + if (typeof payload === "string") payload = Buffer.from(payload); + const len = advertisedLen ?? payload.length; + const header = []; + header.push((fin ? 0x80 : 0) | (rsv << 4) | (opcode & 0x0f)); + let ext = Buffer.alloc(0); + const maskBit = masked ? 0x80 : 0; + if (len < 126) { + header.push(maskBit | len); + } else if (len < 65536) { + header.push(maskBit | 126); + ext = Buffer.alloc(2); + ext.writeUInt16BE(len); + } else { + header.push(maskBit | 127); + ext = Buffer.alloc(8); + ext.writeBigUInt64BE(BigInt(len)); + } + let body = payload; + let maskKey = Buffer.alloc(0); + if (masked) { + maskKey = crypto.randomBytes(4); + body = Buffer.from(payload); + for (let i = 0; i < body.length; i++) body[i] ^= maskKey[i % 4]; + } + this.socket.write(Buffer.concat([Buffer.from(header), ext, maskKey, body])); + } + + /** Read one server->client frame, or {closed:true} if the socket closes first. */ + async readFrame() { + const need = async (n) => { + while (this.buf.length < n && !this.closed) await this._waitData(); + return this.buf.length >= n; + }; + if (!(await need(2))) return { closed: true }; + const b0 = this.buf[0]; + const b1 = this.buf[1]; + const opcode = b0 & 0x0f; + const fin = (b0 & 0x80) !== 0; + let len = b1 & 0x7f; + let offset = 2; + if (len === 126) { + if (!(await need(4))) return { closed: true }; + len = this.buf.readUInt16BE(2); + offset = 4; + } else if (len === 127) { + if (!(await need(10))) return { closed: true }; + len = Number(this.buf.readBigUInt64BE(2)); + offset = 10; + } + if (!(await need(offset + len))) return { closed: true }; + const payload = this.buf.subarray(offset, offset + len); + this.buf = this.buf.subarray(offset + len); + return { opcode, fin, payload: Buffer.from(payload) }; + } + + /** Resolve once the socket has closed (with a timeout guard). */ + async waitClose(timeoutMs = 5000) { + const deadline = Date.now() + timeoutMs; + while (!this.closed && Date.now() < deadline) { + await Promise.race([this._waitData(), new Promise((r) => setTimeout(r, 200))]); + } + return this.closed; + } + + close() { + try { + this.socket.destroy(); + } catch { + /* ignore */ + } + } +} From 265f30b6140a383e964b3ba2f15953e2e598ae47 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 9 Jul 2026 19:28:01 +0200 Subject: [PATCH 17/46] Add exception-path integration tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Covers the error paths not previously exercised: an exception delivered AFTER partial results have streamed (rows 0-4 then the error), session reuse after a mid-stream exception, malformed and mistyped INSERT data (aborts via sendCancel with zero rows committed), faithful ClickHouse message/code preservation, and session health after an INSERT error. All pass — the proxy's exception handling is correct; no code changes needed. Notes a feature gap for later: INSERT-side progress (written_rows) is not forwarded (executeInsert ignores Progress packets). 78 tests total, all green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 14 ++++ .../wsproxy/tests/test/exceptions.test.mjs | 78 +++++++++++++++++++ 2 files changed, 92 insertions(+) create mode 100644 programs/wsproxy/tests/test/exceptions.test.mjs diff --git a/TODO.md b/TODO.md index ac5d0304263d..046346d943ce 100644 --- a/TODO.md +++ b/TODO.md @@ -289,6 +289,20 @@ Test-infra learnings: `spawnBackend`/`spawnProxy` in `test/proc.mjs`; killing `c requires `lsof -ti tcp:` (its watchdog fork evades signalling the spawned pid, its process group, and `pkill -f` on the config path). +### Test push #3 — exceptions (all green, 78 tests) + +**exceptions.test.mjs** (6 tests) — probed the proxy's error paths, all behave correctly (no bugs): +- Exception *after* partial results: rows 0–4 stream, then the `error` event (the `Exception`-after-`Data` path: `output.reset()`/`out_buf.cancel()`). +- Session reuse after a mid-stream exception (reused `Connection` survives). +- Malformed INSERT data → `error` **and 0 rows committed** (abort via `sendCancel`, no partial commit). +- Type error in INSERT data → clean parse error. +- ClickHouse error message/code preserved faithfully. +- Session healthy after an INSERT error (fresh insert commits). + +Progress was already covered for SELECT (progress.test.mjs). **Feature gap:** INSERT-side progress +(`written_rows`) is NOT forwarded — `executeInsert` ignores `Progress`. Worth adding (small) if +INSERT progress matters to clients. + Coverage still thin / future pushes: slow-loris/partial-frame timeouts, TLS, protocol-revision skew across older server versions (need old server binaries). Productionization track (separate): auth, TLS, config file, resource limits, graceful drain, `BaseDaemon`, CI wiring. diff --git a/programs/wsproxy/tests/test/exceptions.test.mjs b/programs/wsproxy/tests/test/exceptions.test.mjs new file mode 100644 index 000000000000..7e9c3b51b1aa --- /dev/null +++ b/programs/wsproxy/tests/test/exceptions.test.mjs @@ -0,0 +1,78 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { runQuery, Session, backendScalar } from "./helpers.mjs"; + +describe("exceptions", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it("streams partial results, then delivers the exception", async () => { + // throwIf fires at number = 5, so rows 0..4 stream first, then the error. + const { text, control } = await runQuery( + "SELECT number, throwIf(number = 5, 'boom') AS t FROM numbers(10) SETTINGS max_block_size = 1", + ); + expect(control.event).toBe("error"); + expect(control.message).toMatch(/boom/); + const lines = text.split("\n").filter((l) => l.length > 0); + expect(lines.length).toBe(5); // partial results were delivered before the error + // First streamed row is number 0 (quoting-agnostic: the proxy renders UInt64 unquoted). + expect(String(JSON.parse(lines[0]).number)).toBe("0"); + }); + + it("reuses the session after a mid-stream exception", async () => { + const s = new Session("JSONEachRow"); + const bad = await s.run( + "SELECT throwIf(number = 3, 'x') FROM numbers(10) SETTINGS max_block_size = 1", + ); + expect(bad.control.event).toBe("error"); + // The reused Connection must survive an exception that occurred mid-stream. + const good = await s.run("SELECT 1 AS n"); + expect(good.control.event).toBe("end"); + expect(JSON.parse(good.text.trim()).n).toBe(1); + s.close(); + }); + + it("aborts an INSERT on malformed data without committing partial rows", async () => { + const s = new Session("JSONEachRow"); + const { control } = await s.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", [ + '{"a":1,"b":"x"}\n', + "NOT JSON AT ALL\n", + ]); + s.close(); + expect(control.event).toBe("error"); + expect(control.message).toMatch(/Cannot parse|parse input|Code: 27/); + // The valid first row must NOT be committed — the insert aborted. + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("0"); + }); + + it("reports a type error in INSERT data", async () => { + const s = new Session("JSONEachRow"); + const { control } = await s.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", [ + '{"a":"notanumber","b":"x"}\n', + ]); + s.close(); + expect(control.event).toBe("error"); + expect(control.message).toMatch(/parse|Cannot|Code: 27/); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("0"); + }); + + it("preserves the ClickHouse error message and code", async () => { + const { control } = await runQuery("SELECT * FROM default.definitely_missing_tbl_xyz"); + expect(control.event).toBe("error"); + expect(control.message).toContain("definitely_missing_tbl_xyz"); + expect(control.message).toMatch(/Unknown table|UNKNOWN_TABLE|Code: 60|Code: 47/); + }); + + it("keeps the session healthy after an INSERT error", async () => { + const s = new Session("JSONEachRow"); + const bad = await s.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", ["garbage\n"]); + expect(bad.control.event).toBe("error"); + s.close(); + // A fresh session works, and a valid insert then commits. + const s2 = new Session("JSONEachRow"); + const ok = await s2.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", ['{"a":7,"b":"z"}\n']); + s2.close(); + expect(ok.control.event).toBe("end"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("1"); + }); +}); From d861c61a5a5022f2a6d0ed7d691e5d79e096e541 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 9 Jul 2026 20:07:49 +0200 Subject: [PATCH 18/46] Add edge-case and INSERT-flow integration tests edge.test.mjs: empty result sets, LIMIT 0, DDL, many-columns-with-NULLs/arrays, and empty-string/unicode INSERT round-trips. insert-flow.test.mjs: closing the socket mid-INSERT aborts via sendCancel with zero rows committed, and the proxy stays healthy for a subsequent clean insert. Also investigated INSERT written-rows progress: the native protocol emits no Progress packets for client-data INSERTs (0 events even for 1M rows), so the forwarding was reverted rather than shipped untested. Noted a separate limitation: no-client-data INSERTs (INSERT ... SELECT / FROM INFILE) would make executeInsert wait for data that never arrives. 85 tests total, all green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 19 +++++-- programs/wsproxy/tests/test/edge.test.mjs | 53 +++++++++++++++++++ .../wsproxy/tests/test/insert-flow.test.mjs | 40 ++++++++++++++ 3 files changed, 109 insertions(+), 3 deletions(-) create mode 100644 programs/wsproxy/tests/test/edge.test.mjs create mode 100644 programs/wsproxy/tests/test/insert-flow.test.mjs diff --git a/TODO.md b/TODO.md index 046346d943ce..849c169dfa21 100644 --- a/TODO.md +++ b/TODO.md @@ -299,9 +299,22 @@ group, and `pkill -f` on the config path). - ClickHouse error message/code preserved faithfully. - Session healthy after an INSERT error (fresh insert commits). -Progress was already covered for SELECT (progress.test.mjs). **Feature gap:** INSERT-side progress -(`written_rows`) is NOT forwarded — `executeInsert` ignores `Progress`. Worth adding (small) if -INSERT progress matters to clients. +Progress was already covered for SELECT (progress.test.mjs). + +### Test push #4 — edge cases + INSERT flow (all green, 85 tests) + +- **edge.test.mjs**: empty result set, `LIMIT 0` (header only), DDL (CREATE/DROP → `end`, no data), + many columns + NULLs + arrays in one row, empty-string/unicode INSERT round-trip. +- **insert-flow.test.mjs**: closing the socket mid-INSERT aborts via `sendCancel` with **0 rows + committed** (no partial commit), and the proxy stays healthy (a clean INSERT commits afterwards). + +**INSERT-progress investigated and dropped:** implemented `written_rows` forwarding, but the native +protocol emits **no** `Progress` packets for client-data INSERTs (verified: 0 events even for 1M +rows into MergeTree, 146ms). Reverted the dormant forwarding to avoid unverified/dead code. Server +INSERT progress would only appear for `INSERT ... SELECT` (server-side), which the current +`executeInsert` streaming model doesn't handle — noted as a separate limitation: +`INSERT ... SELECT` / `INSERT ... FROM INFILE` (no client data) would make `executeInsert` wait for +WS data that never comes. Fix later (detect no-data INSERTs and skip the data-streaming phase). Coverage still thin / future pushes: slow-loris/partial-frame timeouts, TLS, protocol-revision skew across older server versions (need old server binaries). Productionization track (separate): diff --git a/programs/wsproxy/tests/test/edge.test.mjs b/programs/wsproxy/tests/test/edge.test.mjs new file mode 100644 index 000000000000..88b858faa063 --- /dev/null +++ b/programs/wsproxy/tests/test/edge.test.mjs @@ -0,0 +1,53 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { runQuery, Session, backendScalar } from "./helpers.mjs"; + +describe("edge cases", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it("handles an empty result set", async () => { + const { text, control } = await runQuery("SELECT 1 AS n WHERE 0"); + expect(control.event).toBe("end"); + expect(text.split("\n").filter((l) => l.length > 0).length).toBe(0); + }); + + it("handles LIMIT 0 (header only, no rows)", async () => { + const { text, control } = await runQuery("SELECT number FROM numbers(100) LIMIT 0"); + expect(control.event).toBe("end"); + expect(text.split("\n").filter((l) => l.length > 0).length).toBe(0); + }); + + it("runs DDL and reports end with no data", async () => { + const create = await runQuery( + "CREATE TABLE IF NOT EXISTS default.wsp_edge_ddl (x UInt8) ENGINE = Memory", + ); + expect(create.control.event).toBe("end"); + expect(create.text.trim()).toBe(""); + const drop = await runQuery("DROP TABLE default.wsp_edge_ddl"); + expect(drop.control.event).toBe("end"); + }); + + it("handles many columns and NULLs in one row", async () => { + const { text, control } = await runQuery( + "SELECT 1 AS a, NULL AS b, 'x' AS c, [1,2] AS d, toFloat64(1.5) AS e", + ); + expect(control.event).toBe("end"); + const row = JSON.parse(text.trim()); + expect(row.b).toBe(null); + expect(row.c).toBe("x"); + expect(row.d).toEqual([1, 2]); + }); + + it("handles an empty string and unicode payload round-trip via INSERT", async () => { + const s = new Session("JSONEachRow"); + const ok = await s.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", [ + '{"a":1,"b":""}\n{"a":2,"b":"日本語"}\n', + ]); + s.close(); + expect(ok.control.event).toBe("end"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + expect(await backendScalar("SELECT b FROM default.wsp_test WHERE a = 2")).toBe("日本語"); + expect(await backendScalar("SELECT length(b) FROM default.wsp_test WHERE a = 1")).toBe("0"); + }); +}); diff --git a/programs/wsproxy/tests/test/insert-flow.test.mjs b/programs/wsproxy/tests/test/insert-flow.test.mjs new file mode 100644 index 000000000000..d124124f25cd --- /dev/null +++ b/programs/wsproxy/tests/test/insert-flow.test.mjs @@ -0,0 +1,40 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { runQuery, Session, backendScalar, sleep } from "./helpers.mjs"; + +describe("insert flow", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it("aborts an INSERT (no commit) when the client closes mid-stream", async () => { + const s = new Session("JSONEachRow"); + await s.ready(); + s.sendQuery("INSERT INTO default.wsp_test FORMAT JSONEachRow"); + // Send some data but NEVER send the end marker; close the socket instead. + s.sendData('{"a":1,"b":"x"}\n{"a":2,"b":"y"}\n'); + await sleep(200); + s.close(); // Close mid-stream -> proxy calls sendCancel, backend rolls back. + + // Give the backend a moment to process the cancel, then verify nothing committed. + await sleep(500); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("0"); + }, 20000); + + it("commits a normal INSERT after a mid-stream abort (proxy stays healthy)", async () => { + // Abort one insert... + const a = new Session("JSONEachRow"); + await a.ready(); + a.sendQuery("INSERT INTO default.wsp_test FORMAT JSONEachRow"); + a.sendData('{"a":9,"b":"q"}\n'); + await sleep(150); + a.close(); + await sleep(300); + + // ...then a clean insert still works. + const b = new Session("JSONEachRow"); + const ok = await b.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", ['{"a":1,"b":"x"}\n']); + b.close(); + expect(ok.control.event).toBe("end"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("1"); + }, 20000); +}); From 7c2eb230bc1f424a57119f69318def37f63ac4a9 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 9 Jul 2026 20:14:47 +0200 Subject: [PATCH 19/46] Add credential pass-through authentication The proxy resolves per-session backend credentials from the WebSocket upgrade request and opens the backend Connection as that user, delegating authN/authZ to ClickHouse. Priority: Authorization: Basic, then X-ClickHouse-User/-Key headers, then ?user=/?password= URL params, else the configured WSPROXY_BACKEND_* defaults. A per-session BackendParams copy is used (the shared default is never mutated); bad credentials surface as the backend's auth error. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/WsProxyHandler.cpp | 52 +++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-) diff --git a/programs/wsproxy/WsProxyHandler.cpp b/programs/wsproxy/WsProxyHandler.cpp index 33d9e1f11075..fa3db0867409 100644 --- a/programs/wsproxy/WsProxyHandler.cpp +++ b/programs/wsproxy/WsProxyHandler.cpp @@ -5,6 +5,7 @@ #include #include +#include #include #include @@ -75,6 +76,48 @@ String queryParam(const String & uri_string, const String & name, const String & return fallback; } +/// Resolve backend credentials for this session (credential pass-through: the +/// backend performs authentication). Priority: `Authorization: Basic`, then +/// `X-ClickHouse-User`/`-Key` headers, then `?user=`/`?password=` URL params, +/// else the configured defaults already in `backend`. +void resolveCredentials(const HTTPServerRequest & request, const String & uri, BackendParams & backend, LoggerPtr log) +{ + const String auth = request.get("Authorization", ""); + if (auth.starts_with("Basic ")) + { + try + { + const String decoded = base64Decode(auth.substr(6)); + const size_t colon = decoded.find(':'); + if (colon != String::npos) + { + backend.user = decoded.substr(0, colon); + backend.password = decoded.substr(colon + 1); + return; + } + } + catch (...) + { + LOG_DEBUG(log, "Malformed Authorization header; falling back to other credential sources"); + } + } + + const String header_user = request.get("X-ClickHouse-User", ""); + if (!header_user.empty()) + { + backend.user = header_user; + backend.password = request.get("X-ClickHouse-Key", ""); + return; + } + + const String param_user = queryParam(uri, "user", "", log); + if (!param_user.empty()) + { + backend.user = param_user; + backend.password = queryParam(uri, "password", "", log); + } +} + } WsProxyHandler::WsProxyHandler(ContextPtr context_, BackendParams backend_) @@ -166,13 +209,18 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp /// Optional: `?logs=` (e.g. information, trace) makes the backend push /// server-side log lines for the session's queries. const String logs_level = queryParam(uri, "logs", "", log); - LOG_DEBUG(log, "WebSocket session established; output format {}", out_format); + + /// Credential pass-through: resolve this session's backend user/password from + /// the request (never mutate the shared default `backend`). + BackendParams session_backend = backend; + resolveCredentials(request, uri, session_backend, log); + LOG_DEBUG(log, "WebSocket session established; format {}, backend user {}", out_format, session_backend.user); /// Bound each blocking WebSocket read so a stalled client cannot pin the /// handler thread indefinitely between queries. socket.setReceiveTimeout(Poco::Timespan(300, 0)); - ProxySession session(socket, context, backend, out_format, logs_level); + ProxySession session(socket, context, session_backend, out_format, logs_level); session.run(); LOG_DEBUG(log, "WebSocket session closed"); From 6514698ef88958db8b294e4d7ac70581d55b820e Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 9 Jul 2026 20:14:47 +0200 Subject: [PATCH 20/46] Test auth; make the integration suite self-contained auth.test.mjs covers default-user, ?user=/?password= URL params, Authorization: Basic, and X-ClickHouse-User/-Key headers (header cases via the raw-TCP client, since native/browser WebSocket cannot set request headers), plus wrong-password rejection. Helpers gain user/password URL options and raw handshake header support. globalSetup now generates its own backend config (default + a password-protected wsp_user) via spawnBackend/spawnProxy instead of depending on the gitignored tmp/ch config, so the suite is reproducible from a clean checkout. 91 tests total, all green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 23 ++++++ programs/wsproxy/tests/test/auth.test.mjs | 77 +++++++++++++++++++++ programs/wsproxy/tests/test/globalSetup.mjs | 48 ++++--------- programs/wsproxy/tests/test/helpers.mjs | 20 ++++-- programs/wsproxy/tests/test/proc.mjs | 15 ++-- programs/wsproxy/tests/test/raw.mjs | 9 ++- 6 files changed, 145 insertions(+), 47 deletions(-) create mode 100644 programs/wsproxy/tests/test/auth.test.mjs diff --git a/TODO.md b/TODO.md index 849c169dfa21..49cb77c3fdff 100644 --- a/TODO.md +++ b/TODO.md @@ -320,6 +320,29 @@ Coverage still thin / future pushes: slow-loris/partial-frame timeouts, TLS, pro skew across older server versions (need old server binaries). Productionization track (separate): auth, TLS, config file, resource limits, graceful drain, `BaseDaemon`, CI wiring. +## Productionization — Auth DONE (credential pass-through, branch `wsproxy-skeleton`) + +The proxy no longer holds fixed backend creds only from env — it resolves per-session credentials +from the WebSocket upgrade request and opens the backend `Connection` **as that user**, delegating +all authN/authZ to ClickHouse (no auth system reinvented). `WsProxyHandler::resolveCredentials` +priority: `Authorization: Basic` → `X-ClickHouse-User`/`-Key` headers → `?user=`/`?password=` URL +params → the configured `WSPROXY_BACKEND_*` defaults. A per-session `BackendParams` copy is used +(never mutating the shared default). Bad credentials surface as the backend's auth error event +(pass-through; lazy connect on first query). + +Tested in `auth.test.mjs` (6 tests): default user, URL params, `Basic` header, `X-ClickHouse` +headers, and wrong-password rejection via URL params and `Basic`. Header-based creds are tested via +the raw-TCP client (native/browser `WebSocket` can't set request headers — so browsers must use URL +params, which leak in URLs/logs unless behind TLS; note for the TLS work). + +Also made the integration suite **self-contained**: `globalSetup` now generates the backend config +(default + `wsp_user`) via `spawnBackend`/`spawnProxy` instead of depending on gitignored +`tmp/ch/*`. 91 tests total, all green. This is most of what CI wiring needs (just a +`clickhouse-server` binary + the built proxy). + +Auth follow-ups (later): reject bad creds at handshake (eager connect → close with reason) instead +of on first query; TLS (both legs) so URL-param creds and header creds aren't sent in clear. + ## Plan 1. **Skeleton.** Standalone `programs/wsproxy/` binary. **DONE — builds, links, runs-to-listen.** diff --git a/programs/wsproxy/tests/test/auth.test.mjs b/programs/wsproxy/tests/test/auth.test.mjs new file mode 100644 index 000000000000..5d1035f4c91f --- /dev/null +++ b/programs/wsproxy/tests/test/auth.test.mjs @@ -0,0 +1,77 @@ +import { describe, it, expect } from "vitest"; +import { runQuery } from "./helpers.mjs"; +import { RawClient } from "./raw.mjs"; + +// The test backend (tmp/ch/users.xml) has a password-protected user: +const USER = "wsp_user"; +const PASS = "wsp_pass"; + +// Run `SELECT currentUser()` through a raw client that authenticates via the +// given HTTP request headers; returns the authenticated user name. +async function currentUserViaHeaders(headers) { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow", headers); + c.sendFrame({ opcode: 0x1, payload: "SELECT currentUser() AS u" }); + const chunks = []; + let control = null; + for (;;) { + const f = await c.readFrame(); + if (f.closed) break; + if (f.opcode === 0x2) chunks.push(f.payload); + else if (f.opcode === 0x1) { + const ev = JSON.parse(f.payload.toString()); + if (ev.event === "progress" || ev.event === "log" || ev.event === "profile_events") continue; + control = ev; + break; + } + } + c.close(); + const text = Buffer.concat(chunks).toString(); + return { control, user: text.trim() ? JSON.parse(text.trim()).u : null }; +} + +describe("auth (credential pass-through)", () => { + it("authenticates as the default user when no credentials are given", async () => { + const { text, control } = await runQuery("SELECT currentUser() AS u"); + expect(control.event).toBe("end"); + expect(JSON.parse(text.trim()).u).toBe("default"); + }); + + it("authenticates via ?user=/?password= URL params", async () => { + const { text, control } = await runQuery("SELECT currentUser() AS u", { + user: USER, + password: PASS, + }); + expect(control.event).toBe("end"); + expect(JSON.parse(text.trim()).u).toBe(USER); + }); + + it("rejects a wrong password (backend auth error passed through)", async () => { + const { control } = await runQuery("SELECT 1", { user: USER, password: "wrong" }); + expect(control.event).toBe("error"); + expect(control.message).toMatch(/auth|password|ACCESS|denied/i); + }); + + it("authenticates via Authorization: Basic header", async () => { + const basic = Buffer.from(`${USER}:${PASS}`).toString("base64"); + const { control, user } = await currentUserViaHeaders({ Authorization: `Basic ${basic}` }); + expect(control.event).toBe("end"); + expect(user).toBe(USER); + }); + + it("authenticates via X-ClickHouse-User / X-ClickHouse-Key headers", async () => { + const { control, user } = await currentUserViaHeaders({ + "X-ClickHouse-User": USER, + "X-ClickHouse-Key": PASS, + }); + expect(control.event).toBe("end"); + expect(user).toBe(USER); + }); + + it("rejects a wrong password sent via Basic header", async () => { + const basic = Buffer.from(`${USER}:wrong`).toString("base64"); + const { control } = await currentUserViaHeaders({ Authorization: `Basic ${basic}` }); + expect(control.event).toBe("error"); + expect(control.message).toMatch(/auth|password|ACCESS|denied/i); + }); +}); diff --git a/programs/wsproxy/tests/test/globalSetup.mjs b/programs/wsproxy/tests/test/globalSetup.mjs index 792d44f50f70..c1359ba6776d 100644 --- a/programs/wsproxy/tests/test/globalSetup.mjs +++ b/programs/wsproxy/tests/test/globalSetup.mjs @@ -1,21 +1,13 @@ // Vitest global setup: ensure a ClickHouse backend (:9000) and the proxy (:9010) // are running, create the shared test table, and tear down whatever we started. // -// Reuses anything already listening (so you can run against a stack you started -// by hand); only processes we spawn here are killed on teardown. +// Self-contained: the backend config (including the auth-test user) is generated +// by spawnBackend, so no external config files are needed. Anything already +// listening is reused (so you can run against a stack you started by hand); only +// processes we spawn here are stopped on teardown. -import { spawn } from "node:child_process"; import net from "node:net"; -import { dirname, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; - -const testDir = dirname(fileURLToPath(import.meta.url)); -const repoRoot = resolve(testDir, "../../../.."); // programs/wsproxy/tests/test -> repo root - -const CLICKHOUSE_SERVER = process.env.CLICKHOUSE_SERVER ?? "/usr/local/bin/clickhouse-server"; -const CH_CONFIG = process.env.CH_CONFIG ?? resolve(repoRoot, "tmp/ch/config.xml"); -const WSPROXY_BIN = - process.env.WSPROXY_BIN ?? resolve(repoRoot, "build/programs/wsproxy/clickhouse-wsproxy"); +import { spawnBackend, spawnProxy } from "./proc.mjs"; const BACKEND_PORT = 9000; const PROXY_PORT = 9010; @@ -30,31 +22,17 @@ function portOpen(port, host = "127.0.0.1") { }); } -async function waitForPort(port, timeoutMs = 30_000) { - const deadline = Date.now() + timeoutMs; - while (Date.now() < deadline) { - if (await portOpen(port)) return; - await new Promise((r) => setTimeout(r, 200)); - } - throw new Error(`Timed out waiting for port ${port}`); -} - export default async function setup() { - const spawned = []; + const stops = []; if (!(await portOpen(BACKEND_PORT))) { - const proc = spawn(CLICKHOUSE_SERVER, ["--config-file", CH_CONFIG], { - stdio: "ignore", - detached: false, - }); - spawned.push(proc); - await waitForPort(BACKEND_PORT); + const backend = await spawnBackend({ tcpPort: BACKEND_PORT }); + stops.push(() => backend.stop()); } if (!(await portOpen(PROXY_PORT))) { - const proc = spawn(WSPROXY_BIN, [], { stdio: "ignore", detached: false }); - spawned.push(proc); - await waitForPort(PROXY_PORT); + const proxy = await spawnProxy({ listenPort: PROXY_PORT, backendPort: BACKEND_PORT }); + stops.push(() => proxy.stop()); } // Create the shared test table via the proxy (DDL goes through the SELECT path). @@ -66,11 +44,11 @@ export default async function setup() { throw new Error(`failed to create test table: ${JSON.stringify(ddl.control)}`); } - // Teardown: kill only what we started. + // Teardown: stop only what we started. return () => { - for (const proc of spawned) { + for (const stop of stops) { try { - proc.kill("SIGTERM"); + stop(); } catch { /* ignore */ } diff --git a/programs/wsproxy/tests/test/helpers.mjs b/programs/wsproxy/tests/test/helpers.mjs index d849f84de9b5..652c63ecc38c 100644 --- a/programs/wsproxy/tests/test/helpers.mjs +++ b/programs/wsproxy/tests/test/helpers.mjs @@ -15,10 +15,15 @@ export const PROXY_URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; -/** Build the WS URL for a given output format and options ({ logs, path, baseUrl }). */ -export function urlFor(format = "JSONEachRow", { logs = "", path = "/", baseUrl = PROXY_URL } = {}) { +/** Build the WS URL for a format and options ({ logs, path, baseUrl, user, password }). */ +export function urlFor( + format = "JSONEachRow", + { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "" } = {}, +) { const params = new URLSearchParams({ format }); if (logs) params.set("logs", logs); + if (user) params.set("user", user); + if (password) params.set("password", password); return `${baseUrl}${path}?${params}`; } @@ -28,8 +33,8 @@ export function urlFor(format = "JSONEachRow", { logs = "", path = "/", baseUrl * `await` them regardless of arrival timing. */ export class Session { - constructor(format = "JSONEachRow", { logs = "", path = "/", baseUrl = PROXY_URL } = {}) { - this.ws = new WebSocket(urlFor(format, { logs, path, baseUrl })); + constructor(format = "JSONEachRow", { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "" } = {}) { + this.ws = new WebSocket(urlFor(format, { logs, path, baseUrl, user, password })); this.ws.binaryType = "arraybuffer"; this._queue = []; this._waiters = []; @@ -149,8 +154,11 @@ export class Session { } /** One-shot: open a session, run a query, close, return the result object. */ -export async function runQuery(sql, { format = "JSONEachRow", logs = "", baseUrl = PROXY_URL } = {}) { - const s = new Session(format, { logs, baseUrl }); +export async function runQuery( + sql, + { format = "JSONEachRow", logs = "", baseUrl = PROXY_URL, user = "", password = "" } = {}, +) { + const s = new Session(format, { logs, baseUrl, user, password }); try { return await s.run(sql); } finally { diff --git a/programs/wsproxy/tests/test/proc.mjs b/programs/wsproxy/tests/test/proc.mjs index 6961d8380851..8b0a02f06791 100644 --- a/programs/wsproxy/tests/test/proc.mjs +++ b/programs/wsproxy/tests/test/proc.mjs @@ -62,10 +62,17 @@ export async function spawnBackend({ tcpPort }) { join(dir, "users.xml"), ` - - ::/0 - defaultdefault - + + + ::/0 + defaultdefault + + + + wsp_pass::/0 + defaultdefault + + `, ); diff --git a/programs/wsproxy/tests/test/raw.mjs b/programs/wsproxy/tests/test/raw.mjs index f808e7c3d2e2..61fe73a12e3d 100644 --- a/programs/wsproxy/tests/test/raw.mjs +++ b/programs/wsproxy/tests/test/raw.mjs @@ -48,13 +48,18 @@ export class RawClient { }); } - /** Perform the WebSocket upgrade handshake; resolves when the 101 is received. */ - async handshake(path = "/?format=JSONEachRow") { + /** Perform the WebSocket upgrade handshake; resolves when the 101 is received. + * Extra HTTP request headers can be supplied via `headers` (e.g. Authorization). */ + async handshake(path = "/?format=JSONEachRow", headers = {}) { await this.connected(); const key = crypto.randomBytes(16).toString("base64"); + const extra = Object.entries(headers) + .map(([k, v]) => `${k}: ${v}\r\n`) + .join(""); this.socket.write( `GET ${path} HTTP/1.1\r\nHost: 127.0.0.1\r\n` + "Upgrade: websocket\r\nConnection: Upgrade\r\n" + + extra + `Sec-WebSocket-Key: ${key}\r\nSec-WebSocket-Version: 13\r\n\r\n`, ); while (this.buf.indexOf("\r\n\r\n") === -1 && !this.closed) await this._waitData(); From 9730e1d27a78f2b516fb22254d52fbcce024ebc8 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 11:15:16 +0200 Subject: [PATCH 21/46] Authenticate eagerly at session start ProxySession::run now calls Connection::forceConnected before the query loop, so authentication (performed during the native handshake) is validated up front. On failure the client gets an error control frame plus a 1008 (policy violation) WebSocket close before sending any query, instead of the error surfacing only on the first query. Added an auth test asserting the rejection arrives with no query sent. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 7 +++++-- programs/wsproxy/ProxySession.cpp | 23 +++++++++++++++++++++++ programs/wsproxy/tests/test/auth.test.mjs | 20 +++++++++++++++++++- 3 files changed, 47 insertions(+), 3 deletions(-) diff --git a/TODO.md b/TODO.md index 49cb77c3fdff..84a58a017e34 100644 --- a/TODO.md +++ b/TODO.md @@ -340,8 +340,11 @@ Also made the integration suite **self-contained**: `globalSetup` now generates `tmp/ch/*`. 91 tests total, all green. This is most of what CI wiring needs (just a `clickhouse-server` binary + the built proxy). -Auth follow-ups (later): reject bad creds at handshake (eager connect → close with reason) instead -of on first query; TLS (both legs) so URL-param creds and header creds aren't sent in clear. +Eager connect DONE: `run()` calls `Connection::forceConnected` up front, so authentication (which +happens during the native handshake) is validated at session start — bad creds get an `error` +control frame + a `1008` WebSocket close *before any query is sent* (tested). 92 tests, all green. + +Auth follow-up (later): TLS (both legs) so URL-param and header creds aren't sent in clear. ## Plan diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp index e6f94038bf4c..7a2af7f6207c 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/programs/wsproxy/ProxySession.cpp @@ -701,6 +701,29 @@ void ProxySession::run() /* tls_sni_override_ */ "", /* bind_host_ */ ""); + /// Establish the backend connection up front so authentication (which happens + /// during the native handshake) is reported immediately, rather than on the + /// first query. On failure, tell the client and close. + try + { + connection.forceConnected(ConnectionTimeouts::getTCPTimeoutsWithoutFailover(context->getSettingsRef())); + } + catch (...) + { + const String message = getCurrentExceptionMessage(false); + LOG_DEBUG(log, "Backend connect/auth failed: {}", message); + try + { + sendControlEvent("error", message); + sendWebSocketClose(socket, /* 1008 policy violation */ 1008, "Authentication failed"); + } + catch (...) + { + LOG_DEBUG(log, "Failed to deliver auth error to client (already gone)"); + } + return; + } + LOG_DEBUG(log, "Proxy session started; backend {}:{}", backend.host, backend.port); while (true) diff --git a/programs/wsproxy/tests/test/auth.test.mjs b/programs/wsproxy/tests/test/auth.test.mjs index 5d1035f4c91f..f3f9fd06ba25 100644 --- a/programs/wsproxy/tests/test/auth.test.mjs +++ b/programs/wsproxy/tests/test/auth.test.mjs @@ -1,5 +1,5 @@ import { describe, it, expect } from "vitest"; -import { runQuery } from "./helpers.mjs"; +import { runQuery, Session } from "./helpers.mjs"; import { RawClient } from "./raw.mjs"; // The test backend (tmp/ch/users.xml) has a password-protected user: @@ -68,6 +68,24 @@ describe("auth (credential pass-through)", () => { expect(user).toBe(USER); }); + it("rejects bad credentials eagerly, before any query is sent", async () => { + const s = new Session("JSONEachRow", { user: USER, password: "wrong" }); + await s.ready(); + // Do NOT send a query. Eager connect authenticates at session start, so the + // failure must arrive without the client sending anything. + let event = null; + for (let i = 0; i < 4; i++) { + const f = await s.nextFrame(); + if (f.type === "text") { + event = JSON.parse(f.data).event; + break; + } + if (f.type === "close") break; + } + s.close(); + expect(event).toBe("error"); + }, 10000); + it("rejects a wrong password sent via Basic header", async () => { const basic = Buffer.from(`${USER}:wrong`).toString("base64"); const { control } = await currentUserViaHeaders({ Authorization: `Basic ${basic}` }); From b91bc255c4e2f54d941aba4565efdf47c1de50c9 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 11:23:07 +0200 Subject: [PATCH 22/46] Prioritize TLS by leg: proxy->backend first, client->proxy least Client->proxy TLS is deprioritized for now: the proxy is a sidecar next to the app, so that leg is typically loopback / in-pod or TLS-terminated at the ingress. proxy->backend TLS stays the priority leg since it crosses the network to the (remote/managed) ClickHouse. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/TODO.md b/TODO.md index 84a58a017e34..dfca76800302 100644 --- a/TODO.md +++ b/TODO.md @@ -318,7 +318,8 @@ WS data that never comes. Fix later (detect no-data INSERTs and skip the data-st Coverage still thin / future pushes: slow-loris/partial-frame timeouts, TLS, protocol-revision skew across older server versions (need old server binaries). Productionization track (separate): -auth, TLS, config file, resource limits, graceful drain, `BaseDaemon`, CI wiring. +auth (done), proxy→backend TLS, config file, resource limits, graceful drain, `BaseDaemon`, +CI wiring. Client→proxy TLS is LEAST priority (sidecar loopback / ingress-terminated). ## Productionization — Auth DONE (credential pass-through, branch `wsproxy-skeleton`) @@ -344,7 +345,13 @@ Eager connect DONE: `run()` calls `Connection::forceConnected` up front, so auth happens during the native handshake) is validated at session start — bad creds get an `error` control frame + a `1008` WebSocket close *before any query is sent* (tested). 92 tests, all green. -Auth follow-up (later): TLS (both legs) so URL-param and header creds aren't sent in clear. +Auth follow-up — TLS, split by leg and priority: +- **proxy → backend TLS (higher priority):** this leg crosses the network to the (remote/managed) + ClickHouse, so credentials and data must be encrypted here. `Connection` already supports it + (`Protocol::Secure`), currently `Disable`; wire it up + config. +- **client → proxy TLS (LEAST priority for now):** deprioritized — the proxy is a sidecar next to + the app, so this leg is typically loopback / in-pod (or TLS is terminated at the ingress / + service mesh). Revisit only if the proxy is exposed beyond the app's trust boundary. ## Plan From 6c8f88c8f8387a798cd1027f1fa22d0eb3fa72bc Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 11:39:19 +0200 Subject: [PATCH 23/46] Support TLS to the backend (proxy -> backend leg) WSPROXY_BACKEND_SECURE=1 opens the backend Connection over the native secure protocol (Protocol::Secure::Enable). For self-signed / dev certs, WSPROXY_BACKEND_ACCEPT_INVALID_CERT=1 sets openSSL.client.invalidCertificateHandler=AcceptCertificateHandler and verificationMode=none in the app config, which Poco's SSLManager reads when lazily building the default client context (mirrors clickhouse-client --accept-invalid-certificate). BackendParams gains a per-session secure flag. This encrypts the leg that crosses the network to the (remote/managed) ClickHouse. Production follow-up: CA-based verification instead of accept-invalid. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/ProxySession.cpp | 2 +- programs/wsproxy/ProxySession.h | 1 + programs/wsproxy/WsProxy.cpp | 20 ++++++++++++++++++++ 3 files changed, 22 insertions(+), 1 deletion(-) diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp index 7a2af7f6207c..66f30e47ffbf 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/programs/wsproxy/ProxySession.cpp @@ -697,7 +697,7 @@ void ProxySession::run() /* cluster_secret_ */ "", /* client_name_ */ "clickhouse-wsproxy", Protocol::Compression::Enable, - Protocol::Secure::Disable, + backend.secure ? Protocol::Secure::Enable : Protocol::Secure::Disable, /* tls_sni_override_ */ "", /* bind_host_ */ ""); diff --git a/programs/wsproxy/ProxySession.h b/programs/wsproxy/ProxySession.h index 1583ef09fa9a..206201eead54 100644 --- a/programs/wsproxy/ProxySession.h +++ b/programs/wsproxy/ProxySession.h @@ -21,6 +21,7 @@ struct BackendParams String user = "default"; String password; String database; + bool secure = false; /// Connect to the backend over TLS (native secure protocol). }; /// Drives one WebSocket session against a backend ClickHouse server over the diff --git a/programs/wsproxy/WsProxy.cpp b/programs/wsproxy/WsProxy.cpp index d8d6c18c2b65..89aa6ac7938e 100644 --- a/programs/wsproxy/WsProxy.cpp +++ b/programs/wsproxy/WsProxy.cpp @@ -26,6 +26,7 @@ #include +#include #include #include @@ -79,6 +80,14 @@ std::string envOr(const char * name, const std::string & def) return (value && *value) ? std::string(value) : def; } +/// A boolean env var: true for "1"/"true"/"yes" (case-insensitive), else false. +bool envBool(const char * name) +{ + std::string v = envOr(name, ""); + std::transform(v.begin(), v.end(), v.begin(), ::tolower); + return v == "1" || v == "true" || v == "yes"; +} + BackendParams backendParamsFromEnv() { BackendParams params; @@ -87,6 +96,7 @@ BackendParams backendParamsFromEnv() params.user = envOr("WSPROXY_BACKEND_USER", params.user); params.password = envOr("WSPROXY_BACKEND_PASSWORD", params.password); params.database = envOr("WSPROXY_BACKEND_DATABASE", params.database); + params.secure = envBool("WSPROXY_BACKEND_SECURE"); return params; } @@ -140,6 +150,16 @@ class WsProxyServer : public Poco::Util::ServerApplication const BackendParams backend = backendParamsFromEnv(); + /// For a self-signed / dev backend certificate, configure the SSL client + /// context to accept invalid certificates (mirrors clickhouse-client's + /// --accept-invalid-certificate). Poco's SSLManager builds the default + /// client context lazily from this config on the first secure connect. + if (envBool("WSPROXY_BACKEND_ACCEPT_INVALID_CERT")) + { + config().setString("openSSL.client.invalidCertificateHandler.name", "AcceptCertificateHandler"); + config().setString("openSSL.client.verificationMode", "none"); + } + const UInt16 port = static_cast(std::stoul(envOr("WSPROXY_PORT", "9010"))); Poco::Net::ServerSocket socket(port); Poco::ThreadPool server_pool(/* minCapacity= */ 1, /* maxCapacity= */ 16); From 01fa25e1334815141cb6a1ab40eb97614fa82af6 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 11:39:19 +0200 Subject: [PATCH 24/46] Test proxy->backend TLS; make the suite hermetic tls.test.mjs stands up its own TLS-enabled ClickHouse (spawnBackend({securePort}) generates a self-signed cert and config) and a secure proxy (spawnProxy secure/acceptInvalidCert), verifying a TLS round-trip plus credential pass-through and wrong-password rejection over TLS. globalSetup now uses a high private backend port (19000) instead of 9000, so the suite no longer accidentally reuses an unrelated ClickHouse a developer is running (e.g. an OrbStack container on 9000/9100). 94 tests total, all green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 11 ++-- programs/wsproxy/tests/test/globalSetup.mjs | 4 +- programs/wsproxy/tests/test/proc.mjs | 39 ++++++++++++-- programs/wsproxy/tests/test/tls.test.mjs | 60 +++++++++++++++++++++ 4 files changed, 107 insertions(+), 7 deletions(-) create mode 100644 programs/wsproxy/tests/test/tls.test.mjs diff --git a/TODO.md b/TODO.md index dfca76800302..d3a84c161957 100644 --- a/TODO.md +++ b/TODO.md @@ -346,9 +346,14 @@ happens during the native handshake) is validated at session start — bad creds control frame + a `1008` WebSocket close *before any query is sent* (tested). 92 tests, all green. Auth follow-up — TLS, split by leg and priority: -- **proxy → backend TLS (higher priority):** this leg crosses the network to the (remote/managed) - ClickHouse, so credentials and data must be encrypted here. `Connection` already supports it - (`Protocol::Secure`), currently `Disable`; wire it up + config. +- **proxy → backend TLS — DONE.** `WSPROXY_BACKEND_SECURE=1` connects over the native secure + protocol (`Protocol::Secure::Enable`); `WSPROXY_BACKEND_ACCEPT_INVALID_CERT=1` sets + `openSSL.client.invalidCertificateHandler=AcceptCertificateHandler` + `verificationMode=none` + in the app config for self-signed/dev certs (Poco builds the client SSL context lazily from it). + Verified manually against the OrbStack TLS backend (:9440) and reproducibly in `tls.test.mjs`, + which stands up its own TLS ClickHouse (`spawnBackend({securePort})` generates a self-signed cert + + `` config). Cred pass-through and wrong-password rejection both confirmed over + TLS. Production follow-up: proper CA verification instead of accept-invalid (add a CA/cert config). - **client → proxy TLS (LEAST priority for now):** deprioritized — the proxy is a sidecar next to the app, so this leg is typically loopback / in-pod (or TLS is terminated at the ingress / service mesh). Revisit only if the proxy is exposed beyond the app's trust boundary. diff --git a/programs/wsproxy/tests/test/globalSetup.mjs b/programs/wsproxy/tests/test/globalSetup.mjs index c1359ba6776d..2ed780907f76 100644 --- a/programs/wsproxy/tests/test/globalSetup.mjs +++ b/programs/wsproxy/tests/test/globalSetup.mjs @@ -9,7 +9,9 @@ import net from "node:net"; import { spawnBackend, spawnProxy } from "./proc.mjs"; -const BACKEND_PORT = 9000; +// Use a high, private backend port so the suite is hermetic and does not +// accidentally reuse an unrelated ClickHouse (or other service) on 9000/9100. +const BACKEND_PORT = 19000; const PROXY_PORT = 9010; function portOpen(port, host = "127.0.0.1") { diff --git a/programs/wsproxy/tests/test/proc.mjs b/programs/wsproxy/tests/test/proc.mjs index 8b0a02f06791..4e354db45ab1 100644 --- a/programs/wsproxy/tests/test/proc.mjs +++ b/programs/wsproxy/tests/test/proc.mjs @@ -38,15 +38,38 @@ async function waitForPort(port, timeoutMs = 15_000) { * dir, so a test can kill it (e.g. backend-drops-mid-query). Returns * { tcpPort, stop } where stop() kills the process and removes the temp dir. */ -export async function spawnBackend({ tcpPort }) { +export async function spawnBackend({ tcpPort, securePort }) { const dir = fs.mkdtempSync(join(os.tmpdir(), "wsproxy-ch-")); const configPath = join(dir, "config.xml"); + + // Optional TLS: generate a self-signed cert and enable the secure native port. + let secureBlock = ""; + if (securePort) { + const crt = join(dir, "server.crt"); + const key = join(dir, "server.key"); + const gen = spawnSync("openssl", [ + "req", "-subj", "/CN=localhost", "-new", "-newkey", "rsa:2048", + "-days", "3650", "-nodes", "-x509", "-keyout", key, "-out", crt, + ]); + if (gen.status !== 0) throw new Error(`openssl cert generation failed: ${gen.stderr}`); + secureBlock = ` + ${securePort} + + ${crt} + ${key} + none + true + sslv2,sslv3 + true + `; + } + fs.writeFileSync( configPath, ` none0 ${dir}/server.log${dir}/server.err.log - ${tcpPort} + ${tcpPort}${secureBlock} 127.0.0.1 ${dir}/data/ ${dir}/tmp/ @@ -83,8 +106,10 @@ export async function spawnBackend({ tcpPort }) { // reliably hits the actual server process (and the watchdog). const proc = spawn(CLICKHOUSE_SERVER, ["--config-file", configPath], { stdio: "ignore" }); await waitForPort(tcpPort, 30_000); + if (securePort) await waitForPort(securePort, 30_000); return { tcpPort, + securePort, stop() { // Kill whatever holds the TCP port — reliably the real server process, // regardless of the watchdog fork/rename/re-parent shenanigans. @@ -115,7 +140,13 @@ export async function spawnBackend({ tcpPort }) { * Spawn a proxy on `listenPort` pointing at `backendHost:backendPort`. * Returns { url, stop } where stop() kills the process. */ -export async function spawnProxy({ listenPort, backendHost = "127.0.0.1", backendPort }) { +export async function spawnProxy({ + listenPort, + backendHost = "127.0.0.1", + backendPort, + secure = false, + acceptInvalidCert = false, +}) { const proc = spawn(WSPROXY_BIN, [], { stdio: "ignore", env: { @@ -123,6 +154,8 @@ export async function spawnProxy({ listenPort, backendHost = "127.0.0.1", backen WSPROXY_PORT: String(listenPort), WSPROXY_BACKEND_HOST: backendHost, WSPROXY_BACKEND_PORT: String(backendPort), + ...(secure ? { WSPROXY_BACKEND_SECURE: "1" } : {}), + ...(acceptInvalidCert ? { WSPROXY_BACKEND_ACCEPT_INVALID_CERT: "1" } : {}), }, }); await waitForPort(listenPort); diff --git a/programs/wsproxy/tests/test/tls.test.mjs b/programs/wsproxy/tests/test/tls.test.mjs new file mode 100644 index 000000000000..eeff89f4538b --- /dev/null +++ b/programs/wsproxy/tests/test/tls.test.mjs @@ -0,0 +1,60 @@ +import { describe, it, expect } from "vitest"; +import { runQuery, Session } from "./helpers.mjs"; +import { spawnBackend, spawnProxy } from "./proc.mjs"; + +// Stands up its own TLS-enabled ClickHouse (self-signed cert) plus a proxy +// configured to connect to it over the secure native protocol. +describe("proxy -> backend TLS", () => { + it("round-trips a query over a TLS backend connection", async () => { + const backend = await spawnBackend({ tcpPort: 9003, securePort: 9444 }); + const proxy = await spawnProxy({ + listenPort: 9015, + backendPort: 9444, + secure: true, + acceptInvalidCert: true, // self-signed cert + }); + try { + const { text, control } = await runQuery("SELECT 'tls' AS s, currentUser() AS u", { + baseUrl: proxy.url, + }); + expect(control.event).toBe("end"); + const row = JSON.parse(text.trim()); + expect(row.s).toBe("tls"); + expect(row.u).toBe("default"); + } finally { + proxy.stop(); + backend.stop(); + } + }, 60000); + + it("passes credentials through over TLS", async () => { + const backend = await spawnBackend({ tcpPort: 9004, securePort: 9445 }); + const proxy = await spawnProxy({ + listenPort: 9016, + backendPort: 9445, + secure: true, + acceptInvalidCert: true, + }); + try { + // Correct credentials over TLS. + const ok = await runQuery("SELECT currentUser() AS u", { + baseUrl: proxy.url, + user: "wsp_user", + password: "wsp_pass", + }); + expect(ok.control.event).toBe("end"); + expect(JSON.parse(ok.text.trim()).u).toBe("wsp_user"); + + // Wrong password is still rejected over TLS. + const bad = await runQuery("SELECT 1", { + baseUrl: proxy.url, + user: "wsp_user", + password: "nope", + }); + expect(bad.control.event).toBe("error"); + } finally { + proxy.stop(); + backend.stop(); + } + }, 60000); +}); From 6b5d33eda3e648a0be6de0d4cca8a4ac172ce54b Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 14:33:11 +0200 Subject: [PATCH 25/46] Add a client send timeout so a stalled reader cannot pin a thread Backpressure is already correct and bounded (blocking write -> session stops reading the backend -> server throttles via TCP; per-session memory is one block + the 1MB buffer + kernel socket buffers). The gap was liveness: the handler set only a receive timeout, so a client that stopped reading entirely blocked sendBytes forever and pinned a handler thread (pool of 16). Add setSendTimeout (default 30s, env WSPROXY_CLIENT_SEND_TIMEOUT_SEC); a zero-progress write then throws and routes through WriteBufferToWebSocket (broken -> sendCancel) to a clean teardown. It is per-write, so a slow-but-progressing client is unaffected. Tested in backpressure.test.mjs: a client that pauses reading a large result is dropped ~1s later and the proxy stays healthy for the next client (RawClient gains pause/resume; spawnProxy gains a sendTimeoutSec option). 95 tests total, all green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 23 +++++++++ programs/wsproxy/WsProxyHandler.cpp | 13 +++++ .../wsproxy/tests/test/backpressure.test.mjs | 47 +++++++++++++++++++ programs/wsproxy/tests/test/proc.mjs | 2 + programs/wsproxy/tests/test/raw.mjs | 10 ++++ 5 files changed, 95 insertions(+) create mode 100644 programs/wsproxy/tests/test/backpressure.test.mjs diff --git a/TODO.md b/TODO.md index d3a84c161957..8efe4e009f6b 100644 --- a/TODO.md +++ b/TODO.md @@ -358,6 +358,29 @@ Auth follow-up — TLS, split by leg and priority: the app, so this leg is typically loopback / in-pod (or TLS is terminated at the ingress / service mesh). Revisit only if the proxy is exposed beyond the app's trust boundary. +## Backpressure & flow control + +Analysis: backpressure is already correct and bounded by the blocking-socket + demand-driven +model. A slow WS client → blocking `sendBytes` → the session loop stops calling `receivePacket` +→ the backend's TCP window closes → the server throttles its own production. Per-session memory is +bounded (one `Block` + the 1 MB `WriteBufferToWebSocket` + the two kernel socket buffers); no queue +accumulates. No application-level flow-control signal is needed — TCP is the signal. The kernel +socket buffers (`SO_RCVBUF` backend leg / `SO_SNDBUF` client leg) act as the pipeline buffer that +lets the backend keep producing while the proxy writes, up to a bound — so moderate speed mismatch +adds no artificial delay; only a persistently slow client engages throttling. + +**#1 DONE — send timeout.** Previously the handler set only `setReceiveTimeout`; a client that +stopped reading entirely (window stuck at 0) blocked `sendBytes` forever, pinning a handler thread +(pool is 16 → a few stalled clients = exhaustion). Now `setSendTimeout` (default 30s, env +`WSPROXY_CLIENT_SEND_TIMEOUT_SEC`) bounds a zero-progress write; the throw routes through +`WriteBufferToWebSocket` (`broken` → `sendCancel`) to a clean teardown. It's per-write, so a +slow-but-progressing client is unaffected. Tested in `backpressure.test.mjs` (a client that pauses +reading is dropped ~1s and the proxy stays healthy for the next client). + +Deferred: #2 writer thread + bounded queue for full read/write parallelism (kernel buffers already +give most of it — revisit only if a throughput profile justifies it); #3 WS `permessage-deflate` +for bandwidth; expose `SO_SNDBUF`/`SO_RCVBUF` + timeouts via config file. + ## Plan 1. **Skeleton.** Standalone `programs/wsproxy/` binary. **DONE — builds, links, runs-to-listen.** diff --git a/programs/wsproxy/WsProxyHandler.cpp b/programs/wsproxy/WsProxyHandler.cpp index fa3db0867409..3d1a3cf337b5 100644 --- a/programs/wsproxy/WsProxyHandler.cpp +++ b/programs/wsproxy/WsProxyHandler.cpp @@ -19,6 +19,7 @@ #include #include +#include #include @@ -220,6 +221,18 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp /// handler thread indefinitely between queries. socket.setReceiveTimeout(Poco::Timespan(300, 0)); + /// Send timeout: drop a client that stops reading entirely (its TCP window + /// stuck at 0) so a blocking write cannot pin a handler thread forever. The + /// resulting throw routes through WriteBufferToWebSocket (broken -> sendCancel) + /// to a clean teardown. Applies per blocking write, so a slow-but-progressing + /// client is unaffected; only genuine zero-progress stalls trip it. + /// Configurable (mainly for tests); default 30s. + Int64 send_timeout_sec = 30; + if (const char * v = std::getenv("WSPROXY_CLIENT_SEND_TIMEOUT_SEC"); v && *v) + send_timeout_sec = std::strtoll(v, nullptr, 10); + if (send_timeout_sec > 0) + socket.setSendTimeout(Poco::Timespan(send_timeout_sec * 1'000'000)); /// microseconds + ProxySession session(socket, context, session_backend, out_format, logs_level); session.run(); diff --git a/programs/wsproxy/tests/test/backpressure.test.mjs b/programs/wsproxy/tests/test/backpressure.test.mjs new file mode 100644 index 000000000000..fc9fc63e635b --- /dev/null +++ b/programs/wsproxy/tests/test/backpressure.test.mjs @@ -0,0 +1,47 @@ +import { describe, it, expect } from "vitest"; +import { runQuery } from "./helpers.mjs"; +import { RawClient } from "./raw.mjs"; +import { spawnProxy } from "./proc.mjs"; + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + +describe("backpressure / stalled client", () => { + it("drops a client that stops reading (send timeout) and stays healthy", async () => { + // Short send timeout so the test doesn't wait the 30s default. + const proxy = await spawnProxy({ listenPort: 9017, backendPort: 19000, sendTimeoutSec: 1 }); + try { + const c = new RawClient({ port: 9017 }); + await c.handshake("/?format=JSONEachRow"); + // A large result the client will refuse to read, so the proxy's blocking + // writes fill the socket buffers and then stall. + c.sendFrame({ + opcode: 0x1, + payload: "SELECT number, repeat('x', 1000) AS s FROM numbers(5000000)", + }); + c.pause(); // stop reading -> proxy writes block -> ~1s send timeout should fire + + // Wait past the send timeout, then resume and confirm the proxy dropped us + // (rather than blocking a handler thread forever). + await sleep(4000); + c.resume(); + const start = Date.now(); + let closed = false; + while (Date.now() - start < 10000) { + const f = await c.readFrame(); + if (f.closed) { + closed = true; + break; + } + } + c.close(); + expect(closed).toBe(true); + + // The proxy is still healthy for a fresh client (the stalled thread was released). + const ok = await runQuery("SELECT 1 AS n", { baseUrl: proxy.url }); + expect(ok.control.event).toBe("end"); + expect(JSON.parse(ok.text.trim()).n).toBe(1); + } finally { + proxy.stop(); + } + }, 30000); +}); diff --git a/programs/wsproxy/tests/test/proc.mjs b/programs/wsproxy/tests/test/proc.mjs index 4e354db45ab1..d7fb6cfd0dab 100644 --- a/programs/wsproxy/tests/test/proc.mjs +++ b/programs/wsproxy/tests/test/proc.mjs @@ -146,6 +146,7 @@ export async function spawnProxy({ backendPort, secure = false, acceptInvalidCert = false, + sendTimeoutSec, }) { const proc = spawn(WSPROXY_BIN, [], { stdio: "ignore", @@ -156,6 +157,7 @@ export async function spawnProxy({ WSPROXY_BACKEND_PORT: String(backendPort), ...(secure ? { WSPROXY_BACKEND_SECURE: "1" } : {}), ...(acceptInvalidCert ? { WSPROXY_BACKEND_ACCEPT_INVALID_CERT: "1" } : {}), + ...(sendTimeoutSec ? { WSPROXY_CLIENT_SEND_TIMEOUT_SEC: String(sendTimeoutSec) } : {}), }, }); await waitForPort(listenPort); diff --git a/programs/wsproxy/tests/test/raw.mjs b/programs/wsproxy/tests/test/raw.mjs index 61fe73a12e3d..f7cb6221233e 100644 --- a/programs/wsproxy/tests/test/raw.mjs +++ b/programs/wsproxy/tests/test/raw.mjs @@ -143,6 +143,16 @@ export class RawClient { return this.closed; } + /** Stop reading from the socket (simulates a client that can't keep up). */ + pause() { + this.socket.pause(); + } + + /** Resume reading after a pause(). */ + resume() { + this.socket.resume(); + } + close() { try { this.socket.destroy(); From 14afeef6466697d3ca765c042b85d5ea86fc395b Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 17:18:06 +0200 Subject: [PATCH 26/46] Document JS client receive-backpressure guidance The proxy's TCP backpressure does not protect event-based WebSocket clients: browser/Node WebSocket eagerly drains the socket into the JS heap with no receive-backpressure API, so streaming a large result to a slow consumer can OOM the client. Decision: keep the proxy push-only (no opt-in windowing for now) and address it client-side. Documented guidance in the tests README: prefer WebSocketStream (real transport backpressure), pause the underlying ws socket in Node, or keep results bounded for plain browser WebSocket. Recorded the analysis and the (unimplemented) opt-in credit/window design in TODO.md. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 21 ++++++++++++++++ programs/wsproxy/tests/README.md | 41 ++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) diff --git a/TODO.md b/TODO.md index 8efe4e009f6b..92160cef9ef9 100644 --- a/TODO.md +++ b/TODO.md @@ -381,6 +381,27 @@ Deferred: #2 writer thread + bounded queue for full read/write parallelism (kern give most of it — revisit only if a throughput profile justifies it); #3 WS `permessage-deflate` for bandwidth; expose `SO_SNDBUF`/`SO_RCVBUF` + timeouts via config file. +### JS client receive backpressure — decision: client-side only (no proxy change) + +The proxy's TCP backpressure is necessary but NOT sufficient for JS clients: browser/Node +event-based `WebSocket` eagerly drains the socket into the JS heap and dispatches `onmessage` with +no receive-backpressure API, so the client's TCP window stays open, the proxy never blocks, and a +large result to a slow consumer grows the JS heap unbounded → OOM. The proxy can't detect this. + +Decision (per Peter): **keep push-only (opt-in windowing NOT added for now); handle it client-side +and document it.** Rationale: keeps the proxy simple; capable clients can get real backpressure +themselves. Client guidance (see `programs/wsproxy/tests/README.md`): +- **Best:** `WebSocketStream` (Chromium / Node experimental) — its `ReadableStream` applies real + backpressure that closes the TCP window and throttles the proxy→backend. +- **Node (`ws`):** pause the underlying socket (`ws._socket.pause()`/`resume()`) by consumption. +- **Plain browser `WebSocket` (Firefox/Safari):** no receive backpressure — keep results bounded + (LIMIT / server-side aggregation) or consume synchronously; don't stream unbounded results to a + slow consumer. + +If this becomes a real limitation, revisit **opt-in credit/byte-window flow control** (proxy pauses +when `sent − acked ≥ window`; client sends `{"ack":N}`), which is the only portable, streaming- +preserving fix. Design is sketched; not implemented. + ## Plan 1. **Skeleton.** Standalone `programs/wsproxy/` binary. **DONE — builds, links, runs-to-listen.** diff --git a/programs/wsproxy/tests/README.md b/programs/wsproxy/tests/README.md index 486189940edc..5078759ab687 100644 --- a/programs/wsproxy/tests/README.md +++ b/programs/wsproxy/tests/README.md @@ -41,3 +41,44 @@ frames. For INSERT, the client sends `INSERT INTO t [FORMAT X]` then streams dat frames ending with a zero-length binary frame; closing the socket mid-query cancels it. See `test/helpers.mjs` for the small client wrapper (`Session`, `runQuery`, `backendScalar`). + +## Receive backpressure (important for large results) + +The proxy pushes result frames as fast as the connection allows. A plain **event-based +`WebSocket`** (browser or Node) has **no receive-backpressure API** — it eagerly drains the socket +and fires `onmessage` regardless of whether your app has kept up. So streaming a **large** result +to a **slow consumer** accumulates in the JS heap and can OOM the client. The proxy cannot prevent +this (from its side the socket looks healthy). If you stream large results, apply backpressure on +the client: + +- **Best — `WebSocketStream`** (Chromium; Node with `--experimental-websocket-stream`). Its + `ReadableStream` applies real backpressure: when your reader is slow it stops reading the socket, + the TCP window closes, and the proxy throttles the backend. No app protocol needed. + + ```js + const wss = new WebSocketStream("ws://localhost:9010/?format=JSONEachRow"); + const { readable, writable } = await wss.opened; + await writable.getWriter().write("SELECT ... FROM big_table"); + const reader = readable.getReader(); + for (;;) { + const { value, done } = await reader.read(); // slow processing here throttles the proxy + if (done) break; + await process(value); // string = control/progress; Uint8Array = result bytes + } + ``` + +- **Node with the `ws` library** — pause the underlying socket by consumption: + + ```js + ws.on("message", (data, isBinary) => { + ws._socket.pause(); // stop reading -> TCP backpressure to the proxy + process(data).finally(() => ws._socket.resume()); + }); + ``` + +- **Plain browser `WebSocket`** (Firefox/Safari, or not using `WebSocketStream`) — you cannot apply + receive backpressure. Keep results bounded (`LIMIT`, server-side aggregation), or consume + synchronously in `onmessage` without queuing. Don't stream unbounded results to a slow consumer. + +(If this becomes a common need, the proxy could add opt-in credit/byte-window flow control — +`sent − acked ≥ window` pauses the stream — which works for any client. Not implemented yet.) From 1575bee932522b4c9ae8c4ac2614cab1eff7caba Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 19:41:30 +0200 Subject: [PATCH 27/46] Add opt-in credit/window flow control for the SELECT push direction ?flow=N enables frame-based flow control with N frames of initial credit; without it, push mode is unchanged (unbounded). The client grants credit with {"cmd":"next","n":N} and can {"cmd":"pause"}/{"cmd":"resume"}; the proxy sends a binary data frame only while not paused and credit>0. The gate lives in WriteBufferToWebSocket::nextImpl, which blocks reading the client's control frames when starved -> it also stops reading the backend, so backpressure propagates to the server via TCP, paced by client consumption. A frame is one WS binary message (one block flush), so credit needs no byte repacking. Only binary data frames are gated; progress/log/profile_events/end/error text frames flow freely. One applyControlFrame parses control frames for both the between-packets poll and the gate. Bounded by the existing receive timeout (no new hang). Lets event-based JS clients (which cannot apply receive backpressure) bound their buffering; INSERT direction is client-handled. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/ProxySession.cpp | 112 ++++++++++++++++++++++++++-- programs/wsproxy/ProxySession.h | 6 +- programs/wsproxy/WsProxyHandler.cpp | 8 +- 3 files changed, 116 insertions(+), 10 deletions(-) diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp index 66f30e47ffbf..06d1abb65b96 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/programs/wsproxy/ProxySession.cpp @@ -32,6 +32,9 @@ #include #include +#include +#include + namespace DB { @@ -46,6 +49,56 @@ constexpr size_t MAX_QUERY_SIZE = 16 << 20; constexpr size_t MAX_PARSER_DEPTH = 1000; constexpr size_t MAX_PARSER_BACKTRACKS = 1'000'000; +/// Opt-in credit/window flow control for the SELECT push direction. The client +/// grants credit in whole frames (`{"cmd":"next","n":N}`) and can `pause`/`resume`; +/// the proxy sends a data frame only while `!paused && credit > 0`. This lets an +/// event-based JS client (which cannot apply receive backpressure) bound how much +/// it must buffer. `client_gone` latches a Close/broken read. +struct FlowControl +{ + bool enabled = false; + Int64 credit = 0; + bool paused = false; + bool client_gone = false; +}; + +/// Apply one client control frame to the flow state (also answers pings). Used +/// both by the between-packets poll and by the credit gate, so `next`/`pause`/ +/// `resume`/close are handled consistently wherever the client frame is read. +void applyControlFrame(Poco::Net::StreamSocket & socket, const WebSocketFrame & frame, FlowControl & fc) +{ + if (!frame.valid || frame.opcode == Opcode::Close) + { + fc.client_gone = true; + return; + } + if (frame.opcode == Opcode::Ping) + { + sendWebSocketFrame(socket, Opcode::Pong, frame.payload.data(), frame.payload.size()); + return; + } + if (frame.opcode != Opcode::Text) + return; /// Ignore stray binary during a SELECT. + + try + { + Poco::JSON::Parser parser; + const auto obj = parser.parse(frame.payload).extract(); + const String cmd = obj->optValue("cmd", ""); + if (cmd == "next") + fc.credit += obj->optValue("n", 0); + else if (cmd == "pause") + fc.paused = true; + else if (cmd == "resume") + fc.paused = false; + } + catch (...) + { + /// Malformed control frame: ignore (the query stream is unaffected). + LOG_DEBUG(getLogger("ProxySession"), "Ignoring malformed control frame: {}", getCurrentExceptionMessage(false)); + } +} + /// A `WriteBuffer` whose flushes are emitted as WebSocket binary frames. The /// output format writes into it directly, so calling `flush` after each result /// block streams that block to the client as its own frame (mid-query push). @@ -57,8 +110,9 @@ constexpr size_t MAX_PARSER_BACKTRACKS = 1'000'000; class WriteBufferToWebSocket : public BufferWithOwnMemory { public: - explicit WriteBufferToWebSocket(Poco::Net::StreamSocket & socket_, size_t size = DBMS_DEFAULT_BUFFER_SIZE) - : BufferWithOwnMemory(size), socket(socket_) + explicit WriteBufferToWebSocket( + Poco::Net::StreamSocket & socket_, FlowControl * flow_ = nullptr, size_t size = DBMS_DEFAULT_BUFFER_SIZE) + : BufferWithOwnMemory(size), socket(socket_), flow(flow_) { } @@ -71,6 +125,34 @@ class WriteBufferToWebSocket : public BufferWithOwnMemory { if (broken || offset() == 0) return; + + /// Flow control: block this data frame until the client has granted credit + /// (and is not paused). While blocked we read the client's control frames + /// (next/pause/resume, or a Close), which also stops us reading the backend + /// -> TCP backpressure to the server, paced by the client's consumption. + if (flow && flow->enabled) + { + while (!flow->client_gone && (flow->paused || flow->credit <= 0)) + { + WebSocketFrame frame; + try + { + frame = readWebSocketFrame(socket); + } + catch (...) + { + flow->client_gone = true; + break; + } + applyControlFrame(socket, frame, *flow); + } + if (flow->client_gone) + { + broken = true; + return; + } + } + try { sendWebSocketBinary(socket, working_buffer.begin(), offset()); @@ -78,10 +160,15 @@ class WriteBufferToWebSocket : public BufferWithOwnMemory catch (...) { broken = true; + return; } + + if (flow && flow->enabled) + --flow->credit; } Poco::Net::StreamSocket & socket; + FlowControl * flow; bool broken = false; }; @@ -195,12 +282,16 @@ ProxySession::ProxySession( ContextPtr context_, BackendParams backend_, String format_, - String logs_level_) + String logs_level_, + bool flow_enabled_, + Int64 flow_initial_credit_) : socket(socket_) , context(std::move(context_)) , backend(std::move(backend_)) , format(std::move(format_)) , logs_level(std::move(logs_level_)) + , flow_enabled(flow_enabled_) + , flow_initial_credit(flow_initial_credit_) { } @@ -379,7 +470,11 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) sendBackendQuery(connection, query); - WriteBufferToWebSocket out_buf(socket); + FlowControl fc; + fc.enabled = flow_enabled; + fc.credit = flow_initial_credit; + + WriteBufferToWebSocket out_buf(socket, &fc); OutputFormatPtr output; bool cancelled = false; bool client_gone = false; @@ -433,10 +528,11 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) frame.valid = false; } - /// A Close frame (or broken read) means the client is gone: - /// cancel the backend query and drain until EndOfStream. Other - /// in-query frames are ignored for now. - if (!frame.valid || frame.opcode == Opcode::Close) + /// Apply the client frame: flow-control grants (next/pause/resume) + /// update the shared FlowControl; a Close/broken read means the + /// client is gone -> cancel the backend query and drain. + applyControlFrame(socket, frame, fc); + if (fc.client_gone) note_client_gone(); } } diff --git a/programs/wsproxy/ProxySession.h b/programs/wsproxy/ProxySession.h index 206201eead54..dc3dde35959e 100644 --- a/programs/wsproxy/ProxySession.h +++ b/programs/wsproxy/ProxySession.h @@ -40,7 +40,9 @@ class ProxySession ContextPtr context_, BackendParams backend_, String format_, - String logs_level_ = ""); + String logs_level_ = "", + bool flow_enabled_ = false, + Int64 flow_initial_credit_ = 0); void run(); @@ -68,6 +70,8 @@ class ProxySession BackendParams backend; String format; String logs_level; /// If set, sent as `send_logs_level` so the backend pushes Log packets. + bool flow_enabled; /// Opt-in credit/window flow control for the SELECT push direction. + Int64 flow_initial_credit; /// Starting frame credit when flow control is enabled. }; } diff --git a/programs/wsproxy/WsProxyHandler.cpp b/programs/wsproxy/WsProxyHandler.cpp index 3d1a3cf337b5..4f3693d1367f 100644 --- a/programs/wsproxy/WsProxyHandler.cpp +++ b/programs/wsproxy/WsProxyHandler.cpp @@ -210,6 +210,12 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp /// Optional: `?logs=` (e.g. information, trace) makes the backend push /// server-side log lines for the session's queries. const String logs_level = queryParam(uri, "logs", "", log); + /// Opt-in credit/window flow control: `?flow=N` enables it with N frames of + /// initial credit; absent = push mode (unbounded). The client then grants more + /// with {"cmd":"next","n":...} and can pause/resume. + const String flow_param = queryParam(uri, "flow", "", log); + const bool flow_enabled = !flow_param.empty(); + const Int64 flow_credit = flow_enabled ? std::strtoll(flow_param.c_str(), nullptr, 10) : 0; /// Credential pass-through: resolve this session's backend user/password from /// the request (never mutate the shared default `backend`). @@ -233,7 +239,7 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp if (send_timeout_sec > 0) socket.setSendTimeout(Poco::Timespan(send_timeout_sec * 1'000'000)); /// microseconds - ProxySession session(socket, context, session_backend, out_format, logs_level); + ProxySession session(socket, context, session_backend, out_format, logs_level, flow_enabled, flow_credit); session.run(); LOG_DEBUG(log, "WebSocket session closed"); From cbc1712a344ede69240962e5c06a4709cbac8636 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 19:41:30 +0200 Subject: [PATCH 28/46] Client flow-control helpers, test, and docs Session gains next(n)/pause()/resume() and a flow option (?flow=N); nextFrame gains an optional timeout (no frame lost on timeout). flow.test.mjs verifies the credit window (exactly N frames then a stall, resumed by next), pause/resume, and that push mode without ?flow stays unbounded. README documents ?flow=N usage alongside the WebSocketStream / ws-socket-pause alternatives. 98 tests total, all green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 29 ++++---- programs/wsproxy/tests/README.md | 25 ++++++- programs/wsproxy/tests/test/flow.test.mjs | 80 +++++++++++++++++++++++ programs/wsproxy/tests/test/helpers.mjs | 57 ++++++++++++++-- 4 files changed, 169 insertions(+), 22 deletions(-) create mode 100644 programs/wsproxy/tests/test/flow.test.mjs diff --git a/TODO.md b/TODO.md index 92160cef9ef9..ece81ebcd82e 100644 --- a/TODO.md +++ b/TODO.md @@ -388,19 +388,22 @@ event-based `WebSocket` eagerly drains the socket into the JS heap and dispatche no receive-backpressure API, so the client's TCP window stays open, the proxy never blocks, and a large result to a slow consumer grows the JS heap unbounded → OOM. The proxy can't detect this. -Decision (per Peter): **keep push-only (opt-in windowing NOT added for now); handle it client-side -and document it.** Rationale: keeps the proxy simple; capable clients can get real backpressure -themselves. Client guidance (see `programs/wsproxy/tests/README.md`): -- **Best:** `WebSocketStream` (Chromium / Node experimental) — its `ReadableStream` applies real - backpressure that closes the TCP window and throttles the proxy→backend. -- **Node (`ws`):** pause the underlying socket (`ws._socket.pause()`/`resume()`) by consumption. -- **Plain browser `WebSocket` (Firefox/Safari):** no receive backpressure — keep results bounded - (LIMIT / server-side aggregation) or consume synchronously; don't stream unbounded results to a - slow consumer. - -If this becomes a real limitation, revisit **opt-in credit/byte-window flow control** (proxy pauses -when `sent − acked ≥ window`; client sends `{"ack":N}`), which is the only portable, streaming- -preserving fix. Design is sketched; not implemented. +**DONE — opt-in credit/window flow control** (frame-based). `?flow=N` enables it with N frames of +initial credit; without `?flow`, push mode is unchanged (unbounded, zero overhead). Frame = one WS +binary data frame (= one block flush), so no byte repacking. The client controls the stream with: +- `{"cmd":"next","n":N}` — grant N more frames of credit; +- `{"cmd":"pause"}` / `{"cmd":"resume"}` — hard stop / resume. +The proxy sends a data frame only while `!paused && credit > 0`; the gate lives in +`WriteBufferToWebSocket::nextImpl`, which blocks reading the client's control frames when starved — +that also stops it reading the backend, so backpressure propagates to the server (TCP), paced by the +client's consumption. Gate applies to binary data frames only; `progress`/`log`/`profile_events`/ +`end`/`error` text frames flow freely. Control frames are parsed by one `applyControlFrame` used by +both the between-packets poll and the gate. Bounded by the existing receive timeout (no new hang). +INSERT direction is client-handled (per Peter). Tested in `flow.test.mjs` (credit window, pause/ +resume, and unchanged push mode); 98 tests green. + +Client-side alternatives still valid where preferred (documented in the tests README): +`WebSocketStream` (transport backpressure), or pausing the underlying `ws` socket in Node. ## Plan diff --git a/programs/wsproxy/tests/README.md b/programs/wsproxy/tests/README.md index 5078759ab687..46b8e6861d8c 100644 --- a/programs/wsproxy/tests/README.md +++ b/programs/wsproxy/tests/README.md @@ -48,8 +48,29 @@ The proxy pushes result frames as fast as the connection allows. A plain **event `WebSocket`** (browser or Node) has **no receive-backpressure API** — it eagerly drains the socket and fires `onmessage` regardless of whether your app has kept up. So streaming a **large** result to a **slow consumer** accumulates in the JS heap and can OOM the client. The proxy cannot prevent -this (from its side the socket looks healthy). If you stream large results, apply backpressure on -the client: +this (from its side the socket looks healthy). If you stream large results, apply backpressure. The proxy offers a built-in, portable option, +plus client-side alternatives: + +- **Built-in — opt-in credit flow control (`?flow=N`).** Connect with `?flow=N` to start with N + frames of credit; the proxy then sends a frame only while you have credit and aren't paused, and + blocks (throttling the backend) otherwise. Grant more as you consume, or pause/resume: + + ```js + const s = new Session("JSONEachRow", { flow: 8 }); // start with 8 frames of credit + await s.ready(); + s.sendQuery("SELECT ... FROM big_table"); + for (;;) { + const f = await s.nextFrame(); + if (f.type === "binary") { await process(f.data); s.next(1); } // 1 credit per consumed frame + else if (f.type === "text") { const e = JSON.parse(f.data); if (["end","error","cancelled"].includes(e.event)) break; } + } + // s.pause() / s.resume() give a coarse hard stop as well. + ``` + + A frame is one WS binary message (one result block), so credit is in frames — no byte accounting. + Works in any JS runtime (browser + Node). INSERT direction is unaffected (client-driven). + +The client-side transport options also work if you prefer them: - **Best — `WebSocketStream`** (Chromium; Node with `--experimental-websocket-stream`). Its `ReadableStream` applies real backpressure: when your reader is slow it stops reading the socket, diff --git a/programs/wsproxy/tests/test/flow.test.mjs b/programs/wsproxy/tests/test/flow.test.mjs new file mode 100644 index 000000000000..4ecc643c03c4 --- /dev/null +++ b/programs/wsproxy/tests/test/flow.test.mjs @@ -0,0 +1,80 @@ +import { describe, it, expect } from "vitest"; +import { Session } from "./helpers.mjs"; + +// Collect frames until either a terminal control event or a quiet gap (no frame +// within `gapMs`, meaning the server is gated waiting for credit). Returns +// { binary, ended, cancelled }. +async function drainUntilGap(s, gapMs) { + let binary = 0; + let ended = false; + for (;;) { + const f = await s.nextFrame(gapMs); + if (f === null) break; // gated: nothing arrived within the gap + if (f.type === "binary") binary++; + else if (f.type === "text") { + const ev = JSON.parse(f.data).event; + if (ev === "end" || ev === "error" || ev === "cancelled") { + ended = true; + break; + } + // progress / log / profile_events are not gated — ignore + } else if (f.type === "close") { + break; + } + } + return { binary, ended }; +} + +describe("flow control (credit/window)", () => { + it("sends only up to the granted credit, then resumes on next()", async () => { + // One row per block => one binary frame per row; initial credit = 3. + const s = new Session("JSONEachRow", { flow: 3 }); + await s.ready(); + s.sendQuery("SELECT number FROM numbers(20) SETTINGS max_block_size = 1"); + + // Initial window: exactly 3 data frames, then the stream should stall. + const phase1 = await drainUntilGap(s, 600); + expect(phase1.ended).toBe(false); + expect(phase1.binary).toBe(3); + + // Grant enough credit to finish; all remaining rows arrive and we reach end. + s.next(100); + const phase2 = await drainUntilGap(s, 3000); + s.close(); + expect(phase2.ended).toBe(true); + expect(phase1.binary + phase2.binary).toBe(20); + }, 20000); + + it("pause() halts the stream and resume() continues it", async () => { + const s = new Session("JSONEachRow", { flow: 2 }); + await s.ready(); + s.sendQuery("SELECT number FROM numbers(10) SETTINGS max_block_size = 1"); + + // Consume the initial 2, then pause before granting more. + const p1 = await drainUntilGap(s, 600); + expect(p1.binary).toBe(2); + s.pause(); + s.next(100); // credit is available, but paused must keep the stream halted + + const p2 = await drainUntilGap(s, 600); + expect(p2.binary).toBe(0); // paused: no frames despite credit + expect(p2.ended).toBe(false); + + // Resume: the remaining rows flow (credit already granted). + s.resume(); + const p3 = await drainUntilGap(s, 3000); + s.close(); + expect(p3.ended).toBe(true); + expect(p1.binary + p2.binary + p3.binary).toBe(10); + }, 20000); + + it("without ?flow, push mode is unbounded (no credit needed)", async () => { + const s = new Session("JSONEachRow"); // no flow param + await s.ready(); + s.sendQuery("SELECT number FROM numbers(50) SETTINGS max_block_size = 1"); + const r = await drainUntilGap(s, 3000); + s.close(); + expect(r.ended).toBe(true); + expect(r.binary).toBe(50); // all delivered without any next() + }, 20000); +}); diff --git a/programs/wsproxy/tests/test/helpers.mjs b/programs/wsproxy/tests/test/helpers.mjs index 652c63ecc38c..673cd4a014ef 100644 --- a/programs/wsproxy/tests/test/helpers.mjs +++ b/programs/wsproxy/tests/test/helpers.mjs @@ -15,15 +15,16 @@ export const PROXY_URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; -/** Build the WS URL for a format and options ({ logs, path, baseUrl, user, password }). */ +/** Build the WS URL for a format and options ({ logs, path, baseUrl, user, password, flow }). */ export function urlFor( format = "JSONEachRow", - { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "" } = {}, + { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow } = {}, ) { const params = new URLSearchParams({ format }); if (logs) params.set("logs", logs); if (user) params.set("user", user); if (password) params.set("password", password); + if (flow !== undefined) params.set("flow", String(flow)); // opt-in credit flow control return `${baseUrl}${path}?${params}`; } @@ -33,8 +34,11 @@ export function urlFor( * `await` them regardless of arrival timing. */ export class Session { - constructor(format = "JSONEachRow", { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "" } = {}) { - this.ws = new WebSocket(urlFor(format, { logs, path, baseUrl, user, password })); + constructor( + format = "JSONEachRow", + { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow } = {}, + ) { + this.ws = new WebSocket(urlFor(format, { logs, path, baseUrl, user, password, flow })); this.ws.binaryType = "arraybuffer"; this._queue = []; this._waiters = []; @@ -68,10 +72,49 @@ export class Session { return this.opened; } - /** Next incoming frame: {type:'text',data:string} | {type:'binary',data:Buffer} | {type:'close'}. */ - nextFrame() { + /** + * Next incoming frame: {type:'text',data:string} | {type:'binary',data:Buffer} | {type:'close'}. + * With `timeoutMs > 0`, resolves to null if no frame arrives in time (without + * losing a frame that arrives later — the waiter is removed on timeout). + */ + nextFrame(timeoutMs = 0) { if (this._queue.length) return Promise.resolve(this._queue.shift()); - return new Promise((resolve) => this._waiters.push(resolve)); + return new Promise((resolve) => { + let settled = false; + const waiter = (frame) => { + if (settled) { + this._queue.unshift(frame); // raced with the timeout; don't drop it + return; + } + settled = true; + resolve(frame); + }; + this._waiters.push(waiter); + if (timeoutMs > 0) { + setTimeout(() => { + if (settled) return; + settled = true; + const i = this._waiters.indexOf(waiter); + if (i >= 0) this._waiters.splice(i, 1); + resolve(null); + }, timeoutMs); + } + }); + } + + /** Flow-control: grant N more frames of credit. */ + next(n) { + this.ws.send(JSON.stringify({ cmd: "next", n })); + } + + /** Flow-control: pause the server push. */ + pause() { + this.ws.send(JSON.stringify({ cmd: "pause" })); + } + + /** Flow-control: resume after pause(). */ + resume() { + this.ws.send(JSON.stringify({ cmd: "resume" })); } /** Send a SQL query (TEXT frame). */ From e4f2151a840327f377b348827dc91419287fd63d Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 20:47:39 +0200 Subject: [PATCH 29/46] Document why control-during-pause is rejected (ordering) and pause guidance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Forwarding control frames while data is paused was considered and rejected: the backend is a single ordered stream with control interleaved behind data, so forwarding control past a gated data frame needs (bounded) read-ahead buffering of the data, for a narrow payoff — and it is largely moot since control already flows during credit throttling and the backend produces ~none during a full stall. Documented the finding in TODO.md and added client guidance in the tests README: throttle with small next() credits rather than an indefinite pause(); a fully-paused client won't see progress/errors/end until it resumes. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 12 ++++++++++++ programs/wsproxy/tests/README.md | 8 ++++++++ 2 files changed, 20 insertions(+) diff --git a/TODO.md b/TODO.md index ece81ebcd82e..944ed31637d3 100644 --- a/TODO.md +++ b/TODO.md @@ -405,6 +405,18 @@ resume, and unchanged push mode); 98 tests green. Client-side alternatives still valid where preferred (documented in the tests README): `WebSocketStream` (transport backpressure), or pausing the underlying `ws` socket in Node. +**Considered & rejected — forwarding control frames while data is paused** ("keep progress/errors +flowing during a `pause()`"). Killed by ordering: the backend is a single ordered stream with +control interleaved *behind* data, so forwarding a control packet past a gated data frame requires +consuming and buffering that data (unbounded → the OOM we just fixed, or a bounded read-ahead +queue — moderate rework for a narrow payoff). It's also largely moot: during credit *throttling* +control already flows (control frames aren't gated, and the loop only stalls at credit 0 / paused), +and during a *full* stall the backend is blocked and produces ~no new control anyway. So `pause()` +(or credit sitting at 0) halts control too. **Guidance: throttle with small `next()` credits rather +than an indefinite `pause()`** if you want progress/errors to keep flowing; a fully-paused client +won't observe progress / errors / `end` until it grants credit or resumes. (Only revisit the +bounded read-ahead if prompt terminal-event delivery to a *fully paused* client becomes a real need.) + ## Plan 1. **Skeleton.** Standalone `programs/wsproxy/` binary. **DONE — builds, links, runs-to-listen.** diff --git a/programs/wsproxy/tests/README.md b/programs/wsproxy/tests/README.md index 46b8e6861d8c..1b55a4d46a5d 100644 --- a/programs/wsproxy/tests/README.md +++ b/programs/wsproxy/tests/README.md @@ -70,6 +70,14 @@ plus client-side alternatives: A frame is one WS binary message (one result block), so credit is in frames — no byte accounting. Works in any JS runtime (browser + Node). INSERT direction is unaffected (client-driven). + **Prefer throttling over an indefinite `pause()`.** When you grant small `next()` credits as you + consume, control frames (progress / logs / errors and the terminal `end`) keep flowing between + data frames. A hard `pause()` (or letting credit sit at 0) also halts control delivery: a + fully-paused client will not see progress, errors, or `end` until it grants credit or resumes. + This is inherent — once the proxy stops reading, the backend stalls, and its stream is ordered + (control is interleaved behind data), so control cannot "jump ahead" of paused data. Use `pause()` + as a short-term stop, not a long-lived one. + The client-side transport options also work if you prefer them: - **Best — `WebSocketStream`** (Chromium; Node with `--experimental-websocket-stream`). Its From ff70cc643c85b41cc38d176edb2637df15343f3a Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 21:19:07 +0200 Subject: [PATCH 30/46] Record no-SQL-parsing principle; correct the INSERT-SELECT finding Investigated the claimed INSERT ... SELECT hang: it is NOT a bug. INSERT-SELECT, inline INSERT ... VALUES, and DDL all return end and the proxy stays healthy; the only hang is a client that sends INSERT ... FORMAT X and never streams the promised data (incomplete client protocol). Documented the firm principle that the proxy must never parse SQL (the ParserQuery-based INSERT/SELECT routing is to be removed), why the native protocol cannot signal query kind pre-send (with_pending_data precedes any response; the first Data packet is ambiguous), and the planned parser-free redesign: default with_pending_data=false for all queries, with streamed inserts opting in via an explicit client control message. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 30 +++++++++++++++++++++++++----- 1 file changed, 25 insertions(+), 5 deletions(-) diff --git a/TODO.md b/TODO.md index 944ed31637d3..c8ebe7e95270 100644 --- a/TODO.md +++ b/TODO.md @@ -310,11 +310,31 @@ Progress was already covered for SELECT (progress.test.mjs). **INSERT-progress investigated and dropped:** implemented `written_rows` forwarding, but the native protocol emits **no** `Progress` packets for client-data INSERTs (verified: 0 events even for 1M -rows into MergeTree, 146ms). Reverted the dormant forwarding to avoid unverified/dead code. Server -INSERT progress would only appear for `INSERT ... SELECT` (server-side), which the current -`executeInsert` streaming model doesn't handle — noted as a separate limitation: -`INSERT ... SELECT` / `INSERT ... FROM INFILE` (no client data) would make `executeInsert` wait for -WS data that never comes. Fix later (detect no-data INSERTs and skip the data-streaming phase). +rows into MergeTree, 146ms). Reverted the dormant forwarding to avoid unverified/dead code. + +## DESIGN PRINCIPLE: never parse SQL in the proxy + +The proxy must **not** parse or interpret SQL. We tried it (`ParserQuery` to route INSERT vs SELECT) +and it's the wrong approach: fragile, couples the proxy to SQL semantics, and even the real C++ +client only knows a query is a data-INSERT by parsing (`ASTInsertQuery`) — there is **no** protocol +signal at query-send time (`with_pending_data` is in the Query packet before any response, and the +server's first `Data` packet is ambiguous between a sample header and a result header). So query +kind must be driven by the **client's explicit intent**, never by inspecting the SQL. + +### INSERT-SELECT "hang" — investigated: NOT a bug (earlier note was wrong) + +Ran it end-to-end: `INSERT … SELECT`, inline `INSERT … VALUES`, and DDL all return `end` correctly +and the proxy stays healthy. The only hang is a client that sends `INSERT … FORMAT X` and then never +streams the data it promised (an incomplete client protocol, not misrouting) — the proxy waits out +the receive timeout. The current `ParserQuery` routing happens to handle INSERT-SELECT (the sample +loop catches `EndOfStream`), but the parser must go regardless (see principle above). + +**Planned refactor (remove the parser):** default every query to the plain path with +`with_pending_data=false` — this handles SELECT / INSERT-SELECT / inline-INSERT / DDL, and never +waits for client data (so the bare-`FORMAT`-insert hang disappears too). A **streamed data INSERT** +(the edge-format-conversion feature) becomes an explicit client control message +(`{"cmd":"insert",…}`, message-type framing — not SQL parsing) that opts into `with_pending_data=true` ++ the data phase. Client-API change; pending confirmation of the signal mechanism. Coverage still thin / future pushes: slow-loris/partial-frame timeouts, TLS, protocol-revision skew across older server versions (need old server binaries). Productionization track (separate): From dff4f593a2a3e670a0f4cb9a6a4dfd1d1101d7f4 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 21:41:00 +0200 Subject: [PATCH 31/46] Stop parsing SQL: route INSERT vs query by client message kind Remove the ParserQuery/ASTInsertQuery-based routing (the proxy must never parse SQL). run() now dispatches by message envelope: a raw text frame is a plain query -> executeSelect with with_pending_data=false (handles SELECT / INSERT-SELECT / inline INSERT / DDL, and never waits for client data, so the bare-FORMAT-insert hang is gone); a top-level {"cmd":"insert","query":...,"format":...} control message opts into executeInsert (streamed data phase). parseInsertCommand inspects only the JSON envelope (cmd), never SQL. executeInsert simplified: no inline-data handling, always reads the streamed WS frames, input format from the command (or the session ?format=). Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/ProxySession.cpp | 125 ++++++++++++++---------------- programs/wsproxy/ProxySession.h | 15 ++-- 2 files changed, 68 insertions(+), 72 deletions(-) diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp index 06d1abb65b96..5d29053ce400 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/programs/wsproxy/ProxySession.cpp @@ -17,14 +17,9 @@ #include #include -#include -#include -#include - #include #include #include -#include #include #include @@ -44,10 +39,39 @@ using namespace DB::WsProxy; namespace { -/// Generous parser limits for the best-effort INSERT-detection parse. -constexpr size_t MAX_QUERY_SIZE = 16 << 20; -constexpr size_t MAX_PARSER_DEPTH = 1000; -constexpr size_t MAX_PARSER_BACKTRACKS = 1'000'000; +/// A parsed `{"cmd":"insert",...}` control message (a streamed data INSERT). +struct InsertCommand +{ + bool is_insert = false; + String query; + String format; /// Input format for the client's streamed data; empty = use session default. +}; + +/// Detect and parse a streamed-insert control message WITHOUT touching SQL. Only +/// a top-level JSON object with `"cmd":"insert"` is an insert; anything else is a +/// plain query. (SQL never starts with '{', so the check is unambiguous.) +InsertCommand parseInsertCommand(const String & message) +{ + InsertCommand cmd; + const size_t first = message.find_first_not_of(" \t\r\n"); + if (first == String::npos || message[first] != '{') + return cmd; /// Not JSON -> a plain query. + try + { + Poco::JSON::Parser parser; + const auto obj = parser.parse(message).extract(); + if (obj->optValue("cmd", "") != "insert") + return cmd; + cmd.is_insert = true; + cmd.query = obj->optValue("query", ""); + cmd.format = obj->optValue("format", ""); + } + catch (...) + { + cmd.is_insert = false; /// Malformed -> treat as a plain query; the backend reports errors. + } + return cmd; +} /// Opt-in credit/window flow control for the SELECT push direction. The client /// grants credit in whole frames (`{"cmd":"next","n":N}`) and can `pause`/`resume`; @@ -439,31 +463,6 @@ std::optional ProxySession::readClientMessage() } } -bool ProxySession::executeQuery(Connection & connection, const String & query) -{ - /// Best-effort parse to detect an INSERT (with or without inline data). - /// Anything that does not parse as an INSERT takes the generic path, where - /// the backend reports real syntax/semantic errors. - ASTPtr ast; - try - { - ParserQuery parser(query.data() + query.size()); - ast = parseQuery(parser, query, MAX_QUERY_SIZE, MAX_PARSER_DEPTH, MAX_PARSER_BACKTRACKS); - } - catch (...) - { - ast = nullptr; - } - - if (ast) - { - if (const auto * insert = ast->as()) - return executeInsert(connection, query, *insert); - } - - return executeSelect(connection, query); -} - bool ProxySession::executeSelect(Connection & connection, const String & query) { LoggerPtr log = getLogger("ProxySession"); @@ -659,19 +658,16 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) } } -bool ProxySession::executeInsert(Connection & connection, const String & query, const ASTInsertQuery & insert) +bool ProxySession::executeInsert(Connection & connection, const String & query, const String & input_format) { LoggerPtr log = getLogger("ProxySession"); - const char * query_end = query.data() + query.size(); - const bool has_inline_data = insert.data && insert.data < query_end; - /// Format precedence: an explicit `FORMAT` clause; else `Values` for inline - /// `INSERT ... VALUES (...)` data; else the session's `?format=` for streamed data. - const String insert_format = !insert.format.empty() ? insert.format : (has_inline_data ? "Values" : format); - /// Send the query without the inline data section, keeping the FORMAT clause. - const String query_to_send = insert.data ? String(query.data(), insert.data) : query; + /// Format the client's streamed data is in: the command's `format` if given, + /// else the session's `?format=`. The query text is forwarded verbatim (never + /// parsed); the backend receives native blocks via sendData regardless. + const String data_format = !input_format.empty() ? input_format : format; - sendBackendQuery(connection, query_to_send, /* with_pending_data */ true); + sendBackendQuery(connection, query, /* with_pending_data */ true); /// The server waits for external-tables data before replying with the sample /// block; we have none, so send an empty set to unblock the handshake. @@ -696,31 +692,22 @@ bool ProxySession::executeInsert(Connection & connection, const String & query, } if (packet.type == Protocol::Server::EndOfStream) { - /// No structure to send data with (e.g. nothing was expected); we are done. + /// The server did not ask for data (e.g. the query carried its own source); + /// nothing to stream, we are done. sendControlEvent("end", ""); return true; } /// Ignore Log / Progress / TableColumns / TimezoneUpdate while waiting. } - /// Data source: inline data from the query if present, else streamed from the client. - std::unique_ptr data_in; - ReadBufferFromWebSocket * ws_in = nullptr; - if (has_inline_data) - { - data_in = std::make_unique(insert.data, query_end - insert.data); - } - else - { - auto ws = std::make_unique(socket); - ws_in = ws.get(); - data_in = std::move(ws); - } + /// The client streams the INSERT data as WebSocket binary frames. + auto data_in = std::make_unique(socket); + ReadBufferFromWebSocket * ws_in = data_in.get(); bool aborted = false; try { - auto source = context->getInputFormat(insert_format, *data_in, sample, DEFAULT_BLOCK_SIZE); + auto source = context->getInputFormat(data_format, *data_in, sample, DEFAULT_BLOCK_SIZE); Pipe pipe(source); QueryPipeline pipeline(std::move(pipe)); PullingPipelineExecutor executor(pipeline); @@ -731,7 +718,7 @@ bool ProxySession::executeInsert(Connection & connection, const String & query, if (!block.empty()) connection.sendData(block, /* name */ "", /* scalar */ false); } - aborted = ws_in && ws_in->wasAborted(); + aborted = ws_in->wasAborted(); } catch (...) { @@ -824,25 +811,31 @@ void ProxySession::run() while (true) { - auto query = readClientMessage(); - if (!query) + auto message = readClientMessage(); + if (!message) break; - if (query->empty()) + if (message->empty()) continue; try { - if (!executeQuery(connection, *query)) + /// Route by message kind, never by parsing SQL: a {"cmd":"insert",...} + /// control message is a streamed data INSERT; anything else is a plain + /// query (which covers SELECT / INSERT-SELECT / inline INSERT / DDL). + const InsertCommand ins = parseInsertCommand(*message); + const bool keep_open = ins.is_insert ? executeInsert(connection, ins.query, ins.format) + : executeSelect(connection, *message); + if (!keep_open) break; } catch (...) { - const String message = getCurrentExceptionMessage(false); - LOG_DEBUG(log, "Session error: {}", message); + const String error_message = getCurrentExceptionMessage(false); + LOG_DEBUG(log, "Session error: {}", error_message); try { - sendControlEvent("error", message); + sendControlEvent("error", error_message); } catch (...) { diff --git a/programs/wsproxy/ProxySession.h b/programs/wsproxy/ProxySession.h index dc3dde35959e..b44ba9d3f085 100644 --- a/programs/wsproxy/ProxySession.h +++ b/programs/wsproxy/ProxySession.h @@ -11,7 +11,6 @@ namespace DB { class Connection; -class ASTInsertQuery; /// Where the proxy forwards native-protocol queries. struct BackendParams @@ -52,12 +51,16 @@ class ProxySession /// (Close frame, read error, or protocol violation). std::optional readClientMessage(); - /// Execute one query and stream its result. Returns false if the session - /// should end afterwards (client closed mid-query). Dispatches to the - /// INSERT or SELECT path depending on the parsed query kind. - bool executeQuery(Connection & connection, const String & query); + /// Run a plain query and stream its result. Returns false if the session + /// should end afterwards (client closed mid-query). The proxy never parses + /// SQL: routing is driven by the client's message kind (a raw text frame is + /// a query -> executeSelect; a {"cmd":"insert",...} control message opts into + /// executeInsert). `with_pending_data=false` here, so SELECT / INSERT-SELECT / + /// inline INSERT / DDL all work and none waits for client data. bool executeSelect(Connection & connection, const String & query); - bool executeInsert(Connection & connection, const String & query, const ASTInsertQuery & insert); + /// Stream a client-supplied data INSERT: send the query with `with_pending_data`, + /// parse the streamed frames with `input_format`, and `sendData` blocks. + bool executeInsert(Connection & connection, const String & query, const String & input_format); void sendBackendQuery(Connection & connection, const String & query, bool with_pending_data = false); void drainUntilEndOfStream(Connection & connection); From 32ee0ffff522515309b3b86d3d98f1bc1d1f4420 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 21:41:00 +0200 Subject: [PATCH 32/46] Client streamed-insert control message; tests and docs for no-SQL-parse routing Session.insert now declares a streamed insert with a {"cmd":"insert",...} control message (added Session.beginInsert(query,{format}); insert() takes an optional input format). Tests updated: streamed inserts pass an explicit format where it differs from the session (TSV), the mid-stream cancel test uses beginInsert, and new cases assert INSERT-SELECT and inline VALUES run as plain queries. README documents the message-kind routing and the streamed-insert control message. 100 tests, all green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 15 ++++++++----- programs/wsproxy/tests/README.md | 16 +++++++++----- programs/wsproxy/tests/test/helpers.mjs | 17 +++++++++++--- .../wsproxy/tests/test/insert-flow.test.mjs | 4 ++-- programs/wsproxy/tests/test/insert.test.mjs | 22 ++++++++++++++++--- .../wsproxy/tests/test/large-insert.test.mjs | 1 + 6 files changed, 56 insertions(+), 19 deletions(-) diff --git a/TODO.md b/TODO.md index c8ebe7e95270..9b94e0de527e 100644 --- a/TODO.md +++ b/TODO.md @@ -329,12 +329,15 @@ streams the data it promised (an incomplete client protocol, not misrouting) — the receive timeout. The current `ParserQuery` routing happens to handle INSERT-SELECT (the sample loop catches `EndOfStream`), but the parser must go regardless (see principle above). -**Planned refactor (remove the parser):** default every query to the plain path with -`with_pending_data=false` — this handles SELECT / INSERT-SELECT / inline-INSERT / DDL, and never -waits for client data (so the bare-`FORMAT`-insert hang disappears too). A **streamed data INSERT** -(the edge-format-conversion feature) becomes an explicit client control message -(`{"cmd":"insert",…}`, message-type framing — not SQL parsing) that opts into `with_pending_data=true` -+ the data phase. Client-API change; pending confirmation of the signal mechanism. +**DONE — parser removed.** `ProxySession` no longer links or calls `ParserQuery`/`ASTInsertQuery`. +Routing is by message kind: a raw text frame is a plain query → `executeSelect` +(`with_pending_data=false`, handles SELECT / INSERT-SELECT / inline-INSERT / DDL, never waits for +client data → the bare-`FORMAT`-insert hang is gone); a `{"cmd":"insert","query":...,"format":...}` +control message opts into `executeInsert` (streamed data phase, `with_pending_data=true`). +`parseInsertCommand` only inspects the message envelope (top-level JSON `cmd`), never SQL. +`executeInsert` simplified (no inline-data handling; always reads the streamed frames). JS +`Session.insert` sends the control message; added `Session.beginInsert(query,{format})`. 100 tests +green, incl. explicit INSERT-SELECT / inline-VALUES as plain queries. Coverage still thin / future pushes: slow-loris/partial-frame timeouts, TLS, protocol-revision skew across older server versions (need old server binaries). Productionization track (separate): diff --git a/programs/wsproxy/tests/README.md b/programs/wsproxy/tests/README.md index 1b55a4d46a5d..bb2a17359bab 100644 --- a/programs/wsproxy/tests/README.md +++ b/programs/wsproxy/tests/README.md @@ -34,11 +34,17 @@ it started. Override binary/config locations with the `CLICKHOUSE_SERVER`, `CH_C ## Protocol under test -The client sends a SQL query as a **text** frame. SELECT results stream back as **binary** -frames encoded in the output format chosen via the WS URL `?format=` parameter. Mid-query -progress and the terminal outcome (`end` / `error` / `cancelled`) arrive as JSON **text** -frames. For INSERT, the client sends `INSERT INTO t [FORMAT X]` then streams data as binary -frames ending with a zero-length binary frame; closing the socket mid-query cancels it. +The client sends a SQL query as a **text** frame; results stream back as **binary** frames encoded +in the output format chosen via the WS URL `?format=` parameter. Mid-query progress and the terminal +outcome (`end` / `error` / `cancelled`) arrive as JSON **text** frames. This plain-query path covers +SELECT, `INSERT … SELECT`, inline `INSERT … VALUES`, and DDL. + +**The proxy never parses SQL** — routing is by message kind. A **streamed data INSERT** (where the +client sends the rows, e.g. for edge format conversion) is opted into with a control message +`{"cmd":"insert","query":"INSERT INTO t FORMAT JSONEachRow","format":"JSONEachRow"}`, after which the +client streams the data as **binary** frames ending with a zero-length binary frame (`format` is the +input format of that data; it defaults to the session `?format=`). Closing the socket mid-query +cancels it. The `Session.insert(query, chunks, { format })` helper wraps this. See `test/helpers.mjs` for the small client wrapper (`Session`, `runQuery`, `backendScalar`). diff --git a/programs/wsproxy/tests/test/helpers.mjs b/programs/wsproxy/tests/test/helpers.mjs index 673cd4a014ef..340aca6a865a 100644 --- a/programs/wsproxy/tests/test/helpers.mjs +++ b/programs/wsproxy/tests/test/helpers.mjs @@ -175,12 +175,23 @@ export class Session { } /** - * Run an INSERT: send the query, stream the data chunks, end, and collect + * Begin a streamed INSERT: declare it with a control message so the proxy + * opts into the data phase (the proxy never parses SQL). `format` is the input + * format of the data you'll stream (defaults to the session's format). + */ + beginInsert(query, { format } = {}) { + const cmd = { cmd: "insert", query }; + if (format) cmd.format = format; + this.ws.send(JSON.stringify(cmd)); + } + + /** + * Run a streamed INSERT: declare it, stream the data chunks, end, and collect * the terminal control event. `chunks` is an array of strings/Buffers. */ - async insert(query, chunks = []) { + async insert(query, chunks = [], opts = {}) { await this.ready(); - this.sendQuery(query); + this.beginInsert(query, opts); for (const c of chunks) this.sendData(c); this.endData(); return this.collect(); diff --git a/programs/wsproxy/tests/test/insert-flow.test.mjs b/programs/wsproxy/tests/test/insert-flow.test.mjs index d124124f25cd..72e2ec4f54f1 100644 --- a/programs/wsproxy/tests/test/insert-flow.test.mjs +++ b/programs/wsproxy/tests/test/insert-flow.test.mjs @@ -9,7 +9,7 @@ describe("insert flow", () => { it("aborts an INSERT (no commit) when the client closes mid-stream", async () => { const s = new Session("JSONEachRow"); await s.ready(); - s.sendQuery("INSERT INTO default.wsp_test FORMAT JSONEachRow"); + s.beginInsert("INSERT INTO default.wsp_test FORMAT JSONEachRow"); // Send some data but NEVER send the end marker; close the socket instead. s.sendData('{"a":1,"b":"x"}\n{"a":2,"b":"y"}\n'); await sleep(200); @@ -24,7 +24,7 @@ describe("insert flow", () => { // Abort one insert... const a = new Session("JSONEachRow"); await a.ready(); - a.sendQuery("INSERT INTO default.wsp_test FORMAT JSONEachRow"); + a.beginInsert("INSERT INTO default.wsp_test FORMAT JSONEachRow"); a.sendData('{"a":9,"b":"q"}\n'); await sleep(150); a.close(); diff --git a/programs/wsproxy/tests/test/insert.test.mjs b/programs/wsproxy/tests/test/insert.test.mjs index 7394a8911a2d..f28ff537b6ba 100644 --- a/programs/wsproxy/tests/test/insert.test.mjs +++ b/programs/wsproxy/tests/test/insert.test.mjs @@ -6,6 +6,20 @@ describe("INSERT", () => { await runQuery("TRUNCATE TABLE default.wsp_test"); }); + // These INSERTs carry their own data (SELECT source / inline VALUES), so they + // run as plain queries — no streamed data phase, and the proxy never parses SQL. + it("runs INSERT ... SELECT as a plain query (no data phase)", async () => { + const r = await runQuery("INSERT INTO default.wsp_test SELECT number, 'z' FROM numbers(4)"); + expect(r.control.event).toBe("end"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("4"); + }); + + it("runs inline INSERT ... VALUES as a plain query", async () => { + const r = await runQuery("INSERT INTO default.wsp_test VALUES (1, 'a'), (2, 'b')"); + expect(r.control.event).toBe("end"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + }); + it("streams a JSONEachRow insert across multiple chunks", async () => { const s = new Session("JSONEachRow"); try { @@ -41,9 +55,11 @@ describe("INSERT", () => { it("streams an insert in a different input format (TSV)", async () => { const s = new Session("JSONEachRow"); try { - const { control } = await s.insert("INSERT INTO default.wsp_test FORMAT TSV", [ - "5\tp\n6\tq\n", - ]); + const { control } = await s.insert( + "INSERT INTO default.wsp_test FORMAT TSV", + ["5\tp\n6\tq\n"], + { format: "TSV" }, // input format for the streamed data + ); expect(control.event).toBe("end"); } finally { s.close(); diff --git a/programs/wsproxy/tests/test/large-insert.test.mjs b/programs/wsproxy/tests/test/large-insert.test.mjs index 3f941e41876c..66f6b0fcc64b 100644 --- a/programs/wsproxy/tests/test/large-insert.test.mjs +++ b/programs/wsproxy/tests/test/large-insert.test.mjs @@ -91,6 +91,7 @@ describe("large INSERT", () => { ({ control } = await s.insert( "INSERT INTO default.wsp_test FORMAT TSV", [chunk], + { format: "TSV" }, // input format for the streamed data )); } finally { s.close(); From eea660e656afeacdfc98c9235801178aeac98e35 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 22:15:18 +0200 Subject: [PATCH 33/46] Add opt-in SQL parsing (?parse=1) that reports the query kind By default the proxy still never parses SQL. With ?parse=1 the client asks the proxy to parse each query in one place (classifyQuery): it pushes a non-terminal {"event":"query","kind":"insert"|"query","verb":""} frame (clients frequently need the verb and the routing decision) and auto-routes a streamed-data INSERT (INSERT ... [FORMAT X] with no SELECT source, no INFILE and no inline data) so the client need not send the {"cmd":"insert"} envelope. verb comes from the SQL Lexer (skips leading comments); kind is the routing decision. An explicit {"cmd":"insert"} message still wins over parsing, and a parse failure degrades to the plain-query path (the backend reports real errors) so a parser quirk can never wedge a statement. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/ProxySession.cpp | 114 +++++++++++++++++++++++++++- programs/wsproxy/ProxySession.h | 4 +- programs/wsproxy/WsProxyHandler.cpp | 9 ++- 3 files changed, 121 insertions(+), 6 deletions(-) diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp index 5d29053ce400..336a578d1bfd 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/programs/wsproxy/ProxySession.cpp @@ -11,6 +11,11 @@ #include #include +#include +#include +#include +#include + #include #include #include @@ -73,6 +78,78 @@ InsertCommand parseInsertCommand(const String & message) return cmd; } +/// The leading SQL keyword of a query (e.g. `SELECT`, `INSERT`, `CREATE`), +/// uppercased. Uses the SQL lexer so leading comments/whitespace are skipped +/// correctly. Clients frequently want this to drive their own logic without +/// re-implementing a tokenizer. Returns empty if there is no significant token. +String leadingVerb(const String & query) +{ + Lexer lexer(query.data(), query.data() + query.size()); + for (Token token = lexer.nextToken(); !token.isEnd() && !token.isError(); token = lexer.nextToken()) + { + if (!token.isSignificant()) + continue; + String verb(token.begin, token.size()); + for (char & c : verb) + if (c >= 'a' && c <= 'z') + c = static_cast(c - ('a' - 'A')); + return verb; + } + return ""; +} + +/// The proxy's classification of a query, reported to the client in ?parse=1 mode. +struct QueryClass +{ + String verb; /// Leading SQL keyword, uppercased. + bool streamed_insert = false; /// True if it is an INSERT that expects client-streamed data. + String insert_format; /// The FORMAT clause of such an INSERT (empty = session default). +}; + +/// OPT-IN only (?parse=1): parse the query so the proxy can (a) report the leading +/// verb + routing decision to the client and (b) auto-route a streamed-data INSERT +/// without the client sending an explicit {"cmd":"insert",...} envelope. +/// +/// A query needs the client-streamed data phase iff it is an `INSERT` with no +/// SELECT source, no INFILE, and no inline data (`INSERT INTO t [FORMAT X]` with +/// the rows still to come). Everything else (SELECT / INSERT-SELECT / inline +/// VALUES / DDL) is a plain query. +/// +/// This is the ONLY place the proxy parses SQL, and only when the client asks for +/// it. Parse failure degrades to {verb-from-lexer, plain query} so a parser quirk +/// never wedges a statement the backend would accept; the backend reports errors. +QueryClass classifyQuery(const String & query) +{ + QueryClass result; + result.verb = leadingVerb(query); + + try + { + ParserQuery parser(query.data() + query.size()); + ASTPtr ast = parseQuery( + parser, + query, + /* max_query_size */ 0, /// 0 = unlimited; the backend enforces the real limit. + /* max_parser_depth */ 1000, + /* max_parser_backtracks */ 1'000'000); + + if (const auto * insert = ast->as()) + { + const bool has_inline_data = insert->data != nullptr && insert->data != insert->end; + result.streamed_insert = !insert->select && !insert->infile && !has_inline_data; + result.insert_format = insert->format; + } + } + catch (...) + { + LOG_DEBUG( + getLogger("ProxySession"), + "parse=1 classification failed, treating as a plain query: {}", + getCurrentExceptionMessage(false)); + } + return result; +} + /// Opt-in credit/window flow control for the SELECT push direction. The client /// grants credit in whole frames (`{"cmd":"next","n":N}`) and can `pause`/`resume`; /// the proxy sends a data frame only while `!paused && credit > 0`. This lets an @@ -299,6 +376,15 @@ String escapeJSON(const String & s) return out; } +/// Report the proxy's parse decision to the client (only in ?parse=1 mode), as a +/// non-terminal control frame preceding the result. `kind` is the routing +/// decision: "insert" = the proxy will read client-streamed data; "query" = the +/// plain path (results / self-contained statement). `verb` is the leading keyword. +void sendQueryInfoFrame(Poco::Net::StreamSocket & socket, const String & verb, const String & kind) +{ + sendWebSocketText(socket, R"({"event":"query","kind":")" + kind + R"(","verb":")" + escapeJSON(verb) + R"("})"); +} + } ProxySession::ProxySession( @@ -308,7 +394,8 @@ ProxySession::ProxySession( String format_, String logs_level_, bool flow_enabled_, - Int64 flow_initial_credit_) + Int64 flow_initial_credit_, + bool parse_enabled_) : socket(socket_) , context(std::move(context_)) , backend(std::move(backend_)) @@ -316,6 +403,7 @@ ProxySession::ProxySession( , logs_level(std::move(logs_level_)) , flow_enabled(flow_enabled_) , flow_initial_credit(flow_initial_credit_) + , parse_enabled(parse_enabled_) { } @@ -820,12 +908,30 @@ void ProxySession::run() try { - /// Route by message kind, never by parsing SQL: a {"cmd":"insert",...} + /// Route by message kind, not by parsing SQL: a {"cmd":"insert",...} /// control message is a streamed data INSERT; anything else is a plain /// query (which covers SELECT / INSERT-SELECT / inline INSERT / DDL). const InsertCommand ins = parseInsertCommand(*message); - const bool keep_open = ins.is_insert ? executeInsert(connection, ins.query, ins.format) - : executeSelect(connection, *message); + bool keep_open = false; + if (ins.is_insert) + { + /// The client explicitly declared a streamed insert; honour it as-is. + keep_open = executeInsert(connection, ins.query, ins.format); + } + else if (parse_enabled) + { + /// Opt-in (?parse=1): parse the query to classify it, report the + /// decision to the client, and auto-route a streamed-data INSERT + /// without requiring the explicit control message. + const QueryClass qc = classifyQuery(*message); + sendQueryInfoFrame(socket, qc.verb, qc.streamed_insert ? "insert" : "query"); + keep_open = qc.streamed_insert ? executeInsert(connection, *message, qc.insert_format) + : executeSelect(connection, *message); + } + else + { + keep_open = executeSelect(connection, *message); + } if (!keep_open) break; } diff --git a/programs/wsproxy/ProxySession.h b/programs/wsproxy/ProxySession.h index b44ba9d3f085..77a5cd1f24d9 100644 --- a/programs/wsproxy/ProxySession.h +++ b/programs/wsproxy/ProxySession.h @@ -41,7 +41,8 @@ class ProxySession String format_, String logs_level_ = "", bool flow_enabled_ = false, - Int64 flow_initial_credit_ = 0); + Int64 flow_initial_credit_ = 0, + bool parse_enabled_ = false); void run(); @@ -75,6 +76,7 @@ class ProxySession String logs_level; /// If set, sent as `send_logs_level` so the backend pushes Log packets. bool flow_enabled; /// Opt-in credit/window flow control for the SELECT push direction. Int64 flow_initial_credit; /// Starting frame credit when flow control is enabled. + bool parse_enabled; /// Opt-in (?parse=1): parse SQL to auto-route inserts and report the query kind. }; } diff --git a/programs/wsproxy/WsProxyHandler.cpp b/programs/wsproxy/WsProxyHandler.cpp index 4f3693d1367f..6abf61cfdc1e 100644 --- a/programs/wsproxy/WsProxyHandler.cpp +++ b/programs/wsproxy/WsProxyHandler.cpp @@ -216,6 +216,12 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp const String flow_param = queryParam(uri, "flow", "", log); const bool flow_enabled = !flow_param.empty(); const Int64 flow_credit = flow_enabled ? std::strtoll(flow_param.c_str(), nullptr, 10) : 0; + /// Opt-in SQL parsing: `?parse=1` lets the proxy parse each query to auto-route + /// streamed-data INSERTs (no {"cmd":"insert"} needed) and report the parsed + /// verb + routing decision as a {"event":"query",...} frame. Off by default — + /// the proxy does not parse SQL unless the client explicitly asks it to. + const String parse_param = queryParam(uri, "parse", "", log); + const bool parse_enabled = parse_param == "1" || parse_param == "true"; /// Credential pass-through: resolve this session's backend user/password from /// the request (never mutate the shared default `backend`). @@ -239,7 +245,8 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp if (send_timeout_sec > 0) socket.setSendTimeout(Poco::Timespan(send_timeout_sec * 1'000'000)); /// microseconds - ProxySession session(socket, context, session_backend, out_format, logs_level, flow_enabled, flow_credit); + ProxySession session( + socket, context, session_backend, out_format, logs_level, flow_enabled, flow_credit, parse_enabled); session.run(); LOG_DEBUG(log, "WebSocket session closed"); From e1dee772ffb5cf19bd680e45f3540f464f04e90d Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Fri, 10 Jul 2026 22:15:19 +0200 Subject: [PATCH 34/46] Tests and docs for opt-in parse mode MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit helpers.mjs: ?parse=1 session option; collect() captures the {"event":"query"} frame as queryInfo. parse.test.mjs: 9 cases — SELECT/INSERT-SELECT/inline-VALUES/DDL report the right verb+kind, a bare INSERT ... FORMAT X auto-routes as a streamed insert (with the FORMAT clause as the input format), the leading verb survives leading comments, unparseable SQL degrades to the plain path, and parse-off sends no query frame. README documents ?parse=1. 109 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 10 ++ programs/wsproxy/tests/README.md | 10 ++ programs/wsproxy/tests/test/helpers.mjs | 13 +- programs/wsproxy/tests/test/parse.test.mjs | 136 +++++++++++++++++++++ 4 files changed, 164 insertions(+), 5 deletions(-) create mode 100644 programs/wsproxy/tests/test/parse.test.mjs diff --git a/TODO.md b/TODO.md index 9b94e0de527e..481c4a8e22f8 100644 --- a/TODO.md +++ b/TODO.md @@ -339,6 +339,16 @@ control message opts into `executeInsert` (streamed data phase, `with_pending_da `Session.insert` sends the control message; added `Session.beginInsert(query,{format})`. 100 tests green, incl. explicit INSERT-SELECT / inline-VALUES as plain queries. +**Opt-in SQL parsing (`?parse=1`) — DONE.** For clients that don't want to classify their own SQL, +`?parse=1` makes the proxy parse each query in ONE place (`classifyQuery`) to: (a) push a +`{"event":"query","kind":"insert"|"query","verb":""}` frame (clients often need the +verb + kind), and (b) auto-route a streamed-data INSERT (`INSERT … [FORMAT X]` with no +SELECT/INFILE/inline-data) without the `{"cmd":"insert"}` envelope. `verb` comes from the SQL `Lexer` +(skips leading comments); `kind` is the routing decision. Off by default — the no-parse contract is +unchanged for default clients — and a parse failure degrades to the plain-query path (backend reports +real errors), so a parser quirk can never wedge a statement. The explicit `{"cmd":"insert"}` message +always wins over parsing. 109 tests green. + Coverage still thin / future pushes: slow-loris/partial-frame timeouts, TLS, protocol-revision skew across older server versions (need old server binaries). Productionization track (separate): auth (done), proxy→backend TLS, config file, resource limits, graceful drain, `BaseDaemon`, diff --git a/programs/wsproxy/tests/README.md b/programs/wsproxy/tests/README.md index bb2a17359bab..0f9f4ed01025 100644 --- a/programs/wsproxy/tests/README.md +++ b/programs/wsproxy/tests/README.md @@ -46,6 +46,16 @@ client streams the data as **binary** frames ending with a zero-length binary fr input format of that data; it defaults to the session `?format=`). Closing the socket mid-query cancels it. The `Session.insert(query, chunks, { format })` helper wraps this. +**Opt-in SQL parsing (`?parse=1`).** For clients that don't want to classify their own SQL, connect +with `?parse=1`. The proxy then parses each query to (a) push a non-terminal `{"event":"query", +"kind":"insert"|"query","verb":""}` frame telling the client the leading verb and +the routing decision, and (b) auto-route a streamed-data INSERT (`INSERT INTO t [FORMAT X]` with no +SELECT source / INFILE / inline data) without needing the `{"cmd":"insert"}` message — the client +just sends the query text and then streams the binary data frames. `kind` is `insert` when the proxy +will read client data, else `query`. Parsing is **off by default**: the proxy does not parse SQL +unless the client explicitly asks it to, and a parse failure degrades to the plain-query path (the +backend reports any real error). This is the only place the proxy parses SQL. + See `test/helpers.mjs` for the small client wrapper (`Session`, `runQuery`, `backendScalar`). ## Receive backpressure (important for large results) diff --git a/programs/wsproxy/tests/test/helpers.mjs b/programs/wsproxy/tests/test/helpers.mjs index 340aca6a865a..086c032d968a 100644 --- a/programs/wsproxy/tests/test/helpers.mjs +++ b/programs/wsproxy/tests/test/helpers.mjs @@ -15,16 +15,17 @@ export const PROXY_URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; -/** Build the WS URL for a format and options ({ logs, path, baseUrl, user, password, flow }). */ +/** Build the WS URL for a format and options ({ logs, path, baseUrl, user, password, flow, parse }). */ export function urlFor( format = "JSONEachRow", - { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow } = {}, + { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow, parse = false } = {}, ) { const params = new URLSearchParams({ format }); if (logs) params.set("logs", logs); if (user) params.set("user", user); if (password) params.set("password", password); if (flow !== undefined) params.set("flow", String(flow)); // opt-in credit flow control + if (parse) params.set("parse", "1"); // opt-in SQL parsing (auto-route inserts, report kind) return `${baseUrl}${path}?${params}`; } @@ -36,9 +37,9 @@ export function urlFor( export class Session { constructor( format = "JSONEachRow", - { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow } = {}, + { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow, parse = false } = {}, ) { - this.ws = new WebSocket(urlFor(format, { logs, path, baseUrl, user, password, flow })); + this.ws = new WebSocket(urlFor(format, { logs, path, baseUrl, user, password, flow, parse })); this.ws.binaryType = "arraybuffer"; this._queue = []; this._waiters = []; @@ -142,9 +143,10 @@ export class Session { const progress = []; const logs = []; const profileEvents = []; + let queryInfo = null; // {"event":"query",...} sent in ?parse=1 mode const done = (control) => { const data = Buffer.concat(chunks); - return { data, text: data.toString("utf8"), progress, logs, profileEvents, control }; + return { data, text: data.toString("utf8"), progress, logs, profileEvents, queryInfo, control }; }; for (;;) { const frame = await this.nextFrame(); @@ -159,6 +161,7 @@ export class Session { if (msg.event === "progress") progress.push(msg); else if (msg.event === "log") logs.push(msg); else if (msg.event === "profile_events") profileEvents.push(msg); + else if (msg.event === "query") queryInfo = msg; // Unknown non-terminal events are ignored. } else { // Socket closed without a terminal control frame. diff --git a/programs/wsproxy/tests/test/parse.test.mjs b/programs/wsproxy/tests/test/parse.test.mjs new file mode 100644 index 000000000000..d3442c10b9bc --- /dev/null +++ b/programs/wsproxy/tests/test/parse.test.mjs @@ -0,0 +1,136 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { Session, runQuery, backendScalar } from "./helpers.mjs"; + +// Opt-in SQL parsing (`?parse=1`). The proxy never parses SQL by default; with +// this flag it parses each query to (a) report the leading verb + routing +// decision as a {"event":"query",...} frame and (b) auto-route a streamed-data +// INSERT without the client sending an explicit {"cmd":"insert",...} message. +describe("parse mode (?parse=1)", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it("reports a SELECT as a plain query and streams results", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, text, control } = await s.run("SELECT 1 AS x"); + expect(control.event).toBe("end"); + expect(queryInfo).toEqual({ event: "query", kind: "query", verb: "SELECT" }); + expect(JSON.parse(text.trim())).toEqual({ x: 1 }); + } finally { + s.close(); + } + }); + + it("reports INSERT ... SELECT as a plain query and inserts server-side", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, control } = await s.run( + "INSERT INTO default.wsp_test SELECT number, 'z' FROM numbers(4)", + ); + expect(control.event).toBe("end"); + expect(queryInfo.kind).toBe("query"); // self-contained: no client data phase + expect(queryInfo.verb).toBe("INSERT"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("4"); + } finally { + s.close(); + } + }); + + it("reports inline INSERT ... VALUES as a plain query", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, control } = await s.run( + "INSERT INTO default.wsp_test VALUES (1, 'a'), (2, 'b')", + ); + expect(control.event).toBe("end"); + expect(queryInfo.kind).toBe("query"); // inline data: no client data phase + expect(queryInfo.verb).toBe("INSERT"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + } finally { + s.close(); + } + }); + + it("auto-routes a streamed INSERT (no control message needed)", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + await s.ready(); + // Send the INSERT as a plain text frame, then stream the rows. The proxy + // parses it, decides it needs client data, and reads the binary frames. + s.sendQuery("INSERT INTO default.wsp_test FORMAT JSONEachRow"); + s.sendData('{"a":10,"b":"p"}\n{"a":11,"b":"q"}\n'); + s.endData(); + const { queryInfo, control } = await s.collect(); + expect(control.event).toBe("end"); + expect(queryInfo).toEqual({ event: "query", kind: "insert", verb: "INSERT" }); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + } finally { + s.close(); + } + }); + + it("picks up the input format from the FORMAT clause for a streamed INSERT", async () => { + // Session output format is JSONEachRow, but the INSERT declares TSV — the + // proxy should feed the streamed data through TSV, not the session default. + const s = new Session("JSONEachRow", { parse: true }); + try { + await s.ready(); + s.sendQuery("INSERT INTO default.wsp_test FORMAT TSV"); + s.sendData("7\tp\n8\tq\n"); + s.endData(); + const { queryInfo, control } = await s.collect(); + expect(control.event).toBe("end"); + expect(queryInfo.kind).toBe("insert"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + } finally { + s.close(); + } + }); + + it("reports DDL with its leading verb", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, control } = await s.run( + "CREATE TABLE IF NOT EXISTS default.wsp_parse_ddl (x UInt8) ENGINE = Memory", + ); + expect(control.event).toBe("end"); + expect(queryInfo).toEqual({ event: "query", kind: "query", verb: "CREATE" }); + } finally { + s.close(); + await runQuery("DROP TABLE IF EXISTS default.wsp_parse_ddl"); + } + }); + + it("recovers the leading verb through leading comments", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, control } = await s.run("-- a leading comment\nSELECT 42 AS x"); + expect(control.event).toBe("end"); + expect(queryInfo.verb).toBe("SELECT"); + } finally { + s.close(); + } + }); + + it("degrades to a plain query when the SQL fails to parse", async () => { + // Unparseable text: the proxy must not wedge — it forwards the query and the + // backend reports the syntax error. queryInfo still carries the leading verb. + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, control } = await s.run("SELECT this is not valid sql ((("); + expect(control.event).toBe("error"); + expect(queryInfo.kind).toBe("query"); + expect(queryInfo.verb).toBe("SELECT"); + } finally { + s.close(); + } + }); + + it("does not send a query frame when parse mode is off", async () => { + // Default session: no ?parse=1, so no {"event":"query"} frame at all. + const { queryInfo, control } = await runQuery("SELECT 1 AS x"); + expect(control.event).toBe("end"); + expect(queryInfo).toBeNull(); + }); +}); From 48ebb2ead929209bfe95b74f45862261171681f8 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Mon, 13 Jul 2026 11:37:01 +0200 Subject: [PATCH 35/46] Add wsproxy JSONCompactEachRow throughput benchmarks Harness under programs/wsproxy/bench/: bench.mjs drives the proxy over WebSocket and reports min/median wall time + throughput; baselines.sh fetches byte-identical output directly (native client, HTTP plain, HTTP gzip) with output_format_json_quote_64bit_integers=0 so the proxy's unquoted UInt64 matches; tcp_ceiling.mjs measures Node's raw TCP loopback receive ceiling. README documents how to run against a cloud or local backend and how to read WAN- vs CPU-bound results. Kept so the comparison can be re-run against upcoming changes (e.g. parallel output formatting). Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/bench/README.md | 68 ++++++++++++++++++++++++++ programs/wsproxy/bench/baselines.sh | 35 +++++++++++++ programs/wsproxy/bench/bench.mjs | 54 ++++++++++++++++++++ programs/wsproxy/bench/tcp_ceiling.mjs | 25 ++++++++++ 4 files changed, 182 insertions(+) create mode 100644 programs/wsproxy/bench/README.md create mode 100755 programs/wsproxy/bench/baselines.sh create mode 100644 programs/wsproxy/bench/bench.mjs create mode 100644 programs/wsproxy/bench/tcp_ceiling.mjs diff --git a/programs/wsproxy/bench/README.md b/programs/wsproxy/bench/README.md new file mode 100644 index 000000000000..e184490d0481 --- /dev/null +++ b/programs/wsproxy/bench/README.md @@ -0,0 +1,68 @@ +# clickhouse-wsproxy benchmarks + +Throughput benchmarks for fetching a query result as `JSONCompactEachRow`, comparing the +proxy against fetching the same bytes directly (native `clickhouse-client` and the HTTP +interface). Used to measure the cost of edge format conversion and to validate changes such +as parallel output formatting. + +The reference query is +`SELECT number AS n, number*2 AS d, toString(number) AS s FROM numbers(N)` — deterministic, +server-generated, and forces real formatting of two `UInt64` columns and one `String`. + +## Scripts + +- `bench.mjs` — drive the proxy over WebSocket. Opens a fresh session per iteration (each is a + new backend connection, so the cloud TLS handshake is *not* amortized — matching what + `clickhouse-client` pays per run), sends the query, counts the streamed bytes until the + terminal `end`, and reports min / median wall time plus throughput. One warm-up iteration is + discarded. + + ```bash + # against a proxy on ws://127.0.0.1:9010 (override with WSPROXY_URL) + node bench.mjs + node bench.mjs 3000000 JSONCompactEachRow 5 + ``` + +- `baselines.sh` — direct baselines for the same result, with byte-identical output (forces + `output_format_json_quote_64bit_integers=0` so the proxy's unquoted `UInt64` matches + `clickhouse-client`/HTTP, which quote 64-bit ints by default). Reports three paths: native + transport + client-side format, HTTP server-side format (plain), and HTTP + gzip. Reads the + cloud endpoint from the environment. + + ```bash + CLICKHOUSE_CLOUD_HOST=... CLICKHOUSE_CLOUD_PASSWORD=... bash baselines.sh + ``` + +- `tcp_ceiling.mjs` — diagnostic: measures Node's raw TCP loopback receive ceiling, to confirm + the JS client is not the bottleneck when attributing proxy overhead. + +## Running against a backend + +Start a proxy pointed at the backend, then run `bench.mjs`. + +```bash +# Cloud backend (native secure), credentials from the environment: +WSPROXY_BACKEND_HOST="$CLICKHOUSE_CLOUD_HOST" WSPROXY_BACKEND_PORT=9440 \ +WSPROXY_BACKEND_USER=default WSPROXY_BACKEND_PASSWORD="$CLICKHOUSE_CLOUD_PASSWORD" \ +WSPROXY_BACKEND_SECURE=1 WSPROXY_PORT=9010 \ + ../../../build/programs/wsproxy/clickhouse-wsproxy & + +node bench.mjs 3000000 JSONCompactEachRow 5 +CLICKHOUSE_CLOUD_HOST="$CLICKHOUSE_CLOUD_HOST" CLICKHOUSE_CLOUD_PASSWORD="$CLICKHOUSE_CLOUD_PASSWORD" \ + bash baselines.sh 3000000 3 +``` + +For an intrinsic (CPU-bound) measurement without the WAN, point the proxy at a local backend +(`WSPROXY_BACKEND_HOST=127.0.0.1 WSPROXY_BACKEND_PORT= WSPROXY_BACKEND_SECURE=0`) and run +the same commands against `127.0.0.1`. + +## Interpreting results + +- **Over a WAN** the result is network-bound: the proxy pulls native lz4-compressed blocks over + the wire and converts at the edge, so it lands on par with `clickhouse-client` and well ahead + of plain (uncompressed) HTTP JSON. +- **On loopback** the result is CPU-bound and exposes the proxy's conversion throughput. The + proxy formats output on a thread pool when `output_format_parallel_formatting` is enabled and + the format supports it; compare against `baselines.sh` (which formats in parallel by default) + and against a direct client run with `SETTINGS output_format_parallel_formatting=0` to see the + single-threaded floor. diff --git a/programs/wsproxy/bench/baselines.sh b/programs/wsproxy/bench/baselines.sh new file mode 100755 index 000000000000..5074e7723c86 --- /dev/null +++ b/programs/wsproxy/bench/baselines.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Baseline fetch timings for a JSONCompactEachRow result, matched to the proxy's +# output (unquoted 64-bit ints) so byte counts are identical across paths. +set -uo pipefail +ROWS="${1:-1000000}" +ITERS="${2:-5}" +CH_HOST="$CLICKHOUSE_CLOUD_HOST"; CH_PW="$CLICKHOUSE_CLOUD_PASSWORD" +Q="SELECT number AS n, number*2 AS d, toString(number) AS s FROM numbers(${ROWS})" +S="SETTINGS output_format_json_quote_64bit_integers=0" +FQ="$Q $S FORMAT JSONCompactEachRow" + +median() { printf '%s\n' "$@" | sort -n | awk '{a[NR]=$1} END{print a[int((NR+1)/2)]}'; } + +# Run "$@" with up to 3 retries on transient failure; echoes stdout byte count. +retry_bytes() { local n=0; while :; do b=$("$@" | wc -c) && [ "$b" -gt 100 ] && { echo "$b"; return 0; }; n=$((n+1)); [ "$n" -ge 3 ] && { echo 0; return 1; }; sleep 2; done; } +nat() { clickhouse-client --host "$CH_HOST" --secure --user default --password "$CH_PW" --query "$FQ"; } +httpp() { curl -sS "https://${CH_HOST}:8443/" -H "X-ClickHouse-User: default" -H "X-ClickHouse-Key: ${CH_PW}" --data-binary "$FQ"; } +httpz() { curl -sS --compressed "https://${CH_HOST}:8443/?enable_http_compression=1" -H "X-ClickHouse-User: default" -H "X-ClickHouse-Key: ${CH_PW}" --data-binary "$FQ"; } + +echo "### rows=$ROWS iters=$ITERS" + +clickhouse-client --host "$CH_HOST" --secure --user default --password "$CH_PW" --query "SELECT 1" >/dev/null 2>&1 || true + +run_path() { # $1=label $2=fn + local label="$1" fn="$2" bytes=0 t=() s e b + for _ in $(seq "$ITERS"); do + s=$(date +%s.%N); b=$(retry_bytes "$fn"); e=$(date +%s.%N) + t+=("$(echo "$e - $s" | bc)"); bytes=$b + done + printf '%-40s median=%ss bytes=%s\n' "$label" "$(median "${t[@]}")" "$bytes" +} + +run_path "NATIVE (native lz4 -> client format)" nat +run_path "HTTP (server format, plain JSON)" httpp +run_path "HTTP-GZ (server format, gzip WAN)" httpz diff --git a/programs/wsproxy/bench/bench.mjs b/programs/wsproxy/bench/bench.mjs new file mode 100644 index 000000000000..43e24c601600 --- /dev/null +++ b/programs/wsproxy/bench/bench.mjs @@ -0,0 +1,54 @@ +// Benchmark: fetch a query result through the wsproxy and measure wall time + +// bytes received. Usage: node bench.mjs +const ROWS = Number(process.argv[2] ?? 1_000_000); +const FORMAT = process.argv[3] ?? "JSONCompactEachRow"; +const ITERS = Number(process.argv[4] ?? 5); +const URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; + +const QUERY = `SELECT number AS n, number*2 AS d, toString(number) AS s FROM numbers(${ROWS})`; + +function once() { + return new Promise((resolve, reject) => { + const ws = new WebSocket(`${URL}/?format=${FORMAT}`); + ws.binaryType = "arraybuffer"; + let bytes = 0; + let t0 = 0; + ws.addEventListener("open", () => { + t0 = performance.now(); + ws.send(QUERY); + }); + ws.addEventListener("message", (ev) => { + if (typeof ev.data === "string") { + const msg = JSON.parse(ev.data); + if (msg.event === "end" || msg.event === "error" || msg.event === "cancelled") { + const ms = performance.now() - t0; + ws.close(); + if (msg.event === "end") resolve({ ms, bytes }); + else reject(new Error(`${msg.event}: ${msg.message ?? ""}`)); + } + } else { + bytes += ev.data.byteLength; + } + }); + ws.addEventListener("error", (e) => reject(new Error(`ws error: ${e?.message ?? e}`))); + }); +} + +const results = []; +// One warm-up iteration (not counted). +await once(); +for (let i = 0; i < ITERS; i++) { + const r = await once(); + results.push(r); + console.log(` iter ${i + 1}: ${r.ms.toFixed(0)} ms, ${(r.bytes / 1e6).toFixed(1)} MB`); +} +const times = results.map((r) => r.ms).sort((a, b) => a - b); +const median = times[Math.floor(times.length / 2)]; +const min = times[0]; +const bytes = results[0].bytes; +const mbps = bytes / 1e6 / (min / 1000); // best-case throughput +const rowsPerSec = ROWS / (min / 1000); +console.log( + `PROXY ${FORMAT} rows=${ROWS}: min=${min.toFixed(0)}ms median=${median.toFixed(0)}ms ` + + `bytes=${(bytes / 1e6).toFixed(1)}MB throughput=${mbps.toFixed(0)}MB/s (${(rowsPerSec / 1e6).toFixed(1)}M rows/s)`, +); diff --git a/programs/wsproxy/bench/tcp_ceiling.mjs b/programs/wsproxy/bench/tcp_ceiling.mjs new file mode 100644 index 000000000000..600ee3ef742f --- /dev/null +++ b/programs/wsproxy/bench/tcp_ceiling.mjs @@ -0,0 +1,25 @@ +import net from "node:net"; +const BYTES = 962_000_000; +const buf = Buffer.allocUnsafe(1 << 20); // 1MB chunks +const server = net.createServer((sock) => { + let sent = 0; + const pump = () => { + while (sent < BYTES) { + sent += buf.length; + if (!sock.write(buf)) { sock.once("drain", pump); return; } + } + sock.end(); + }; + pump(); +}); +server.listen(0, "127.0.0.1", () => { + const port = server.address().port; + let got = 0, t0 = performance.now(); + const c = net.connect(port, "127.0.0.1"); + c.on("data", (d) => { got += d.length; }); + c.on("end", () => { + const ms = performance.now() - t0; + console.log(`raw TCP recv: ${(got/1e6).toFixed(0)}MB in ${ms.toFixed(0)}ms = ${(got/1e6/(ms/1000)).toFixed(0)} MB/s`); + server.close(); + }); +}); From 082fcb6ce2cb856b7e5991f2f6512ffb676332f6 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Mon, 13 Jul 2026 11:56:45 +0200 Subject: [PATCH 36/46] Add opt-in parallel output formatting (?parallel=1) for SELECT executeSelect uses FormatFactory::getOutputFormatParallelIfPossible when ?parallel=1 and flow control is off, spreading output formatting across a thread pool. The parallel formatter runs a collector thread that writes result frames concurrently with the session thread's progress/log/control frames, so all whole-frame sends are serialized behind a new ProxySession::ws_write_mutex (passed to WriteBufferToWebSocket and to the cancel-poll applyControlFrame; sendControlEvent/sendBlockEvent take it too; WriteBufferToWebSocket::broken is now atomic). Mutually exclusive with ?flow, whose credit gate reads the client socket on the session thread (flow wins when both are set). Trade-off: coarser (batched) result frames; delivered bytes are unchanged. Also drop a per-frame alloc+copy in sendWebSocketFrame: with whole-frame sends serialized by the caller, the header and payload can be written directly (two sendAllBytes) instead of copied into a combined buffer, which for large result frames was a significant memcpy. Loopback JSONCompactEachRow benchmarks: ~1.6-1.7x (10M rows 531->333ms, 30M 1658->972ms), reaching the cheap-format floor; the remaining serial native-receive/decompress and WS-send path is now the ceiling (~990 MB/s). Network-bound over a WAN, so no change there. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/ProxySession.cpp | 55 +++++++++++++++++++++++----- programs/wsproxy/ProxySession.h | 12 +++++- programs/wsproxy/WebSocketFrames.cpp | 12 +++--- programs/wsproxy/WsProxyHandler.cpp | 8 +++- 4 files changed, 70 insertions(+), 17 deletions(-) diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp index 336a578d1bfd..35f97f9d6c15 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/programs/wsproxy/ProxySession.cpp @@ -16,6 +16,9 @@ #include #include +#include +#include + #include #include #include @@ -166,7 +169,10 @@ struct FlowControl /// Apply one client control frame to the flow state (also answers pings). Used /// both by the between-packets poll and by the credit gate, so `next`/`pause`/ /// `resume`/close are handled consistently wherever the client frame is read. -void applyControlFrame(Poco::Net::StreamSocket & socket, const WebSocketFrame & frame, FlowControl & fc) +/// `write_mutex` (if given) serializes the Pong send against other socket writers +/// (the parallel-format collector thread). +void applyControlFrame( + Poco::Net::StreamSocket & socket, const WebSocketFrame & frame, FlowControl & fc, std::mutex * write_mutex = nullptr) { if (!frame.valid || frame.opcode == Opcode::Close) { @@ -175,6 +181,9 @@ void applyControlFrame(Poco::Net::StreamSocket & socket, const WebSocketFrame & } if (frame.opcode == Opcode::Ping) { + std::unique_lock lock; + if (write_mutex) + lock = std::unique_lock(*write_mutex); sendWebSocketFrame(socket, Opcode::Pong, frame.payload.data(), frame.payload.size()); return; } @@ -212,8 +221,11 @@ class WriteBufferToWebSocket : public BufferWithOwnMemory { public: explicit WriteBufferToWebSocket( - Poco::Net::StreamSocket & socket_, FlowControl * flow_ = nullptr, size_t size = DBMS_DEFAULT_BUFFER_SIZE) - : BufferWithOwnMemory(size), socket(socket_), flow(flow_) + Poco::Net::StreamSocket & socket_, + FlowControl * flow_ = nullptr, + std::mutex * write_mutex_ = nullptr, + size_t size = DBMS_DEFAULT_BUFFER_SIZE) + : BufferWithOwnMemory(size), socket(socket_), flow(flow_), write_mutex(write_mutex_) { } @@ -245,7 +257,7 @@ class WriteBufferToWebSocket : public BufferWithOwnMemory flow->client_gone = true; break; } - applyControlFrame(socket, frame, *flow); + applyControlFrame(socket, frame, *flow, write_mutex); } if (flow->client_gone) { @@ -256,6 +268,11 @@ class WriteBufferToWebSocket : public BufferWithOwnMemory try { + /// The send must be atomic against the session thread's control-frame + /// writes (used when parallel formatting runs this on a collector thread). + std::unique_lock lock; + if (write_mutex) + lock = std::unique_lock(*write_mutex); sendWebSocketBinary(socket, working_buffer.begin(), offset()); } catch (...) @@ -270,7 +287,8 @@ class WriteBufferToWebSocket : public BufferWithOwnMemory Poco::Net::StreamSocket & socket; FlowControl * flow; - bool broken = false; + std::mutex * write_mutex; + std::atomic broken = false; }; /// A `ReadBuffer` that yields the payloads of incoming WebSocket binary frames, @@ -395,7 +413,8 @@ ProxySession::ProxySession( String logs_level_, bool flow_enabled_, Int64 flow_initial_credit_, - bool parse_enabled_) + bool parse_enabled_, + bool parallel_enabled_) : socket(socket_) , context(std::move(context_)) , backend(std::move(backend_)) @@ -404,6 +423,7 @@ ProxySession::ProxySession( , flow_enabled(flow_enabled_) , flow_initial_credit(flow_initial_credit_) , parse_enabled(parse_enabled_) + , parallel_enabled(parallel_enabled_) { } @@ -413,6 +433,7 @@ void ProxySession::sendControlEvent(const String & event, const String & message if (!message.empty()) json += R"(,"message":")" + escapeJSON(message) + "\""; json += "}"; + std::lock_guard lock(ws_write_mutex); sendWebSocketText(socket, json); } @@ -450,6 +471,7 @@ void ProxySession::sendBlockEvent(const String & event, const Block & block) start = newline + 1; } + std::lock_guard lock(ws_write_mutex); sendWebSocketText(socket, R"({"event":")" + event + R"(","rows":[)" + rows_array + "]}"); } @@ -561,7 +583,16 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) fc.enabled = flow_enabled; fc.credit = flow_initial_credit; - WriteBufferToWebSocket out_buf(socket, &fc); + /// Parallel output formatting (opt-in, ?parallel=1) spreads the otherwise single-threaded + /// (~500 MB/s) format work across cores. It runs a collector thread that writes result frames, + /// so all frame sends are serialized behind `ws_write_mutex`. Trade-off: result frames are + /// coarser (blocks are batched, not one frame per block). It is incompatible with flow control + /// (whose credit gate reads the client socket, which must stay on the single session thread), + /// so flow control wins when both are requested. `getOutputFormatParallelIfPossible` still + /// honours the `output_format_parallel_formatting` setting and the format's own support. + const bool use_parallel = parallel_enabled && !flow_enabled; + + WriteBufferToWebSocket out_buf(socket, &fc, &ws_write_mutex); OutputFormatPtr output; bool cancelled = false; bool client_gone = false; @@ -618,7 +649,7 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) /// Apply the client frame: flow-control grants (next/pause/resume) /// update the shared FlowControl; a Close/broken read means the /// client is gone -> cancel the backend query and drain. - applyControlFrame(socket, frame, fc); + applyControlFrame(socket, frame, fc, &ws_write_mutex); if (fc.client_gone) note_client_gone(); } @@ -632,7 +663,12 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) if (client_gone) break; /// Draining after cancel; discard. if (!output) - output = FormatFactory::instance().getOutputFormat(format, out_buf, packet.block.cloneEmpty(), context); + { + auto & factory = FormatFactory::instance(); + const Block header = packet.block.cloneEmpty(); + output = use_parallel ? factory.getOutputFormatParallelIfPossible(format, out_buf, header, context) + : factory.getOutputFormat(format, out_buf, header, context); + } if (packet.block.rows() > 0) { output->write(packet.block); @@ -685,6 +721,7 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) + R"(,"total_rows_to_read":)" + std::to_string(total_rows_to_read) + "}"; try { + std::lock_guard lock(ws_write_mutex); sendWebSocketText(socket, json); } catch (...) diff --git a/programs/wsproxy/ProxySession.h b/programs/wsproxy/ProxySession.h index 77a5cd1f24d9..45975e7b345d 100644 --- a/programs/wsproxy/ProxySession.h +++ b/programs/wsproxy/ProxySession.h @@ -1,5 +1,6 @@ #pragma once +#include #include #include @@ -42,7 +43,8 @@ class ProxySession String logs_level_ = "", bool flow_enabled_ = false, Int64 flow_initial_credit_ = 0, - bool parse_enabled_ = false); + bool parse_enabled_ = false, + bool parallel_enabled_ = false); void run(); @@ -77,6 +79,14 @@ class ProxySession bool flow_enabled; /// Opt-in credit/window flow control for the SELECT push direction. Int64 flow_initial_credit; /// Starting frame credit when flow control is enabled. bool parse_enabled; /// Opt-in (?parse=1): parse SQL to auto-route inserts and report the query kind. + /// Opt-in (?parallel=1): format SELECT output on a thread pool for higher conversion + /// throughput, at the cost of coarser (batched) result frames. Ignored when flow control is on. + bool parallel_enabled; + + /// Serializes complete WebSocket frame sends. With parallel output formatting the format's + /// collector thread writes result frames while the session thread pushes progress/log/control + /// frames; without this lock the two writers would interleave bytes and corrupt the stream. + std::mutex ws_write_mutex; }; } diff --git a/programs/wsproxy/WebSocketFrames.cpp b/programs/wsproxy/WebSocketFrames.cpp index f243e2f00e32..37d0b4c1f154 100644 --- a/programs/wsproxy/WebSocketFrames.cpp +++ b/programs/wsproxy/WebSocketFrames.cpp @@ -105,13 +105,13 @@ void sendWebSocketFrame(Poco::Net::StreamSocket & socket, uint8_t opcode, const header_len = 10; } - /// Combine header and payload into a single send to avoid partial frame writes. - String buf; - buf.reserve(header_len + len); - buf.append(reinterpret_cast(header), header_len); + /// Send the header then the payload. The caller serializes whole-frame sends + /// (see ProxySession's write mutex), so these two writes cannot interleave with + /// another frame; sending the payload directly avoids copying it into a + /// combined buffer (which, for large result frames, is a significant memcpy). + sendAllBytes(socket, reinterpret_cast(header), header_len); if (len > 0) - buf.append(data, len); - sendAllBytes(socket, buf.data(), buf.size()); + sendAllBytes(socket, data, len); } void sendWebSocketBinary(Poco::Net::StreamSocket & socket, const char * data, size_t len) diff --git a/programs/wsproxy/WsProxyHandler.cpp b/programs/wsproxy/WsProxyHandler.cpp index 6abf61cfdc1e..d7bb278d6839 100644 --- a/programs/wsproxy/WsProxyHandler.cpp +++ b/programs/wsproxy/WsProxyHandler.cpp @@ -222,6 +222,11 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp /// the proxy does not parse SQL unless the client explicitly asks it to. const String parse_param = queryParam(uri, "parse", "", log); const bool parse_enabled = parse_param == "1" || parse_param == "true"; + /// Opt-in parallel output formatting: `?parallel=1` formats SELECT output on a thread pool + /// for higher conversion throughput, at the cost of coarser (batched) result frames. Ignored + /// when flow control is on. Default off preserves fine-grained one-frame-per-block streaming. + const String parallel_param = queryParam(uri, "parallel", "", log); + const bool parallel_enabled = parallel_param == "1" || parallel_param == "true"; /// Credential pass-through: resolve this session's backend user/password from /// the request (never mutate the shared default `backend`). @@ -246,7 +251,8 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp socket.setSendTimeout(Poco::Timespan(send_timeout_sec * 1'000'000)); /// microseconds ProxySession session( - socket, context, session_backend, out_format, logs_level, flow_enabled, flow_credit, parse_enabled); + socket, context, session_backend, out_format, logs_level, flow_enabled, flow_credit, parse_enabled, + parallel_enabled); session.run(); LOG_DEBUG(log, "WebSocket session closed"); From c8659566efdfb965b49c4d474d210abb1f6636e5 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Mon, 13 Jul 2026 11:56:45 +0200 Subject: [PATCH 37/46] Tests, docs, and bench support for ?parallel=1 helpers.mjs: parallel session/url option. parallel.test.mjs: parallel output is byte-identical to the default path, delivers all rows, and propagates backend exceptions. bench.mjs: PARALLEL=1 env opts into ?parallel=1 and the summary now reports throughput (MB/s + rows/s). README/TODO document the opt-in and the benchmark findings. 112 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 16 ++++++ programs/wsproxy/bench/bench.mjs | 3 +- programs/wsproxy/tests/README.md | 8 +++ programs/wsproxy/tests/test/helpers.mjs | 7 +-- programs/wsproxy/tests/test/parallel.test.mjs | 51 +++++++++++++++++++ 5 files changed, 81 insertions(+), 4 deletions(-) create mode 100644 programs/wsproxy/tests/test/parallel.test.mjs diff --git a/TODO.md b/TODO.md index 481c4a8e22f8..f44010206748 100644 --- a/TODO.md +++ b/TODO.md @@ -349,6 +349,22 @@ unchanged for default clients — and a parse failure degrades to the plain-quer real errors), so a parser quirk can never wedge a statement. The explicit `{"cmd":"insert"}` message always wins over parsing. 109 tests green. +**Parallel output formatting (`?parallel=1`) — PROTOTYPE DONE.** Opt-in: `executeSelect` uses +`FormatFactory::getOutputFormatParallelIfPossible` when `?parallel=1` and flow control is off. Since +the parallel formatter runs a collector thread that writes result frames concurrently with the +session thread's progress/log/control frames, all whole-frame sends are serialized behind a new +`ProxySession::ws_write_mutex` (passed to `WriteBufferToWebSocket` and the cancel-poll +`applyControlFrame`; `sendControlEvent`/`sendBlockEvent` lock it too). Also removed a per-frame +alloc+copy in `sendWebSocketFrame` (send header then payload directly, safe under the lock). Trade-off: +coarser (batched) result frames, and it is mutually exclusive with `?flow` (whose credit gate reads +the client socket on the session thread). Benchmarks (loopback, `JSONCompactEachRow`): ~1.6-1.7× +(10M rows 531→333ms, 30M 1658→972ms), reaching the cheap-format (`RowBinary`) floor — i.e. formatting +is fully hidden and the remaining ceiling (~990 MB/s, ~1.8 cores) is the *serial* native +receive+lz4-decompress and WS send path. Irrelevant over a WAN (network-bound). Bytes identical to the +default path (parallel.test.mjs). 112 tests green. NB the earlier "single-threaded format is THE +bottleneck" claim was too strong: format is only ~40% of the loopback pipeline; receive+send are the +rest and are still serial (future work: dedicated send thread / faster decompress). + Coverage still thin / future pushes: slow-loris/partial-frame timeouts, TLS, protocol-revision skew across older server versions (need old server binaries). Productionization track (separate): auth (done), proxy→backend TLS, config file, resource limits, graceful drain, `BaseDaemon`, diff --git a/programs/wsproxy/bench/bench.mjs b/programs/wsproxy/bench/bench.mjs index 43e24c601600..9f3c5bacada4 100644 --- a/programs/wsproxy/bench/bench.mjs +++ b/programs/wsproxy/bench/bench.mjs @@ -4,12 +4,13 @@ const ROWS = Number(process.argv[2] ?? 1_000_000); const FORMAT = process.argv[3] ?? "JSONCompactEachRow"; const ITERS = Number(process.argv[4] ?? 5); const URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; +const PARALLEL = process.env.PARALLEL === "1"; // opt into ?parallel=1 (parallel output formatting) const QUERY = `SELECT number AS n, number*2 AS d, toString(number) AS s FROM numbers(${ROWS})`; function once() { return new Promise((resolve, reject) => { - const ws = new WebSocket(`${URL}/?format=${FORMAT}`); + const ws = new WebSocket(`${URL}/?format=${FORMAT}${PARALLEL ? "¶llel=1" : ""}`); ws.binaryType = "arraybuffer"; let bytes = 0; let t0 = 0; diff --git a/programs/wsproxy/tests/README.md b/programs/wsproxy/tests/README.md index 0f9f4ed01025..dac462904058 100644 --- a/programs/wsproxy/tests/README.md +++ b/programs/wsproxy/tests/README.md @@ -39,6 +39,14 @@ in the output format chosen via the WS URL `?format=` parameter. Mid-query progr outcome (`end` / `error` / `cancelled`) arrive as JSON **text** frames. This plain-query path covers SELECT, `INSERT … SELECT`, inline `INSERT … VALUES`, and DDL. +**Opt-in parallel output formatting (`?parallel=1`).** By default the proxy formats SELECT output +on the single session thread (one WS frame per result block, fine-grained streaming). With +`?parallel=1` it formats on a thread pool (`getOutputFormatParallelIfPossible`), which on a fast link +raises conversion throughput (~1.6× for `JSONCompactEachRow` in local benchmarks) at the cost of +coarser, batched result frames. It is mutually exclusive with flow control (`?flow`), which reads the +client socket on the session thread; flow control wins when both are set. Delivered bytes are +identical either way. + **The proxy never parses SQL** — routing is by message kind. A **streamed data INSERT** (where the client sends the rows, e.g. for edge format conversion) is opted into with a control message `{"cmd":"insert","query":"INSERT INTO t FORMAT JSONEachRow","format":"JSONEachRow"}`, after which the diff --git a/programs/wsproxy/tests/test/helpers.mjs b/programs/wsproxy/tests/test/helpers.mjs index 086c032d968a..87f54c905847 100644 --- a/programs/wsproxy/tests/test/helpers.mjs +++ b/programs/wsproxy/tests/test/helpers.mjs @@ -18,7 +18,7 @@ export const PROXY_URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; /** Build the WS URL for a format and options ({ logs, path, baseUrl, user, password, flow, parse }). */ export function urlFor( format = "JSONEachRow", - { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow, parse = false } = {}, + { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow, parse = false, parallel = false } = {}, ) { const params = new URLSearchParams({ format }); if (logs) params.set("logs", logs); @@ -26,6 +26,7 @@ export function urlFor( if (password) params.set("password", password); if (flow !== undefined) params.set("flow", String(flow)); // opt-in credit flow control if (parse) params.set("parse", "1"); // opt-in SQL parsing (auto-route inserts, report kind) + if (parallel) params.set("parallel", "1"); // opt-in parallel output formatting (coarser frames) return `${baseUrl}${path}?${params}`; } @@ -37,9 +38,9 @@ export function urlFor( export class Session { constructor( format = "JSONEachRow", - { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow, parse = false } = {}, + { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow, parse = false, parallel = false } = {}, ) { - this.ws = new WebSocket(urlFor(format, { logs, path, baseUrl, user, password, flow, parse })); + this.ws = new WebSocket(urlFor(format, { logs, path, baseUrl, user, password, flow, parse, parallel })); this.ws.binaryType = "arraybuffer"; this._queue = []; this._waiters = []; diff --git a/programs/wsproxy/tests/test/parallel.test.mjs b/programs/wsproxy/tests/test/parallel.test.mjs new file mode 100644 index 000000000000..b10bc4fe4a98 --- /dev/null +++ b/programs/wsproxy/tests/test/parallel.test.mjs @@ -0,0 +1,51 @@ +import { describe, it, expect } from "vitest"; +import { Session } from "./helpers.mjs"; + +// Opt-in parallel output formatting (`?parallel=1`). Formats SELECT output on a +// thread pool for throughput; result frames are coarser (blocks are batched) but +// the delivered bytes must be identical to the default single-threaded path. +describe("parallel output formatting (?parallel=1)", () => { + const QUERY = "SELECT number AS n, toString(number) AS s FROM numbers(200000)"; + + async function fetchText(opts) { + const s = new Session("JSONCompactEachRow", opts); + try { + const { text, control } = await s.run(QUERY); + expect(control.event).toBe("end"); + return text; + } finally { + s.close(); + } + } + + it("produces byte-identical output to the default path", async () => { + const [plain, parallel] = await Promise.all([fetchText({}), fetchText({ parallel: true })]); + expect(parallel.length).toBe(plain.length); + expect(parallel).toBe(plain); + }); + + it("delivers all rows", async () => { + const s = new Session("JSONCompactEachRow", { parallel: true }); + try { + const { text, control } = await s.run(QUERY); + expect(control.event).toBe("end"); + const lines = text.split("\n").filter((l) => l.length > 0); + expect(lines.length).toBe(200000); + expect(JSON.parse(lines[0])).toEqual([0, "0"]); + expect(JSON.parse(lines[199999])).toEqual([199999, "199999"]); + } finally { + s.close(); + } + }); + + it("propagates a backend exception as an error event", async () => { + const s = new Session("JSONCompactEachRow", { parallel: true }); + try { + const { control } = await s.run("SELECT throwIf(number = 5) FROM numbers(100)"); + expect(control.event).toBe("error"); + expect(control.message).toBeTruthy(); + } finally { + s.close(); + } + }); +}); From cd8d008ff4901b63be996bda211dae66775492ee Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Mon, 13 Jul 2026 12:41:41 +0200 Subject: [PATCH 38/46] Make backend native compression codec configurable (WSPROXY_BACKEND_COMPRESSION) Adds WSPROXY_BACKEND_COMPRESSION = lz4 (default) | zstd | none, applied via the query's network_compression_method (the server compresses result blocks with it) and, for none, the connection compression flag. The proxy's Connection decompresses whatever codec the server sends (self-describing), so this only trades wire bytes for backend/proxy CPU. Motivation: over a bandwidth-limited WAN, gzipped HTTP beat the proxy on highly compressible data because the proxy's default lz4 trades ratio for speed (36.5 MB on the wire for the 3M-row benchmark vs gzip's 21.3 MB). With zstd, columnar native compresses far better than row JSON (~7-10 MB, 2-3x smaller than gzip), and the end-to-end cloud fetch drops ~5.0s -> ~2.5s (2x), beating gzip-HTTP (~3.5s) while keeping edge conversion and native signalling. On realistic/high-entropy data columnar native beats row JSON even with lz4. compression.test.mjs verifies lz4/zstd/none deliver byte-identical output. 113 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 15 +++++++ programs/wsproxy/ProxySession.cpp | 11 ++++- programs/wsproxy/ProxySession.h | 4 ++ programs/wsproxy/WsProxy.cpp | 3 ++ programs/wsproxy/bench/README.md | 11 +++-- .../wsproxy/tests/test/compression.test.mjs | 42 +++++++++++++++++++ programs/wsproxy/tests/test/proc.mjs | 2 + 7 files changed, 84 insertions(+), 4 deletions(-) create mode 100644 programs/wsproxy/tests/test/compression.test.mjs diff --git a/TODO.md b/TODO.md index f44010206748..00ec9924504b 100644 --- a/TODO.md +++ b/TODO.md @@ -365,6 +365,21 @@ default path (parallel.test.mjs). 112 tests green. NB the earlier "single-thread bottleneck" claim was too strong: format is only ~40% of the loopback pipeline; receive+send are the rest and are still serial (future work: dedicated send thread / faster decompress). +**Why gzip-HTTP beat the proxy over the WAN — ANSWERED + fixed.** The WAN is bandwidth-bound (~8.5 +MB/s). The proxy's native transport used **LZ4**, which trades ratio for speed: for the (highly +compressible) benchmark data, LZ4 put 36.5 MB on the wire vs gzip-on-JSON's 21.3 MB, so gzip-HTTP was +faster purely on bytes. Fix: make the backend codec configurable via `WSPROXY_BACKEND_COMPRESSION` +(`lz4` default | `zstd` | `none`); the server compresses result blocks with the client's +`network_compression_method`, and the proxy's `Connection` decompresses any codec (self-describing). +With **ZSTD**, columnar native compresses far better than row JSON: 3M-row wire bytes drop to ~7-10 MB +(2-3× smaller than gzip's 21 MB), and end-to-end cloud time drops 5.0s → 2.5s (2×), now *beating* +gzip-HTTP (3.5s). **When gzip wins:** only bandwidth-bound link AND highly compressible data (the +sequential-integer benchmark is the worst case for the proxy). For realistic/high-entropy data, +columnar native beats row JSON even with LZ4 (123 vs 140 MB at 3M) and ZSTD wins outright. Codec is +delivered-byte-identical (compression.test.mjs). 113 tests green. Recommend ZSTD default for WAN +deployments (costs backend CPU to compress — but that's cheaper than the JSON formatting we already +moved to the edge). + Coverage still thin / future pushes: slow-loris/partial-frame timeouts, TLS, protocol-revision skew across older server versions (need old server binaries). Productionization track (separate): auth (done), proxy→backend TLS, config file, resource limits, graceful drain, `BaseDaemon`, diff --git a/programs/wsproxy/ProxySession.cpp b/programs/wsproxy/ProxySession.cpp index 35f97f9d6c15..2415cb5870e5 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/programs/wsproxy/ProxySession.cpp @@ -484,6 +484,15 @@ void ProxySession::sendBackendQuery(Connection & connection, const String & quer if (!logs_level.empty()) settings.set("send_logs_level", logs_level); + /// The server compresses the result blocks it sends using the client's + /// `network_compression_method`. On a bandwidth-limited WAN this codec choice + /// dominates end-to-end time; ZSTD's higher ratio (columnar native compresses + /// far better than row JSON) makes the proxy beat gzipped HTTP. Empty = leave + /// the backend/connection default (LZ4). "none" disables block compression via + /// the connection's compression flag, so nothing to set here. + if (backend.compression_method != "none" && !backend.compression_method.empty()) + settings.set("network_compression_method", backend.compression_method); + auto timeouts = ConnectionTimeouts::getTCPTimeoutsWithoutFailover(settings); ClientInfo client_info; @@ -904,7 +913,7 @@ void ProxySession::run() /* cluster_ */ "", /* cluster_secret_ */ "", /* client_name_ */ "clickhouse-wsproxy", - Protocol::Compression::Enable, + backend.compression_method == "none" ? Protocol::Compression::Disable : Protocol::Compression::Enable, backend.secure ? Protocol::Secure::Enable : Protocol::Secure::Disable, /* tls_sni_override_ */ "", /* bind_host_ */ ""); diff --git a/programs/wsproxy/ProxySession.h b/programs/wsproxy/ProxySession.h index 45975e7b345d..000a2d8af211 100644 --- a/programs/wsproxy/ProxySession.h +++ b/programs/wsproxy/ProxySession.h @@ -22,6 +22,10 @@ struct BackendParams String password; String database; bool secure = false; /// Connect to the backend over TLS (native secure protocol). + /// Native-protocol compression codec for backend->proxy result blocks: "lz4" (fast, default), + /// "zstd" (higher ratio — fewer bytes on a bandwidth-limited WAN, at more backend CPU), or + /// "none". The proxy decompresses whatever the server sends regardless (codec is self-describing). + String compression_method = "lz4"; }; /// Drives one WebSocket session against a backend ClickHouse server over the diff --git a/programs/wsproxy/WsProxy.cpp b/programs/wsproxy/WsProxy.cpp index 89aa6ac7938e..7bcc1f9f4405 100644 --- a/programs/wsproxy/WsProxy.cpp +++ b/programs/wsproxy/WsProxy.cpp @@ -97,6 +97,9 @@ BackendParams backendParamsFromEnv() params.password = envOr("WSPROXY_BACKEND_PASSWORD", params.password); params.database = envOr("WSPROXY_BACKEND_DATABASE", params.database); params.secure = envBool("WSPROXY_BACKEND_SECURE"); + /// Native result-block compression codec: lz4 (default) | zstd | none. ZSTD trades backend + /// CPU for far fewer bytes on the wire — recommended for bandwidth-limited (WAN) deployments. + params.compression_method = envOr("WSPROXY_BACKEND_COMPRESSION", params.compression_method); return params; } diff --git a/programs/wsproxy/bench/README.md b/programs/wsproxy/bench/README.md index e184490d0481..703964512138 100644 --- a/programs/wsproxy/bench/README.md +++ b/programs/wsproxy/bench/README.md @@ -58,9 +58,14 @@ the same commands against `127.0.0.1`. ## Interpreting results -- **Over a WAN** the result is network-bound: the proxy pulls native lz4-compressed blocks over - the wire and converts at the edge, so it lands on par with `clickhouse-client` and well ahead - of plain (uncompressed) HTTP JSON. +- **Over a WAN** the result is network-bound, so the codec choice dominates. The proxy pulls + compressed native blocks and converts at the edge. With the default **lz4** it can lose to gzipped + HTTP on *highly compressible* data (lz4 trades ratio for speed: e.g. 36.5 MB on the wire vs gzip's + 21.3 MB for the sequential-integer query). Set `WSPROXY_BACKEND_COMPRESSION=zstd` and columnar + native drops to ~7-10 MB — 2-3× smaller than gzip-on-JSON — so the proxy wins outright (3M-row + cloud fetch ~2.5s vs gzip-HTTP ~3.5s). On realistic/high-entropy data columnar native beats row + JSON even with lz4. Compare with `baselines.sh` and re-run the proxy with different + `WSPROXY_BACKEND_COMPRESSION` values. - **On loopback** the result is CPU-bound and exposes the proxy's conversion throughput. The proxy formats output on a thread pool when `output_format_parallel_formatting` is enabled and the format supports it; compare against `baselines.sh` (which formats in parallel by default) diff --git a/programs/wsproxy/tests/test/compression.test.mjs b/programs/wsproxy/tests/test/compression.test.mjs new file mode 100644 index 000000000000..5dbdb01ed329 --- /dev/null +++ b/programs/wsproxy/tests/test/compression.test.mjs @@ -0,0 +1,42 @@ +import { describe, it, expect, afterAll } from "vitest"; +import { spawnProxy } from "./proc.mjs"; +import { Session } from "./helpers.mjs"; + +// The native result-block codec (backend -> proxy) is configurable via +// WSPROXY_BACKEND_COMPRESSION (lz4 default | zstd | none). It only changes bytes +// on the wire and backend/proxy CPU; the data delivered to the client must be +// byte-identical across codecs. On a bandwidth-limited WAN, zstd is much smaller +// than lz4 (columnar native compresses far better than row JSON), which is what +// lets the proxy beat gzipped HTTP — see programs/wsproxy/bench/. +describe("backend compression codec", () => { + const BACKEND_PORT = 19000; + const QUERY = "SELECT number AS n, number*2 AS d, toString(number) AS s FROM numbers(50000)"; + const proxies = []; + + afterAll(() => proxies.forEach((p) => p.stop())); + + async function fetchVia(compression, listenPort) { + const proxy = await spawnProxy({ listenPort, backendPort: BACKEND_PORT, compression }); + proxies.push(proxy); + const s = new Session("JSONCompactEachRow", { baseUrl: proxy.url }); + try { + const { text, control } = await s.run(QUERY); + expect(control.event).toBe("end"); + return text; + } finally { + s.close(); + } + } + + it("delivers byte-identical output for lz4, zstd, and none", async () => { + const [lz4, zstd, none] = await Promise.all([ + fetchVia("lz4", 9031), + fetchVia("zstd", 9032), + fetchVia("none", 9033), + ]); + expect(zstd).toBe(lz4); + expect(none).toBe(lz4); + // sanity: the payload is non-trivial + expect(lz4.split("\n").filter((l) => l).length).toBe(50000); + }, 30000); +}); diff --git a/programs/wsproxy/tests/test/proc.mjs b/programs/wsproxy/tests/test/proc.mjs index d7fb6cfd0dab..8e0009a95b40 100644 --- a/programs/wsproxy/tests/test/proc.mjs +++ b/programs/wsproxy/tests/test/proc.mjs @@ -147,6 +147,7 @@ export async function spawnProxy({ secure = false, acceptInvalidCert = false, sendTimeoutSec, + compression, // native codec for backend->proxy blocks: "lz4" | "zstd" | "none" }) { const proc = spawn(WSPROXY_BIN, [], { stdio: "ignore", @@ -158,6 +159,7 @@ export async function spawnProxy({ ...(secure ? { WSPROXY_BACKEND_SECURE: "1" } : {}), ...(acceptInvalidCert ? { WSPROXY_BACKEND_ACCEPT_INVALID_CERT: "1" } : {}), ...(sendTimeoutSec ? { WSPROXY_CLIENT_SEND_TIMEOUT_SEC: String(sendTimeoutSec) } : {}), + ...(compression ? { WSPROXY_BACKEND_COMPRESSION: compression } : {}), }, }); await waitForPort(listenPort); From 0ff4ec9ef4f450780dbcda003aec016bc41b9d89 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Mon, 13 Jul 2026 13:59:35 +0200 Subject: [PATCH 39/46] Consolidate wsproxy status: summary, next steps, and known limitations Add a 'Current status' section (delivered features, key findings, ordered next steps, lowlights/risks) to the top of the design doc, and mark the original open questions resolved. No code change. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 80 insertions(+), 6 deletions(-) diff --git a/TODO.md b/TODO.md index 00ec9924504b..7723a9690c9d 100644 --- a/TODO.md +++ b/TODO.md @@ -40,6 +40,75 @@ A new native-protocol WebSocket proxy, built **in-tree**, that: - **drops** the entire CLI/REPL surface (`ClientBase`, Replxx, `Suggest`, progress bars, pager, history, autocomplete). +## Current status (summary) + +Branch `wsproxy-skeleton`, draft PR against the fork (`peter-leonov-ch/ClickHouse` #4, **not** +upstream). Working prototype, **113 Node/vitest integration tests green** (hermetic suite, spawns its +own backend + proxy). Benchmark harness committed under `programs/wsproxy/bench/`. + +### Delivered + +- **Standalone binary** `programs/wsproxy/clickhouse-wsproxy` (own `clickhouse_add_executable`, not in + multi-call dispatch; ~306 MB stripped — *separate ≠ smaller*, links `dbms`). +- **WS front door** — hand-rolled RFC 6455 framing (`WebSocketFrames.{h,cpp}`), HTTP upgrade handler. +- **Native bridge** (`ProxySession.{h,cpp}`), strict 1:1, thread-per-session: + - **SELECT** → output format at the edge → binary frames (one flush per block). + - **Streamed INSERT** via explicit `{"cmd":"insert",...}` control message (the proxy never parses SQL). + - **Mid-query push**: `progress` / `log` / `profile_events` as JSON text frames. + - **Cancel** by Close frame → `sendCancel` → drain. +- **Auth** — credential pass-through (Basic / `X-ClickHouse-*` / URL params / env default), eager + connect (bad creds → error + `1008` close before any query). +- **proxy → backend TLS** (`WSPROXY_BACKEND_SECURE`; dev-only accept-invalid-cert flag). +- **Opt-in knobs**: `?flow=N` (credit flow control for slow JS clients), `?parse=1` (parse to + auto-route inserts + report `{verb,kind}`), `?parallel=1` (parallel output formatting), + `WSPROXY_BACKEND_COMPRESSION` (`lz4`|`zstd`|`none`). +- **Backpressure**: TCP-driven + send timeout so a stalled client can't pin a handler thread. + +### Key findings (for the pitch) + +- **Edge conversion + compressed wire is the win.** WAN is bandwidth-bound; **ZSTD native ≈ 7-10 MB** + for the 3M-row benchmark vs **gzip-JSON 21 MB** vs **lz4 36 MB** — columnar+zstd beats row-JSON+gzip + 2-3×, and JSON-formatting CPU moves off the cloud to the sidecar. This is the rebuttal to "why not + just gzip the HTTP interface?". +- **Parallel formatting** gives ~1.6-1.7× locally but the format is only ~40% of the pipeline; the + serial native-receive/decompress + WS-send path (~990 MB/s) is the real ceiling. +- **Never parse SQL** (principle) — routing is by client message kind; parsing is strictly opt-in. + +### Next steps (roughly ordered) + +1. **Decide the default codec.** Recommend **ZSTD default for WAN** (it's the whole point); currently + defaults to `lz4` for safety. Trade-off: more backend CPU to compress (cheaper than the JSON work + we already moved off-cloud). +2. **Lift the serial throughput ceiling** (only if a same-region/LAN profile justifies it): dedicated + WS-send thread and/or parallel receive+decompress. Parallel *formatting* alone won't help further. +3. **Format-settings faithfulness** — let clients pass format settings (URL/query), fixing the + unquoted-`UInt64` precision gap for JS clients (see lowlights). +4. **Productionization track**: config file (replace the `WSPROXY_*` env sprawl), CA-based cert + verification (replace accept-invalid), resource limits + graceful drain, `BaseDaemon`, CI wiring. +5. **Robustness coverage**: protocol-revision skew vs older server binaries; slow-loris / partial-frame + timeouts. +6. **Docs/pitch**: update the team artifact's evidence table with the ZSTD row; README for the client + protocol (`{"cmd":"insert"}`, `?parse`/`?parallel`/`?flow`). + +### Lowlights / known limitations / risks + +- **64-bit ints render unquoted in JSON** — `JSON.parse` loses precision for large `UInt64`/`Int64`. + Faithfulness gap (no client control of format settings yet). See next-step 3. +- **`?parallel=1` caveats**: coarser (batched) frames; **mutually exclusive with `?flow`**; client-gone + detection lags mid-stream (the collector drops on broken, session drains to finalize — wasteful but + not a hang). +- **`WSPROXY_BACKEND_ACCEPT_INVALID_CERT` is dev-only** — disables backend cert verification. Needs + real CA verification before production. +- **Client → proxy TLS not implemented** (deprioritized: sidecar loopback / ingress-terminated). Browser + clients can't set headers → creds go in URL params (leak in logs unless the leg is TLS). +- **Big binary (~306 MB) + rebase treadmill** — must track `Connection` / `FormatFactory` API churn. +- **All config is env vars**; no config file, no resource limits, no graceful drain, not on `BaseDaemon`. +- **ZSTD costs backend CPU** to compress — a deliberate bytes-for-CPU trade; fine for WAN, reconsider on LAN. +- **A fully `pause()`d flow-control client** stops seeing progress/logs/`end` too (ordering; documented) — + guidance is to throttle with small `next()` credits, not an indefinite pause. +- Benchmarking gotcha: `clickhouse-client FORMAT Null` **short-circuits** (never transfers) — don't use + it as a transfer measure. + ## Design decisions ### Cut line — move it down a layer @@ -559,9 +628,14 @@ bounded read-ahead if prompt terminal-event delivery to a *fully paused* client ## Open questions -- ~~Separate-binary feasibility~~ — **RESOLVED (step 1):** clean via - `clickhouse_add_executable`, precedent is `BUILD_STANDALONE_KEEPER`. See plan step 1. -- **Auth model:** in-band first-frame auth like the web terminal, or pass-through of the - app's credentials to the backend `Connection`? -- **Backend target config:** fixed host/port from config/CLI (sidecar assumption), or per-session - target from the WS handshake? +All original open questions are resolved — see "Current status" for the live state and next steps. + +- ~~Separate-binary feasibility~~ — **RESOLVED (step 1):** clean via `clickhouse_add_executable`, + precedent is `BUILD_STANDALONE_KEEPER`. +- ~~Auth model~~ — **RESOLVED:** credential pass-through to the backend `Connection` (no in-band + first-frame auth reinvented); priority Basic → `X-ClickHouse-*` → URL params → env default. +- ~~Backend target config~~ — **RESOLVED:** fixed host/port from env (`WSPROXY_BACKEND_*`, sidecar + assumption); per-session *credentials* come from the handshake, the target does not. + +Remaining product decision (not an original open question): **should ZSTD be the default backend +codec?** (recommended for the WAN use case — see "Next steps"). From eab2e3345c50b1196a0daa57cdcb4811eaefc823 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Tue, 14 Jul 2026 21:06:02 +0200 Subject: [PATCH 40/46] Document Cloudflare (Containers + Workers) deployment for wsproxy Add programs/wsproxy/CLOUDFLARE.md: feasibility verdict, the stateless-Worker + stateful-container architecture, the verified 'a Container is a Durable Object' relationship, multi-container scaling via getRandom/getContainer (and the no-autoscaling-yet caveat), concrete work items (amd64 Linux build is the long pole), Cloudflare-specific frictions (thread-per-session memory, vCPU limits, egress IP allowlisting, cold starts, placement), the 1:1-sidecar -> shared-multi-tenant model shift, and why a pure-Worker approach is rejected. Linked from TODO.md next-steps and the tests/ and bench/ READMEs. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 3 + programs/wsproxy/CLOUDFLARE.md | 152 +++++++++++++++++++++++++++++++ programs/wsproxy/bench/README.md | 3 + programs/wsproxy/tests/README.md | 3 + 4 files changed, 161 insertions(+) create mode 100644 programs/wsproxy/CLOUDFLARE.md diff --git a/TODO.md b/TODO.md index 7723a9690c9d..9db4a51b77d5 100644 --- a/TODO.md +++ b/TODO.md @@ -89,6 +89,9 @@ own backend + proxy). Benchmark harness committed under `programs/wsproxy/bench/ timeouts. 6. **Docs/pitch**: update the team artifact's evidence table with the ZSTD row; README for the client protocol (`{"cmd":"insert"}`, `?parse`/`?parallel`/`?flow`). +7. **Cloudflare deployment** (Containers + stateless Workers) — feasibility, architecture, scaling, + and frictions are written up in `programs/wsproxy/CLOUDFLARE.md`. Long pole is the amd64 Linux + build; the Worker/DO glue is small. ### Lowlights / known limitations / risks diff --git a/programs/wsproxy/CLOUDFLARE.md b/programs/wsproxy/CLOUDFLARE.md new file mode 100644 index 000000000000..bc7e8f97f64d --- /dev/null +++ b/programs/wsproxy/CLOUDFLARE.md @@ -0,0 +1,152 @@ +# Deploying `clickhouse-wsproxy` on Cloudflare (Containers + Workers) + +Feasibility notes and a deployment sketch for running the proxy as a Cloudflare Container fronted +by stateless Workers. **Status: analysis only — nothing here is built yet.** See `TODO.md` (repo +root) for the proxy's overall status, and `programs/wsproxy/bench/` for the throughput/compression +numbers this design leans on. + +## Verdict + +**Moderate effort, strong fit.** The proxy is already container-shaped: a single self-contained +binary, configured entirely by `WSPROXY_*` env vars, holding **no local/disk state** (all session +state is in-memory per connection), speaking WebSocket in and the native protocol out. Running it in +a container needs essentially no code change; the work is the amd64 Linux build plus a thin +Worker/Durable Object glue layer. + +It is arguably a *better* showcase of the value prop than a k8s sidecar, because the Worker gives you +a programmable, DDoS-protected, auth-capable front door for free, and the compressed-native-to-region +hop is exactly what the pitch is about. + +## Architecture: stateless Worker + stateful container + +The proxy's stateless/stateful split maps cleanly onto the platform: + +- **Worker = stateless front door / router.** Terminates the client WSS at the true edge, does + auth/routing, and forwards the upgrade to a container. Holds no session state. +- **Container = the stateful proxy.** Runs the real binary, owns the per-session native `Connection` + and in-flight query, and does the format conversion. This *must* be a Container, not a Worker: the + value is the C++ `FormatFactory` + native protocol, which cannot realistically be reimplemented in + a Worker (JS/WASM) — that is the "clickhouse-cpp has no formats" problem again. +- **Compressed hop to the region.** The container opens the outbound native + **ZSTD** connection to + ClickHouse Cloud (`:9440`). Converting at the container and keeping the ClickHouse-region leg + compressed is the whole point (see the compression findings in `bench/README.md`). + +``` +client --WSS--> Worker (true edge PoP) --CF backbone--> Container (regional) --native+zstd (WAN)--> ClickHouse Cloud + stateless router stateful proxy = a Durable Object +``` + +### A Container *is* a Durable Object (verified) + +The `Container` class from the [`@cloudflare/containers`](https://github.com/cloudflare/containers) +library **extends `DurableObject`**. Cloudflare's docs state it directly: *"a request passes through +a Durable Object instance (the Container class extends a Durable Object class)."* Each container +instance is backed by **exactly one** DO instance — the DO is the programmable sidecar that owns the +container's lifecycle and routing; the container is the workload attached to it. + +Consequences: + +- Your container subclass gets a free per-instance place for coordination logic — warm-up, health, + draining, `alarm`-based idle handling, small storage — with no extra moving part. +- The DO is single-threaded JS, but it is **not** in the per-frame data path: the client WebSocket + rides the container's exposed port via the stub's `fetch`; the DO only manages lifecycle. So + "DO is single-threaded" is **not** a throughput bottleneck for the proxy — as long as you forward + to the container port and do not proxy bytes through DO JS. +- **A DO and its container may run in different locations** (placement is optimized for routing and + startup). This feeds the placement nuance below. + +## Running many proxy containers (scaling out) + +The proxy's sessions are **interchangeable** — each is just a fresh native `Connection`, a stateless +pool — which is exactly the case the platform's helpers target: + +- `getRandom(env.WSPROXY, N)` — picks a random instance out of `N`. The stateless load balancer, and + the right fit here: each client WS upgrade → random container → new backend connection. The + WebSocket then naturally pins to that instance for its lifetime. +- `getContainer(env.WSPROXY, name)` — addresses a *specific* named instance by ID (sticky/stateful + routing). Probably unneeded for the proxy, but available. +- Multiple instances = multiple DO IDs; the library manages spin-up, pulling pre-fetched images at + other locations for fast cold starts. + +The Worker glue is small: + +```js +import { getRandom } from "@cloudflare/containers"; + +export default { + async fetch(request, env) { + // Optional: authenticate here, or let creds pass through to the container -> ClickHouse. + const instance = getRandom(env.WSPROXY, N); // N interchangeable proxy containers + return instance.fetch(request); // forwards the WS upgrade to the container port + }, +}; +``` + +**The catch — no autoscaling yet.** You pick and manage `N` yourself (a fixed fleet, or your own +logic). Cloudflare says built-in autoscaling is planned but not available today. Capacity ≈ +`N × sessions-per-container`, so size `N` against a per-container session cap (see the thread-per- +session note below). Until autoscaling lands, elasticity is on you — over-provision a fixed `N`, or +build a small coordinator (itself a DO) that adjusts routing under load. + +## Concrete work items + +1. **x86_64 Linux build** *(the long pole).* Everything so far was built on macOS/arm64; Cloudflare + Containers are **amd64 Linux only**. ClickHouse builds amd64 Linux routinely (CI does), so this is + well-trodden — just not yet done for this binary. +2. **Dockerfile.** A slim/distroless base + the ~306 MB binary + env; expose the listen port + (`WSPROXY_PORT`), set a `CMD`. Image ≈ 400-500 MB (under the platform's GB-scale image limits — + verify current limits). +3. **Worker + Durable Object + `wrangler` config.** ~50-150 lines: accept the WS upgrade, `getRandom` + to a container, forward. Secrets (backend password) via Worker/container secrets → container env. +4. **Secrets & egress.** Map `WSPROXY_BACKEND_*` into the container; settle ClickHouse-side source-IP + policy (see egress gotcha). + +Rough estimate: **PoC in 1-2 days** (the Linux build dominates), **production-ready in 1-2 weeks** +once the frictions below are handled. + +## Frictions / gotchas (Cloudflare-specific) + +- **Thread-per-session vs small containers *(biggest one)*.** The proxy is thread-per-session blocking + IO with default ~8 MB stacks → ~100 sessions ≈ ~800 MB just in stacks, and container instances are + memory-capped. Sessions-per-container is therefore bounded; lean on `getRandom` across instances + and/or shrink the thread stack size (a small proxy change worth doing before serious fan-in). +- **CPU limits cap the conversion win.** Formatting is CPU-heavy and `?parallel` wants multiple cores, + but instance types meter vCPU. Right-size the instance; the parallel-formatting speedup is bounded + by the vCPU allotted. +- **Egress IP allowlisting.** The container's outbound to ClickHouse Cloud uses Cloudflare's + shared/dynamic egress IPs. If the service restricts source IPs, you need CH-side allow-all or + Cloudflare egress ranges — settle this early (security posture). +- **Cold starts.** Containers sleep on idle; the first connection pays container boot + proxy start + + backend handshake (seconds). Eager-connect adds to that. +- **Placement affects the compression benefit.** Containers run in *regional* locations, not literally + every edge PoP, and may not co-locate with their DO. A container near the ClickHouse region → short + compressed hop (smaller win); near the user → long compressed hop (bigger win). Less placement + control than k8s. +- **Keep the DO out of the per-frame data path** (forward to the container port; do not proxy frames + through DO JS). + +## Model shift: 1:1 sidecar → shared multi-tenant + +On k8s the proxy is a strict 1:1 sidecar (one proxy per app). On Cloudflare you would run a **shared +multi-tenant** pool (many sessions per container, the Worker fanning out across instances). The proxy +already supports this safely: it does **per-session credential pass-through** and keeps zero +cross-session state, so multi-tenant is fine as long as each session carries its own credentials. + +## Alternative considered — pure Worker (no container) + +Workers can open outbound TCP (`connect` from `cloudflare:sockets`), so in principle the proxy could +live entirely in a Worker. Rejected: it would require reimplementing the ClickHouse native protocol +**and** the full format matrix in JS/WASM. Compiling ClickHouse itself to WASM is not feasible. The +container is the right vehicle precisely because it runs the real C++ binary with full format +coverage. + +## Sources + +- [Lifecycle of a Container — architecture](https://developers.cloudflare.com/containers/platform-details/architecture/) +- [Scaling and Routing](https://developers.cloudflare.com/containers/platform-details/scaling-and-routing/) +- [`cloudflare/containers` library README](https://github.com/cloudflare/containers/blob/main/README.md) +- [Containers overview](https://developers.cloudflare.com/containers/) +- [Containers coming to Workers (announcement)](https://blog.cloudflare.com/cloudflare-containers-coming-2025/) + +*(Cloudflare Containers are a recently-GA product; exact limits and APIs move — verify image-size, +memory, vCPU, WS-to-container specifics, and egress ranges against the live docs before committing.)* diff --git a/programs/wsproxy/bench/README.md b/programs/wsproxy/bench/README.md index 703964512138..31a0f61bf198 100644 --- a/programs/wsproxy/bench/README.md +++ b/programs/wsproxy/bench/README.md @@ -66,6 +66,9 @@ the same commands against `127.0.0.1`. cloud fetch ~2.5s vs gzip-HTTP ~3.5s). On realistic/high-entropy data columnar native beats row JSON even with lz4. Compare with `baselines.sh` and re-run the proxy with different `WSPROXY_BACKEND_COMPRESSION` values. + +The edge-conversion + compressed-wire model these numbers support is what a Cloudflare deployment +would exploit — see [`../CLOUDFLARE.md`](../CLOUDFLARE.md). - **On loopback** the result is CPU-bound and exposes the proxy's conversion throughput. The proxy formats output on a thread pool when `output_format_parallel_formatting` is enabled and the format supports it; compare against `baselines.sh` (which formats in parallel by default) diff --git a/programs/wsproxy/tests/README.md b/programs/wsproxy/tests/README.md index dac462904058..6c6d4509f195 100644 --- a/programs/wsproxy/tests/README.md +++ b/programs/wsproxy/tests/README.md @@ -66,6 +66,9 @@ backend reports any real error). This is the only place the proxy parses SQL. See `test/helpers.mjs` for the small client wrapper (`Session`, `runQuery`, `backendScalar`). +For deploying the proxy on Cloudflare (Containers fronted by stateless Workers), see +[`../CLOUDFLARE.md`](../CLOUDFLARE.md). + ## Receive backpressure (important for large results) The proxy pushes result frames as fast as the connection allows. A plain **event-based From a2bda9bb46f08258d86116b6959599dfccf56383 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 15 Jul 2026 18:11:01 +0200 Subject: [PATCH 41/46] Document in-server WS port feasibility (alternative to the sidecar) Add programs/wsproxy/IN_SERVER.md: feasibility of a ws_port inside clickhouse-server. Verdict is technically moderate (mostly a port) because the server already ships WS framing + HTTP-upgrade + the port/handler-factory pattern, and LocalConnection shares the IServerConnection interface ProxySession already drives, so the handler is ProxySession against a LocalConnection. Covers what's reused vs new, two shapes (dedicated ws_port vs endpoint on http_port), a concrete file-touch list, and the strategic catch: it's a core diff on the cloud-release cycle and loses the edge/compression value, so it's a complementary upstream play, not a faster sidecar. Cross-linked from TODO.md and CLOUDFLARE.md. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 4 ++ programs/wsproxy/CLOUDFLARE.md | 3 +- programs/wsproxy/IN_SERVER.md | 103 +++++++++++++++++++++++++++++++++ 3 files changed, 109 insertions(+), 1 deletion(-) create mode 100644 programs/wsproxy/IN_SERVER.md diff --git a/TODO.md b/TODO.md index 9db4a51b77d5..0d4ace77852a 100644 --- a/TODO.md +++ b/TODO.md @@ -92,6 +92,10 @@ own backend + proxy). Benchmark harness committed under `programs/wsproxy/bench/ 7. **Cloudflare deployment** (Containers + stateless Workers) — feasibility, architecture, scaling, and frictions are written up in `programs/wsproxy/CLOUDFLARE.md`. Long pole is the amd64 Linux build; the Worker/DO glue is small. +8. **In-server WS port** (alternative home) — feasibility of a `ws_port` inside `clickhouse-server` + (bridge `ProxySession` to a `LocalConnection`) is written up in `programs/wsproxy/IN_SERVER.md`. + Technically moderate, but a core diff on the cloud-release cycle and it loses the edge/compression + value — a complementary upstream play, not a faster sidecar. ### Lowlights / known limitations / risks diff --git a/programs/wsproxy/CLOUDFLARE.md b/programs/wsproxy/CLOUDFLARE.md index bc7e8f97f64d..286379ef2a24 100644 --- a/programs/wsproxy/CLOUDFLARE.md +++ b/programs/wsproxy/CLOUDFLARE.md @@ -3,7 +3,8 @@ Feasibility notes and a deployment sketch for running the proxy as a Cloudflare Container fronted by stateless Workers. **Status: analysis only — nothing here is built yet.** See `TODO.md` (repo root) for the proxy's overall status, and `programs/wsproxy/bench/` for the throughput/compression -numbers this design leans on. +numbers this design leans on. For the alternative of putting a WebSocket port *inside* +`clickhouse-server` (rather than as this edge sidecar), see `IN_SERVER.md`. ## Verdict diff --git a/programs/wsproxy/IN_SERVER.md b/programs/wsproxy/IN_SERVER.md new file mode 100644 index 000000000000..d67b0dcf02ea --- /dev/null +++ b/programs/wsproxy/IN_SERVER.md @@ -0,0 +1,103 @@ +# Adding a WebSocket port to `clickhouse-server` itself + +Feasibility notes for exposing a WebSocket interface **inside the server**, as a first-class port +next to `tcp_port` and `http_port` — instead of (or alongside) the standalone `wsproxy` sidecar. +**Status: analysis only — nothing here is built yet.** See `TODO.md` (repo root) for the proxy's +status and `programs/wsproxy/CLOUDFLARE.md` for the sidecar/edge deployment story. + +## Verdict + +**Technically moderate — mostly a port of existing code.** The server already ships the hard parts +(WebSocket framing, HTTP-upgrade, the port/handler-factory machinery), and the proxy's bridge logic +maps onto in-process execution almost unchanged. Estimate: **~1 week** for a working `ws_port` doing +SELECT + format + progress + cancel; **1-3 weeks** total for INSERT, auth, tests, config, and edge +cases — i.e. something upstreamable. + +**But it is a different value proposition, and it re-incurs the cost the sidecar exists to avoid.** +See "The strategic catch" below before treating this as a cheaper version of the sidecar. It is not; +it is a complementary, longer-horizon play. + +## Why it is mostly a port (verified against the code) + +- **The WS machinery already ships in the server.** `src/Server/WebTerminalRequestHandler.cpp` + implements RFC 6455 framing (`sendWebSocketFrame`, `readWebSocketFrame`), the HTTP upgrade, + ping/pong and close, and is registered through the normal factory in + `src/Server/HTTPHandlerFactory.cpp` (`handler_type == "webterminal"`). Framing + upgrade + + registration are solved. (That handler bridges WS to an interactive **PTY terminal** + via `ClientEmbeddedRunner`, which is *not* the query/format API we want — but it proves the plumbing.) +- **Adding a port is a well-trodden ~15-line pattern.** In `programs/server/Server.cpp`, every + interface is a small `createServer(...)` block plus a `ServerType::Type` enum entry + (`src/Server/ServerType.h`): `http_port`, `tcp_port`, `tcp_with_proxy_port`, `mysql_port`, + `postgresql_port`, `grpc_port`, `prometheus`, `arrowflight_port`, interserver, keeper. A WS port is + just another `HTTPServer` (WebSocket *is* an HTTP Upgrade) whose factory returns a WS handler. +- **The bridge target is free and identical to the proxy's abstraction.** Both the remote + `Connection` (`src/Client/Connection.h`) and the in-process `LocalConnection` + (`src/Client/LocalConnection.h`) implement the **same `IServerConnection`** interface + (`sendQuery` / `receivePacket` / `sendData` / `sendCancel` / `Packet`) — exactly what the proxy's + `ProxySession` already drives. So the server-side handler is `ProxySession` with the remote + `Connection` swapped for a `LocalConnection`; the packet↔frame loop, format conversion, + progress/log/profile push, cancel, and the INSERT handshake all carry over. + +## What you reuse vs. what is new + +**Reuse:** WS framing, the HTTP port + handler-factory machinery, `IServerConnection` / +`LocalConnection`, and the whole tested `ProxySession` bridge. + +**New / changed:** + +- A `WSQueryHandler : HTTPRequestHandler` — do the upgrade, then run the `ProxySession`-style bridge + against a per-connection `LocalConnection` bound to the request's session. +- A small refactor: `ProxySession` currently takes a concrete `Connection &`; generalize it to + `IServerConnection &` so it can drive either transport. +- **Auth is actually easier than the sidecar** — no credential pass-through to a remote backend; + reuse the server's existing HTTP session/authentication like every other handler. +- Config docs + an integration test (model on `tests/integration/test_webterminal`). + +## Two shapes + +1. **Dedicated `ws_port`** (what "next to native and HTTP" asks for). Adds a core diff: + - `programs/server/Server.cpp` — a `createServer(config, listen_host, "ws_port", …)` block returning + a `ProtocolServerAdapter` wrapping an `HTTPServer` whose factory produces `WSQueryHandler` + (model on the existing `http_port` block). + - `src/Server/ServerType.h` — a `WS` entry in `enum Type` (+ its name mappings). + - Server settings / config — a `ws_port` (and `ws_port_secure`) declaration, docs in the default + `config.xml`. + - `src/Server/HTTPHandlerFactory.cpp` — a factory that mounts `WSQueryHandler`. + - `src/Server/WSQueryHandler.{h,cpp}` — the handler (the real work). +2. **Endpoint on the existing `http_port`** (like `/webterminal`). **Zero `Server.cpp` changes** — add + a `handler_type == "ws"` (or a fixed `/ws` path) in `HTTPHandlerFactory.cpp` mounting + `WSQueryHandler`. Least invasive; not a separate port, but delivers the same capability. + +Recommendation: prototype as shape (2) (no core-server diff), and only promote to a dedicated +`ws_port` (shape 1) if a first-class port is wanted for ops parity with `tcp_port`/`http_port`. + +## The strategic catch + +Technically moderate — but putting it *in the server* re-incurs the exact cost the whole sidecar +project exists to avoid, and changes what the feature is worth: + +- **It is a core diff, shipped on the managed-cloud release cycle.** Touching `Server.cpp`, + `ServerType`, server settings and config is the rebase treadmill, and — more importantly — shipping + it goes through the slow, expensive cloud-release path. **Deployment velocity was the entire ROI of + the sidecar.** In-server is an upstream play measured in release cycles; the sidecar is a + ship-this-week play. +- **The edge value evaporates.** In-server there is no separate hop: conversion runs *on the cluster* + (the CPU we wanted to offload), and there is no compressed-native-over-WAN leg (the client talks WS + straight to the server). The edge-offload and ZSTD-on-the-wire pitch (see `bench/README.md` and + `CLOUDFLARE.md`) does **not** apply. + +So it is a **different, complementary value proposition**: a server-side WS port gives clients the +*protocol* wins the HTTP interface cannot do cleanly — mid-query progress/logs, cancel-by-close, +bidirectional streaming, format-at-source — **without needing a sidecar at all**. That is genuinely +valuable and arguably the "right" long-term home for a WebSocket interface. The two are complementary: +iterate on the sidecar now (fast, edge-deployable, offloads CPU, compresses the WAN hop); propose the +in-server `ws_port` upstream once the WS protocol/shape is proven. + +## Key references + +- `programs/server/Server.cpp` — `http_port` / `tcp_port` `createServer` blocks (the port pattern). +- `src/Server/ServerType.h` — the interface `enum Type`. +- `src/Server/HTTPHandlerFactory.cpp` — handler registration; the `webterminal` type is the template. +- `src/Server/WebTerminalRequestHandler.cpp` — shipped WS framing + upgrade (bridges to a PTY, not us). +- `src/Client/LocalConnection.h` / `src/Client/Connection.h` — both `: public IServerConnection`. +- `programs/wsproxy/ProxySession.{h,cpp}` — the bridge to port from `Connection&` to `IServerConnection&`. From b2db89233807872efadb7b97f235f2335af81c1d Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Wed, 15 Jul 2026 18:30:47 +0200 Subject: [PATCH 42/46] Relocate the WS<->native bridge into dbms and make it transport-agnostic Move WebSocketFrames.{h,cpp} and ProxySession.{h,cpp} from programs/wsproxy into src/Server (compiled into dbms), renaming ProxySession -> WebSocketSession. The bridge no longer builds its own remote Connection: run() now takes an already-connected IServerConnection&, and the remote-Connection creation + eager-connect/auth-error handling moved to WsProxyHandler. A new compression_method option replaces the BackendParams dependency (BackendParams moved to WsProxyHandler.h, proxy-side). This lets the same tested bridge serve both the standalone edge proxy (remote Connection) and an upcoming in-server WebSocket endpoint (in-process LocalConnection). Proxy links dbms so it picks up the relocated code; 113 tests still green. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/wsproxy/CMakeLists.txt | 5 +- programs/wsproxy/WsProxy.cpp | 1 - programs/wsproxy/WsProxyHandler.cpp | 92 ++++++++++++++++-- programs/wsproxy/WsProxyHandler.h | 20 +++- .../Server}/WebSocketFrames.cpp | 2 +- .../wsproxy => src/Server}/WebSocketFrames.h | 0 .../Server/WebSocketSession.cpp | 94 ++++++------------- .../Server/WebSocketSession.h | 65 ++++++------- 8 files changed, 163 insertions(+), 116 deletions(-) rename {programs/wsproxy => src/Server}/WebSocketFrames.cpp (99%) rename {programs/wsproxy => src/Server}/WebSocketFrames.h (100%) rename programs/wsproxy/ProxySession.cpp => src/Server/WebSocketSession.cpp (91%) rename programs/wsproxy/ProxySession.h => src/Server/WebSocketSession.h (54%) diff --git a/programs/wsproxy/CMakeLists.txt b/programs/wsproxy/CMakeLists.txt index 2cdfc5005507..742921545f7d 100644 --- a/programs/wsproxy/CMakeLists.txt +++ b/programs/wsproxy/CMakeLists.txt @@ -1,9 +1,10 @@ set (CLICKHOUSE_WSPROXY_SOURCES WsProxy.cpp WsProxyHandler.cpp - WebSocketFrames.cpp - ProxySession.cpp ) +# WebSocketFrames + WebSocketSession (the transport-agnostic WS<->native bridge) now live in +# src/Server (compiled into `dbms`), shared with the in-server WebSocket endpoint. This binary +# links `dbms`, so it picks them up automatically. set (CLICKHOUSE_WSPROXY_LINK PRIVATE diff --git a/programs/wsproxy/WsProxy.cpp b/programs/wsproxy/WsProxy.cpp index 7bcc1f9f4405..a91188e3b021 100644 --- a/programs/wsproxy/WsProxy.cpp +++ b/programs/wsproxy/WsProxy.cpp @@ -22,7 +22,6 @@ #include #include -#include #include diff --git a/programs/wsproxy/WsProxyHandler.cpp b/programs/wsproxy/WsProxyHandler.cpp index d7bb278d6839..866a4765e247 100644 --- a/programs/wsproxy/WsProxyHandler.cpp +++ b/programs/wsproxy/WsProxyHandler.cpp @@ -1,6 +1,14 @@ #include -#include -#include + +#include +#include + +#include +#include +#include +#include +#include +#include #include #include @@ -31,6 +39,35 @@ using namespace DB::WsProxy; namespace { +/// Minimal JSON string escaping for a short error message embedded in a control frame. +String jsonEscape(const String & s) +{ + String out; + out.reserve(s.size() + 2); + for (char c : s) + { + switch (c) + { + case '"': out += R"(\")"; break; + case '\\': out += R"(\\)"; break; + case '\n': out += R"(\n)"; break; + case '\r': out += R"(\r)"; break; + case '\t': out += R"(\t)"; break; + default: + if (static_cast(c) < 0x20) + { + static const char * hex = "0123456789abcdef"; + out += R"(\u00)"; + out += hex[(c >> 4) & 0xF]; + out += hex[c & 0xF]; + } + else + out += c; + } + } + return out; +} + /// Per RFC 7230 the `Connection` header is a comma-separated token list. Match /// the exact `upgrade` token rather than a substring (which would also accept /// unrelated values containing those letters). @@ -250,10 +287,53 @@ void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResp if (send_timeout_sec > 0) socket.setSendTimeout(Poco::Timespan(send_timeout_sec * 1'000'000)); /// microseconds - ProxySession session( - socket, context, session_backend, out_format, logs_level, flow_enabled, flow_credit, parse_enabled, - parallel_enabled); - session.run(); + /// Open one native-protocol connection per session to the remote backend. + Connection connection( + session_backend.host, + session_backend.port, + session_backend.database, + session_backend.user, + session_backend.password, + /* proto_send_chunked_ */ "chunked_optional", + /* proto_recv_chunked_ */ "chunked_optional", + /* ssh_private_key_ */ SSHKey{}, + /* jwt_ */ "", + /* quota_key_ */ "", + /* cluster_ */ "", + /* cluster_secret_ */ "", + /* client_name_ */ "clickhouse-wsproxy", + session_backend.compression_method == "none" ? Protocol::Compression::Disable : Protocol::Compression::Enable, + session_backend.secure ? Protocol::Secure::Enable : Protocol::Secure::Disable, + /* tls_sni_override_ */ "", + /* bind_host_ */ ""); + + /// Establish the backend connection up front so authentication (during the + /// native handshake) is reported immediately rather than on the first query. + /// On failure, tell the client and close. + try + { + connection.forceConnected(ConnectionTimeouts::getTCPTimeoutsWithoutFailover(context->getSettingsRef())); + } + catch (...) + { + const String message = getCurrentExceptionMessage(false); + LOG_DEBUG(log, "Backend connect/auth failed: {}", message); + try + { + sendWebSocketText(socket, R"({"event":"error","message":")" + jsonEscape(message) + "\"}"); + sendWebSocketClose(socket, /* 1008 policy violation */ 1008, "Authentication failed"); + } + catch (...) + { + LOG_DEBUG(log, "Failed to deliver auth error to client (already gone)"); + } + return; + } + + WebSocketSession session( + socket, context, out_format, logs_level, flow_enabled, flow_credit, parse_enabled, + parallel_enabled, session_backend.compression_method); + session.run(connection); LOG_DEBUG(log, "WebSocket session closed"); } diff --git a/programs/wsproxy/WsProxyHandler.h b/programs/wsproxy/WsProxyHandler.h index 8d9ee303c047..2ee869df9d5e 100644 --- a/programs/wsproxy/WsProxyHandler.h +++ b/programs/wsproxy/WsProxyHandler.h @@ -1,17 +1,31 @@ #pragma once -#include - #include #include +#include namespace DB { +/// Where the edge proxy forwards native-protocol queries (a remote backend). +struct BackendParams +{ + String host = "localhost"; + UInt16 port = 9000; + String user = "default"; + String password; + String database; + bool secure = false; /// Connect to the backend over TLS (native secure protocol). + /// Native-protocol compression codec for backend->proxy result blocks: "lz4" (fast, default), + /// "zstd" (higher ratio — fewer bytes on a bandwidth-limited WAN, at more backend CPU), or + /// "none". The proxy decompresses whatever the server sends regardless (codec is self-describing). + String compression_method = "lz4"; +}; + /// HTTP entry point for the proxy. /// /// A plain HTTP request serves a short info page. A WebSocket upgrade completes -/// the RFC 6455 handshake and then hands the socket to a `ProxySession`, which +/// the RFC 6455 handshake and then hands the socket to a `WebSocketSession`, which /// bridges the WebSocket to a native-protocol `Connection` against the backend. /// The desired output format is taken from the `format` query parameter of the /// WebSocket URL (default `JSONEachRow`). diff --git a/programs/wsproxy/WebSocketFrames.cpp b/src/Server/WebSocketFrames.cpp similarity index 99% rename from programs/wsproxy/WebSocketFrames.cpp rename to src/Server/WebSocketFrames.cpp index 37d0b4c1f154..334c80bdd37b 100644 --- a/programs/wsproxy/WebSocketFrames.cpp +++ b/src/Server/WebSocketFrames.cpp @@ -1,4 +1,4 @@ -#include +#include #include diff --git a/programs/wsproxy/WebSocketFrames.h b/src/Server/WebSocketFrames.h similarity index 100% rename from programs/wsproxy/WebSocketFrames.h rename to src/Server/WebSocketFrames.h diff --git a/programs/wsproxy/ProxySession.cpp b/src/Server/WebSocketSession.cpp similarity index 91% rename from programs/wsproxy/ProxySession.cpp rename to src/Server/WebSocketSession.cpp index 2415cb5870e5..a19f86bcfc27 100644 --- a/programs/wsproxy/ProxySession.cpp +++ b/src/Server/WebSocketSession.cpp @@ -1,7 +1,7 @@ -#include -#include +#include +#include -#include +#include #include #include @@ -146,7 +146,7 @@ QueryClass classifyQuery(const String & query) catch (...) { LOG_DEBUG( - getLogger("ProxySession"), + getLogger("WebSocketSession"), "parse=1 classification failed, treating as a plain query: {}", getCurrentExceptionMessage(false)); } @@ -205,7 +205,7 @@ void applyControlFrame( catch (...) { /// Malformed control frame: ignore (the query stream is unaffected). - LOG_DEBUG(getLogger("ProxySession"), "Ignoring malformed control frame: {}", getCurrentExceptionMessage(false)); + LOG_DEBUG(getLogger("WebSocketSession"), "Ignoring malformed control frame: {}", getCurrentExceptionMessage(false)); } } @@ -405,29 +405,29 @@ void sendQueryInfoFrame(Poco::Net::StreamSocket & socket, const String & verb, c } -ProxySession::ProxySession( +WebSocketSession::WebSocketSession( Poco::Net::StreamSocket & socket_, ContextPtr context_, - BackendParams backend_, String format_, String logs_level_, bool flow_enabled_, Int64 flow_initial_credit_, bool parse_enabled_, - bool parallel_enabled_) + bool parallel_enabled_, + String compression_method_) : socket(socket_) , context(std::move(context_)) - , backend(std::move(backend_)) , format(std::move(format_)) , logs_level(std::move(logs_level_)) , flow_enabled(flow_enabled_) , flow_initial_credit(flow_initial_credit_) , parse_enabled(parse_enabled_) , parallel_enabled(parallel_enabled_) + , compression_method(std::move(compression_method_)) { } -void ProxySession::sendControlEvent(const String & event, const String & message) +void WebSocketSession::sendControlEvent(const String & event, const String & message) { String json = R"({"event":")" + event + "\""; if (!message.empty()) @@ -437,7 +437,7 @@ void ProxySession::sendControlEvent(const String & event, const String & message sendWebSocketText(socket, json); } -void ProxySession::sendBlockEvent(const String & event, const Block & block) +void WebSocketSession::sendBlockEvent(const String & event, const Block & block) { if (block.rows() == 0) return; @@ -475,7 +475,7 @@ void ProxySession::sendBlockEvent(const String & event, const Block & block) sendWebSocketText(socket, R"({"event":")" + event + R"(","rows":[)" + rows_array + "]}"); } -void ProxySession::sendBackendQuery(Connection & connection, const String & query, bool with_pending_data) +void WebSocketSession::sendBackendQuery(IServerConnection & connection, const String & query, bool with_pending_data) { /// Copy the settings so we can opt into server log delivery. The backend /// attaches its log queue based on `send_logs_level` in the query packet's @@ -490,8 +490,8 @@ void ProxySession::sendBackendQuery(Connection & connection, const String & quer /// far better than row JSON) makes the proxy beat gzipped HTTP. Empty = leave /// the backend/connection default (LZ4). "none" disables block compression via /// the connection's compression flag, so nothing to set here. - if (backend.compression_method != "none" && !backend.compression_method.empty()) - settings.set("network_compression_method", backend.compression_method); + if (compression_method != "none" && !compression_method.empty()) + settings.set("network_compression_method", compression_method); auto timeouts = ConnectionTimeouts::getTCPTimeoutsWithoutFailover(settings); @@ -513,7 +513,7 @@ void ProxySession::sendBackendQuery(Connection & connection, const String & quer /* process_progress_callback */ {}); } -void ProxySession::drainUntilEndOfStream(Connection & connection) +void WebSocketSession::drainUntilEndOfStream(IServerConnection & connection) { try { @@ -527,11 +527,11 @@ void ProxySession::drainUntilEndOfStream(Connection & connection) catch (...) { /// The connection may be broken after a cancel; nothing more to drain. - LOG_DEBUG(getLogger("ProxySession"), "drain after cancel: {}", getCurrentExceptionMessage(false)); + LOG_DEBUG(getLogger("WebSocketSession"), "drain after cancel: {}", getCurrentExceptionMessage(false)); } } -std::optional ProxySession::readClientMessage() +std::optional WebSocketSession::readClientMessage() { String buffer; bool in_fragmented_message = false; @@ -582,9 +582,9 @@ std::optional ProxySession::readClientMessage() } } -bool ProxySession::executeSelect(Connection & connection, const String & query) +bool WebSocketSession::executeSelect(IServerConnection & connection, const String & query) { - LoggerPtr log = getLogger("ProxySession"); + LoggerPtr log = getLogger("WebSocketSession"); sendBackendQuery(connection, query); @@ -792,9 +792,9 @@ bool ProxySession::executeSelect(Connection & connection, const String & query) } } -bool ProxySession::executeInsert(Connection & connection, const String & query, const String & input_format) +bool WebSocketSession::executeInsert(IServerConnection & connection, const String & query, const String & input_format) { - LoggerPtr log = getLogger("ProxySession"); + LoggerPtr log = getLogger("WebSocketSession"); /// Format the client's streamed data is in: the command's `format` if given, /// else the session's `?format=`. The query text is forwarded verbatim (never @@ -895,53 +895,13 @@ bool ProxySession::executeInsert(Connection & connection, const String & query, } } -void ProxySession::run() +void WebSocketSession::run(IServerConnection & connection) { - LoggerPtr log = getLogger("ProxySession"); - - Connection connection( - backend.host, - backend.port, - backend.database, - backend.user, - backend.password, - /* proto_send_chunked_ */ "chunked_optional", - /* proto_recv_chunked_ */ "chunked_optional", - /* ssh_private_key_ */ SSHKey{}, - /* jwt_ */ "", - /* quota_key_ */ "", - /* cluster_ */ "", - /* cluster_secret_ */ "", - /* client_name_ */ "clickhouse-wsproxy", - backend.compression_method == "none" ? Protocol::Compression::Disable : Protocol::Compression::Enable, - backend.secure ? Protocol::Secure::Enable : Protocol::Secure::Disable, - /* tls_sni_override_ */ "", - /* bind_host_ */ ""); - - /// Establish the backend connection up front so authentication (which happens - /// during the native handshake) is reported immediately, rather than on the - /// first query. On failure, tell the client and close. - try - { - connection.forceConnected(ConnectionTimeouts::getTCPTimeoutsWithoutFailover(context->getSettingsRef())); - } - catch (...) - { - const String message = getCurrentExceptionMessage(false); - LOG_DEBUG(log, "Backend connect/auth failed: {}", message); - try - { - sendControlEvent("error", message); - sendWebSocketClose(socket, /* 1008 policy violation */ 1008, "Authentication failed"); - } - catch (...) - { - LOG_DEBUG(log, "Failed to deliver auth error to client (already gone)"); - } - return; - } + LoggerPtr log = getLogger("WebSocketSession"); - LOG_DEBUG(log, "Proxy session started; backend {}:{}", backend.host, backend.port); + /// The caller owns connection setup (and, for a remote transport, eager + /// authentication). `connection` is expected to be usable here. + LOG_DEBUG(log, "WebSocket session started"); while (true) { @@ -998,7 +958,7 @@ void ProxySession::run() } } - LOG_DEBUG(log, "Proxy session ended"); + LOG_DEBUG(log, "WebSocket session ended"); } } diff --git a/programs/wsproxy/ProxySession.h b/src/Server/WebSocketSession.h similarity index 54% rename from programs/wsproxy/ProxySession.h rename to src/Server/WebSocketSession.h index 000a2d8af211..283c15cb593c 100644 --- a/programs/wsproxy/ProxySession.h +++ b/src/Server/WebSocketSession.h @@ -11,46 +11,36 @@ namespace DB { -class Connection; +class IServerConnection; -/// Where the proxy forwards native-protocol queries. -struct BackendParams -{ - String host = "localhost"; - UInt16 port = 9000; - String user = "default"; - String password; - String database; - bool secure = false; /// Connect to the backend over TLS (native secure protocol). - /// Native-protocol compression codec for backend->proxy result blocks: "lz4" (fast, default), - /// "zstd" (higher ratio — fewer bytes on a bandwidth-limited WAN, at more backend CPU), or - /// "none". The proxy decompresses whatever the server sends regardless (codec is self-describing). - String compression_method = "lz4"; -}; - -/// Drives one WebSocket session against a backend ClickHouse server over the -/// native protocol. +/// Bridges one WebSocket session to a ClickHouse query connection, in both +/// directions, over the native `IServerConnection` abstraction. /// -/// Owns a `Connection` for the lifetime of the session. For each query the -/// client sends as a WebSocket text frame, it runs `sendQuery` / `receivePacket` -/// and streams the result blocks back as binary frames (encoded with the chosen -/// output format), followed by a JSON control frame (`end` / `error` / -/// `cancelled`). A Close frame arriving mid-query triggers `Connection::sendCancel`. -class ProxySession +/// Transport-agnostic: the connection is injected into `run`, so the same bridge +/// serves the standalone edge proxy (a remote `Connection`) and an in-server +/// WebSocket endpoint (an in-process `LocalConnection`). For each query the client +/// sends as a text frame it runs `sendQuery` / `receivePacket` and streams the +/// result blocks back as binary frames (encoded with the chosen output format), +/// followed by a JSON control frame (`end` / `error` / `cancelled`). A Close frame +/// arriving mid-query triggers `IServerConnection::sendCancel`. +class WebSocketSession { public: - ProxySession( + WebSocketSession( Poco::Net::StreamSocket & socket_, ContextPtr context_, - BackendParams backend_, String format_, String logs_level_ = "", bool flow_enabled_ = false, Int64 flow_initial_credit_ = 0, bool parse_enabled_ = false, - bool parallel_enabled_ = false); + bool parallel_enabled_ = false, + String compression_method_ = ""); - void run(); + /// Drive the session loop over an already-connected `connection` until the + /// client closes or errors. The caller owns connection setup/teardown and + /// (for a remote transport) authentication. + void run(IServerConnection & connection); private: /// Read one complete client message (reassembling fragments, answering pings). @@ -59,25 +49,24 @@ class ProxySession std::optional readClientMessage(); /// Run a plain query and stream its result. Returns false if the session - /// should end afterwards (client closed mid-query). The proxy never parses - /// SQL: routing is driven by the client's message kind (a raw text frame is - /// a query -> executeSelect; a {"cmd":"insert",...} control message opts into + /// should end afterwards (client closed mid-query). SQL is never parsed: + /// routing is driven by the client's message kind (a raw text frame is a + /// query -> executeSelect; a {"cmd":"insert",...} control message opts into /// executeInsert). `with_pending_data=false` here, so SELECT / INSERT-SELECT / /// inline INSERT / DDL all work and none waits for client data. - bool executeSelect(Connection & connection, const String & query); + bool executeSelect(IServerConnection & connection, const String & query); /// Stream a client-supplied data INSERT: send the query with `with_pending_data`, /// parse the streamed frames with `input_format`, and `sendData` blocks. - bool executeInsert(Connection & connection, const String & query, const String & input_format); + bool executeInsert(IServerConnection & connection, const String & query, const String & input_format); - void sendBackendQuery(Connection & connection, const String & query, bool with_pending_data = false); - void drainUntilEndOfStream(Connection & connection); + void sendBackendQuery(IServerConnection & connection, const String & query, bool with_pending_data = false); + void drainUntilEndOfStream(IServerConnection & connection); void sendControlEvent(const String & event, const String & message); /// Serialize a Log / ProfileEvents block to a `{"event":...,"rows":[...]}` text frame. void sendBlockEvent(const String & event, const Block & block); Poco::Net::StreamSocket & socket; ContextPtr context; - BackendParams backend; String format; String logs_level; /// If set, sent as `send_logs_level` so the backend pushes Log packets. bool flow_enabled; /// Opt-in credit/window flow control for the SELECT push direction. @@ -86,6 +75,10 @@ class ProxySession /// Opt-in (?parallel=1): format SELECT output on a thread pool for higher conversion /// throughput, at the cost of coarser (batched) result frames. Ignored when flow control is on. bool parallel_enabled; + /// Native-protocol codec the backend uses for result blocks (`network_compression_method`): + /// "lz4" / "zstd" for a remote transport, empty (or "none") to leave it unset (in-process + /// connections have no wire, so this is a no-op there). + String compression_method; /// Serializes complete WebSocket frame sends. With parallel output formatting the format's /// collector thread writes result frames while the session thread pushes progress/log/control From b0ee7757beb37d1bbd8172976bd2fdaedd85a03b Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 16 Jul 2026 15:08:45 +0200 Subject: [PATCH 43/46] Add an in-server WebSocket query endpoint (ws_port) A first-class WebSocket interface inside clickhouse-server, next to tcp_port/http_port, reusing the shared WebSocketSession bridge against an in-process LocalConnection. New WSHandler (src/Server/WSHandler.{h,cpp}) does the RFC 6455 upgrade, authenticates via the server's own session auth (Basic / X-ClickHouse headers / ?user=&password= / default), and runs the bridge; registered as a WSHandler-factory in HTTPHandlerFactory and wired as ws_port in Server.cpp with a new ServerType::WS. Same wire protocol as clickhouse-wsproxy (query text frame -> binary result frames in ?format=, JSON control frames, mid-query progress/logs/profile-events, cancel-by-close, ?logs/?flow/?parse/?parallel). WebSocketSession fixes needed for the in-process transport (all also correct for the remote proxy): materialize const/sparse columns before output->write (LocalConnection yields e.g. ColumnConst for SELECT 1, which row formats mishandled -> was a fatal abort); pass no ClientInfo to sendQuery so LocalConnection derives the user from its authenticated session (fixes currentUser()); skip sendExternalTablesData for LOCAL connections (not implemented / not needed); check the client socket on every packet iteration in executeSelect (LocalConnection::poll returns immediately with data, so cancel-by-close must not depend on poll timing out). Logs level is applied to the session context in WSHandler because LocalConnection ignores the per-query settings. Both suites pass with the shared bridge: standalone proxy 113/113, in-server ws_port 102/102. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/server/Server.cpp | 23 +++ src/Server/HTTPHandlerFactory.cpp | 14 ++ src/Server/ServerType.cpp | 4 + src/Server/ServerType.h | 1 + src/Server/WSHandler.cpp | 314 ++++++++++++++++++++++++++++++ src/Server/WSHandler.h | 33 ++++ src/Server/WebSocketSession.cpp | 88 +++++---- 7 files changed, 442 insertions(+), 35 deletions(-) create mode 100644 src/Server/WSHandler.cpp create mode 100644 src/Server/WSHandler.h diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 1f4f9d6c2c6f..52b5d64ede7a 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -3653,6 +3653,29 @@ void Server::createServers( }); } + if (server_type.shouldStart(ServerType::Type::WS)) + { + /// WebSocket. An HTTP server whose handler upgrades the connection to a WebSocket and + /// bridges it to an in-process query connection (same wire protocol as clickhouse-wsproxy): + /// query as a text frame, results as binary frames in the ?format= output format, plus + /// mid-query progress/logs/profile-events and cancel-by-close. + port_name = "ws_port"; + createServer(config, listen_host, port_name, listen_try, start_servers, servers, [&](UInt16 port) -> ProtocolServerAdapter + { + Poco::Net::ServerSocket socket; + auto address = socketBindListen(server_settings, socket, listen_host, port); + socket.setReceiveTimeout(settings[Setting::http_receive_timeout]); + socket.setSendTimeout(settings[Setting::http_send_timeout]); + + return ProtocolServerAdapter( + listen_host, + port_name, + "websocket: " + address.toString(), + std::make_unique( + httpContext(), createHandlerFactory(*this, config, async_metrics, "WSHandler-factory"), server_pool, socket, http_params, connection_filter, ProfileEvents::InterfaceHTTPReceiveBytes, ProfileEvents::InterfaceHTTPSendBytes)); + }); + } + if (server_type.shouldStart(ServerType::Type::TCP)) { /// TCP diff --git a/src/Server/HTTPHandlerFactory.cpp b/src/Server/HTTPHandlerFactory.cpp index 31c22da8256b..5d5454688736 100644 --- a/src/Server/HTTPHandlerFactory.cpp +++ b/src/Server/HTTPHandlerFactory.cpp @@ -10,6 +10,7 @@ #include #include #include +#include #if CLICKHOUSE_CLOUD #include #endif @@ -302,6 +303,17 @@ createHTTPHandlerFactory(IServer & server, const Poco::Util::AbstractConfigurati return factory; } +static inline HTTPRequestHandlerFactoryPtr createWSHandlerFactory(IServer & server, const std::string & name) +{ + /// The `ws_port` speaks WebSocket only: every request goes to the WSHandler, + /// which serves an info page for a plain GET and upgrades a WebSocket request. + auto factory = std::make_shared(name); + auto main_handler = std::make_shared>(server); + main_handler->allowGetAndHeadRequest(); + factory->addHandler(main_handler); + return factory; +} + static inline HTTPRequestHandlerFactoryPtr createInterserverHTTPHandlerFactory(IServer & server, const std::string & name, const Poco::Util::AbstractConfiguration & config) { auto factory = std::make_shared(name); @@ -320,6 +332,8 @@ HTTPRequestHandlerFactoryPtr createHandlerFactory(IServer & server, const Poco:: return createHTTPHandlerFactory(server, config, name, async_metrics, http_handlers_key.empty() ? "http_handlers" : http_handlers_key); if (name == "InterserverIOHTTPHandler-factory" || name == "InterserverIOHTTPSHandler-factory") return createInterserverHTTPHandlerFactory(server, name, config); + if (name == "WSHandler-factory") + return createWSHandlerFactory(server, name); if (name == "PrometheusHandler-factory") return createPrometheusHandlerFactory(server, config, async_metrics, name); if (name == "KeeperPrometheusHandler-factory") diff --git a/src/Server/ServerType.cpp b/src/Server/ServerType.cpp index 932894e06447..eea3835d5b9b 100644 --- a/src/Server/ServerType.cpp +++ b/src/Server/ServerType.cpp @@ -52,6 +52,7 @@ bool ServerType::shouldStart(Type server_type, const std::string & server_custom case Type::TCP_SECURE: case Type::HTTP: case Type::HTTPS: + case Type::WS: case Type::MYSQL: case Type::GRPC: case Type::POSTGRESQL: @@ -109,6 +110,9 @@ bool ServerType::shouldStop(const std::string & port_name) const else if (port_name == "https_port") port_type = Type::HTTPS; + else if (port_name == "ws_port") + port_type = Type::WS; + else if (port_name == "tcp_port") port_type = Type::TCP; diff --git a/src/Server/ServerType.h b/src/Server/ServerType.h index 2839ccfd99a3..0f5f53c0a73a 100644 --- a/src/Server/ServerType.h +++ b/src/Server/ServerType.h @@ -17,6 +17,7 @@ class ServerType TCP, HTTP, HTTPS, + WS, MYSQL, GRPC, ARROW_FLIGHT, diff --git a/src/Server/WSHandler.cpp b/src/Server/WSHandler.cpp new file mode 100644 index 000000000000..36616aeff373 --- /dev/null +++ b/src/Server/WSHandler.cpp @@ -0,0 +1,314 @@ +#include + +#include +#include + +#include +#include +#include + +#include +#include +#include +#include + +#include +#include +#include + +#include + +#include +#include +#include +#include + +#include + +#include + +namespace DB +{ + +using namespace DB::WsProxy; + +namespace +{ + +/// One resolved credential pair (delegated to ClickHouse for authN/authZ). +struct ResolvedCredentials +{ + String user = "default"; + String password; +}; + +String queryParam(const String & uri_string, const String & name, const String & fallback, LoggerPtr log) +{ + try + { + Poco::URI uri(uri_string); + for (const auto & param : uri.getQueryParameters()) + if (param.first == name && !param.second.empty()) + return param.second; + } + catch (...) + { + LOG_DEBUG(log, "Could not parse request URI for the {} parameter; using default", name); + } + return fallback; +} + +/// Per RFC 7230 the `Connection` header is a comma-separated token list; match the +/// exact `upgrade` token rather than a substring. +bool hasUpgradeToken(const String & connection_header) +{ + String value = connection_header; + std::transform(value.begin(), value.end(), value.begin(), ::tolower); + size_t pos = 0; + while (pos <= value.size()) + { + size_t comma = value.find(',', pos); + size_t end_pos = (comma == String::npos) ? value.size() : comma; + size_t start = value.find_first_not_of(" \t", pos); + if (start != String::npos && start < end_pos) + { + size_t last = value.find_last_not_of(" \t", end_pos - 1); + if (value.substr(start, last - start + 1) == "upgrade") + return true; + } + if (comma == String::npos) + break; + pos = comma + 1; + } + return false; +} + +/// Minimal JSON string escaping for a short error message in a control frame. +String jsonEscape(const String & s) +{ + String out; + out.reserve(s.size() + 2); + for (char c : s) + { + switch (c) + { + case '"': out += R"(\")"; break; + case '\\': out += R"(\\)"; break; + case '\n': out += R"(\n)"; break; + case '\r': out += R"(\r)"; break; + case '\t': out += R"(\t)"; break; + default: + if (static_cast(c) < 0x20) + { + static const char * hex = "0123456789abcdef"; + out += R"(\u00)"; + out += hex[(c >> 4) & 0xF]; + out += hex[c & 0xF]; + } + else + out += c; + } + } + return out; +} + +/// Resolve credentials the same way clickhouse-wsproxy does, so the same clients +/// work against either deployment: Authorization: Basic > X-ClickHouse-User/-Key +/// headers > ?user=/?password= URL params > the `default` user. +ResolvedCredentials resolveCredentials(const HTTPServerRequest & request, const String & uri, LoggerPtr log) +{ + ResolvedCredentials creds; + + const String auth = request.get("Authorization", ""); + if (auth.starts_with("Basic ")) + { + try + { + const String decoded = base64Decode(auth.substr(6)); + const size_t colon = decoded.find(':'); + if (colon != String::npos) + { + creds.user = decoded.substr(0, colon); + creds.password = decoded.substr(colon + 1); + return creds; + } + } + catch (...) + { + LOG_DEBUG(log, "Malformed Authorization header; falling back to other credential sources"); + } + } + + const String header_user = request.get("X-ClickHouse-User", ""); + if (!header_user.empty()) + { + creds.user = header_user; + creds.password = request.get("X-ClickHouse-Key", ""); + return creds; + } + + const String param_user = queryParam(uri, "user", "", log); + if (!param_user.empty()) + { + creds.user = param_user; + creds.password = queryParam(uri, "password", "", log); + } + return creds; +} + +} + +WSHandler::WSHandler(IServer & server_) : server(server_) +{ +} + +void WSHandler::handleRequest(HTTPServerRequest & request, HTTPServerResponse & response, const ProfileEvents::Event &) +{ + try + { + String upgrade = request.get("Upgrade", ""); + std::transform(upgrade.begin(), upgrade.end(), upgrade.begin(), ::tolower); + + if (upgrade == "websocket" && hasUpgradeToken(request.get("Connection", ""))) + handleWebSocket(request, response); + else + serveInfo(request, response); + } + catch (...) + { + tryLogCurrentException("WSHandler"); + } +} + +void WSHandler::serveInfo(HTTPServerRequest & request, HTTPServerResponse & response) +{ + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_OK); + response.setContentType("text/plain; charset=UTF-8"); + *response.send() + << "ClickHouse WebSocket endpoint\n" + << "Requested: " << request.getURI() << "\n" + << "Open a WebSocket connection to run queries. Send a query as a text\n" + << "frame; results stream back as binary frames in the output format\n" + << "(set via ?format=..., default JSONEachRow), followed by a JSON\n" + << "control frame ({\"event\":\"end\"} / \"error\" / \"cancelled\").\n"; +} + +void WSHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResponse & response) +{ + LoggerPtr log = getLogger("WSHandler"); + + if (request.getMethod() != Poco::Net::HTTPRequest::HTTP_GET) + { + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_METHOD_NOT_ALLOWED); + *response.send() << "WebSocket upgrade requires GET method.\n"; + return; + } + + String ws_key = request.get("Sec-WebSocket-Key", ""); + if (!isValidWebSocketKey(ws_key)) + { + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_BAD_REQUEST); + *response.send() << "Invalid or missing Sec-WebSocket-Key.\n"; + return; + } + if (request.get("Sec-WebSocket-Version", "") != "13") + { + response.set("Sec-WebSocket-Version", "13"); + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_BAD_REQUEST); + *response.send() << "Unsupported WebSocket version.\n"; + return; + } + + /// Complete the handshake by writing 101 directly to the socket; from here on + /// the stream is in WebSocket framing mode and we own the socket. + Poco::Net::StreamSocket & socket = response.getSocket(); + const String handshake + = "HTTP/1.1 101 Switching Protocols\r\n" + "Upgrade: websocket\r\n" + "Connection: Upgrade\r\n" + "Sec-WebSocket-Accept: " + + computeWebSocketAccept(ws_key) + "\r\n\r\n"; + + size_t sent_total = 0; + while (sent_total < handshake.size()) + { + int sent = socket.sendBytes(handshake.data() + sent_total, static_cast(handshake.size() - sent_total)); + if (sent <= 0) + { + LOG_DEBUG(log, "Failed to write WebSocket handshake"); + return; + } + sent_total += static_cast(sent); + } + + SCOPE_EXIT({ + try + { + socket.shutdown(); + } + catch (...) + { + LOG_DEBUG(log, "WebSocket socket shutdown: {}", getCurrentExceptionMessage(false)); + } + }); + + const String & uri = request.getURI(); + const String out_format = queryParam(uri, "format", "JSONEachRow", log); + const String logs_level = queryParam(uri, "logs", "", log); + const String flow_param = queryParam(uri, "flow", "", log); + const bool flow_enabled = !flow_param.empty(); + const Int64 flow_credit = flow_enabled ? std::strtoll(flow_param.c_str(), nullptr, 10) : 0; + const String parse_param = queryParam(uri, "parse", "", log); + const bool parse_enabled = parse_param == "1" || parse_param == "true"; + const String parallel_param = queryParam(uri, "parallel", "", log); + const bool parallel_enabled = parallel_param == "1" || parallel_param == "true"; + + /// Bound each blocking read; bound blocking writes so a client that stops + /// reading cannot pin a handler thread (mirrors the standalone proxy). + socket.setReceiveTimeout(Poco::Timespan(300, 0)); + socket.setSendTimeout(Poco::Timespan(30, 0)); + + /// Authenticate against the server (ClickHouse performs authN/authZ) and + /// build an in-process connection bound to that session. + const ResolvedCredentials creds = resolveCredentials(request, uri, log); + ContextMutablePtr session_context; + std::unique_ptr connection; + try + { + auto session = std::make_unique(server.context(), ClientInfo::Interface::HTTP, request.isSecure()); + session->authenticate(BasicCredentials(creds.user, creds.password), request.clientAddress()); + session_context = session->makeSessionContext(); + /// LocalConnection ignores the per-query settings passed to sendQuery and reads them from + /// its context, so apply the session-level knobs here. send_logs_level drives whether the + /// in-process query pushes Log packets (which WebSocketSession forwards as `log` frames). + if (!logs_level.empty()) + session_context->setSetting("send_logs_level", logs_level); + connection = std::make_unique( + std::move(session), /* in */ nullptr, /* send_progress */ true, /* send_profile_events */ true, /* server_display_name */ ""); + } + catch (...) + { + const String message = getCurrentExceptionMessage(false); + LOG_DEBUG(log, "WebSocket authentication failed: {}", message); + try + { + sendWebSocketText(socket, R"({"event":"error","message":")" + jsonEscape(message) + "\"}"); + sendWebSocketClose(socket, /* 1008 policy violation */ 1008, "Authentication failed"); + } + catch (...) + { + LOG_DEBUG(log, "Failed to deliver auth error to client (already gone)"); + } + return; + } + + LOG_DEBUG(log, "WebSocket session established for user {}; format {}", creds.user, out_format); + + WebSocketSession session( + socket, session_context, out_format, logs_level, flow_enabled, flow_credit, parse_enabled, parallel_enabled); + session.run(*connection); + + LOG_DEBUG(log, "WebSocket session closed"); +} + +} diff --git a/src/Server/WSHandler.h b/src/Server/WSHandler.h new file mode 100644 index 000000000000..89f0e0851a80 --- /dev/null +++ b/src/Server/WSHandler.h @@ -0,0 +1,33 @@ +#pragma once + +#include + +namespace DB +{ + +class IServer; + +/// In-server WebSocket query endpoint (the `ws_port`). +/// +/// A plain HTTP request serves a short info page. A WebSocket upgrade completes +/// the RFC 6455 handshake, authenticates via the server's own session auth +/// (Basic / X-ClickHouse headers / ?user=&password= / default), and then bridges +/// the socket to an in-process `LocalConnection` using the shared +/// `WebSocketSession`. The wire protocol is identical to `clickhouse-wsproxy`: +/// query as a text frame, results as binary frames in the `?format=` output +/// format, JSON control frames, and the `?logs`/`?flow`/`?parse`/`?parallel` knobs. +class WSHandler : public HTTPRequestHandler +{ +public: + explicit WSHandler(IServer & server_); + + void handleRequest(HTTPServerRequest & request, HTTPServerResponse & response, const ProfileEvents::Event & write_event) override; + +private: + void handleWebSocket(HTTPServerRequest & request, HTTPServerResponse & response); + void serveInfo(HTTPServerRequest & request, HTTPServerResponse & response); + + IServer & server; +}; + +} diff --git a/src/Server/WebSocketSession.cpp b/src/Server/WebSocketSession.cpp index a19f86bcfc27..d8bcc4c98a9f 100644 --- a/src/Server/WebSocketSession.cpp +++ b/src/Server/WebSocketSession.cpp @@ -3,6 +3,7 @@ #include +#include #include #include #include @@ -448,7 +449,7 @@ void WebSocketSession::sendBlockEvent(const String & event, const Block & block) /// correct for whatever columns/types the server sends. WriteBufferFromOwnString buf; auto out = FormatFactory::instance().getOutputFormat("JSONEachRow", buf, block.cloneEmpty(), context); - out->write(block); + out->write(materializeBlock(block, !out->supportsSpecialSerializationKinds())); out->finalize(); const String & rows_text = buf.str(); @@ -495,11 +496,12 @@ void WebSocketSession::sendBackendQuery(IServerConnection & connection, const St auto timeouts = ConnectionTimeouts::getTCPTimeoutsWithoutFailover(settings); - ClientInfo client_info; - client_info.query_kind = ClientInfo::QueryKind::INITIAL_QUERY; - - /// `with_pending_data` must be true for INSERTs so the server enters the - /// send-data handshake and replies with the sample/header block. + /// Pass no ClientInfo: a remote Connection then lets the server default the + /// query kind for a direct client, and an in-process LocalConnection derives + /// the query context (and current user) from its authenticated session rather + /// than being clobbered by an empty ClientInfo. + /// `with_pending_data` must be true for INSERTs so the connection enters the + /// send-data phase and replies with the sample/header block. connection.sendQuery( timeouts, query, @@ -507,7 +509,7 @@ void WebSocketSession::sendBackendQuery(IServerConnection & connection, const St /* query_id */ "", QueryProcessingStage::Complete, &settings, - &client_info, + /* client_info */ nullptr, with_pending_data, /* external_roles */ {}, /* process_progress_callback */ {}); @@ -637,32 +639,38 @@ bool WebSocketSession::executeSelect(IServerConnection & connection, const Strin UInt64 total_read_bytes = 0; UInt64 total_rows_to_read = 0; - while (true) + /// Read any pending client frame (non-blocking): flow-control grants + /// (next/pause/resume) update the shared FlowControl; a Close/broken read means + /// the client is gone -> cancel the query and drain. + auto poll_client_frame = [&] { - /// While no server data is pending, watch the client socket so a Close - /// frame mid-query cancels the running query promptly. - while (!connection.poll(50'000 /* microseconds */)) + if (client_gone || !socket.poll(Poco::Timespan(0), Poco::Net::Socket::SELECT_READ)) + return; + WebSocketFrame frame; + try { - if (socket.poll(Poco::Timespan(0), Poco::Net::Socket::SELECT_READ)) - { - WebSocketFrame frame; - try - { - frame = readWebSocketFrame(socket); - } - catch (...) - { - frame.valid = false; - } - - /// Apply the client frame: flow-control grants (next/pause/resume) - /// update the shared FlowControl; a Close/broken read means the - /// client is gone -> cancel the backend query and drain. - applyControlFrame(socket, frame, fc, &ws_write_mutex); - if (fc.client_gone) - note_client_gone(); - } + frame = readWebSocketFrame(socket); } + catch (...) + { + frame.valid = false; + } + applyControlFrame(socket, frame, fc, &ws_write_mutex); + if (fc.client_gone) + note_client_gone(); + }; + + while (true) + { + /// Watch the client on every iteration, not only when the connection has no + /// data ready: an in-process LocalConnection::poll returns immediately with + /// data, so a mid-query Close would otherwise go unnoticed until the query + /// finished. This is what makes cancel-by-close work for the in-server path. + poll_client_frame(); + + /// While no server data is pending, keep watching the client socket. + while (!connection.poll(50'000 /* microseconds */)) + poll_client_frame(); Packet packet = connection.receivePacket(); switch (packet.type) @@ -680,7 +688,12 @@ bool WebSocketSession::executeSelect(IServerConnection & connection, const Strin } if (packet.block.rows() > 0) { - output->write(packet.block); + /// Materialize const/sparse/low-cardinality columns before writing: an + /// in-process LocalConnection hands blocks straight from the pipeline (e.g. + /// `SELECT 1` yields a ColumnConst), which row output formats mishandle. + /// Remote connections already materialize over the wire, so this is a cheap + /// no-op there. Mirrors ClientBase::onData. + output->write(materializeBlock(packet.block, !output->supportsSpecialSerializationKinds())); output->flush(); /// Stream this block as its own frame(s). if (out_buf.isBroken()) note_client_gone(); /// Client left mid-stream. @@ -803,10 +816,15 @@ bool WebSocketSession::executeInsert(IServerConnection & connection, const Strin sendBackendQuery(connection, query, /* with_pending_data */ true); - /// The server waits for external-tables data before replying with the sample - /// block; we have none, so send an empty set to unblock the handshake. - ExternalTablesData external_tables_data; - connection.sendExternalTablesData(external_tables_data); + /// A remote server waits for external-tables data before replying with the + /// sample block; we have none, so send an empty set to unblock that native + /// handshake. An in-process LocalConnection has no such handshake (and does + /// not implement sendExternalTablesData), so skip it there. + if (connection.getConnectionType() != IServerConnection::Type::LOCAL) + { + ExternalTablesData external_tables_data; + connection.sendExternalTablesData(external_tables_data); + } /// The server replies with the sample block describing the target structure. Block sample; From 55efa00b247eb6ceeff6e000d08d292c8e4cddaf Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 16 Jul 2026 15:09:58 +0200 Subject: [PATCH 44/46] Test harness + docs for the in-server ws_port Add spawnServerWithWs (runs the built clickhouse binary with a ws_port), serverSetup.mjs, and vitest.server.config.mjs (npm run test:server) that runs the shared query/protocol suite against the in-server endpoint, excluding proxy-only suites. Document both targets in the tests README, mark IN_SERVER.md as prototype-built-and-tested with the LocalConnection gotchas, and update TODO.md. In-server 102/102, proxy 113/113. Co-Authored-By: Claude Opus 4.8 (1M context) --- TODO.md | 11 +-- programs/wsproxy/IN_SERVER.md | 45 ++++++++++-- programs/wsproxy/tests/README.md | 15 ++++ programs/wsproxy/tests/package.json | 3 +- programs/wsproxy/tests/test/proc.mjs | 68 +++++++++++++++++++ programs/wsproxy/tests/test/serverSetup.mjs | 50 ++++++++++++++ .../wsproxy/tests/vitest.server.config.mjs | 26 +++++++ 7 files changed, 209 insertions(+), 9 deletions(-) create mode 100644 programs/wsproxy/tests/test/serverSetup.mjs create mode 100644 programs/wsproxy/tests/vitest.server.config.mjs diff --git a/TODO.md b/TODO.md index 0d4ace77852a..dfa7c3476476 100644 --- a/TODO.md +++ b/TODO.md @@ -92,10 +92,13 @@ own backend + proxy). Benchmark harness committed under `programs/wsproxy/bench/ 7. **Cloudflare deployment** (Containers + stateless Workers) — feasibility, architecture, scaling, and frictions are written up in `programs/wsproxy/CLOUDFLARE.md`. Long pole is the amd64 Linux build; the Worker/DO glue is small. -8. **In-server WS port** (alternative home) — feasibility of a `ws_port` inside `clickhouse-server` - (bridge `ProxySession` to a `LocalConnection`) is written up in `programs/wsproxy/IN_SERVER.md`. - Technically moderate, but a core diff on the cloud-release cycle and it loses the edge/compression - value — a complementary upstream play, not a faster sidecar. +8. **In-server WS port** (alternative home) — **PROTOTYPE BUILT AND TESTED.** A `ws_port` inside + `clickhouse-server` bridges the shared `WebSocketSession` to an in-process `LocalConnection`; the + same vitest suite passes against it (in-server 102/102, proxy 113/113). Details + the + LocalConnection gotchas in `programs/wsproxy/IN_SERVER.md`. Still a core diff on the cloud-release + cycle and it loses the edge/compression value — a complementary upstream play, not a faster + sidecar. The bridge (`WebSocketFrames` + `WebSocketSession`) now lives in `src/Server` (dbms), + shared by both the proxy binary and the server. ### Lowlights / known limitations / risks diff --git a/programs/wsproxy/IN_SERVER.md b/programs/wsproxy/IN_SERVER.md index d67b0dcf02ea..6a8be7ae51d2 100644 --- a/programs/wsproxy/IN_SERVER.md +++ b/programs/wsproxy/IN_SERVER.md @@ -1,9 +1,14 @@ # Adding a WebSocket port to `clickhouse-server` itself -Feasibility notes for exposing a WebSocket interface **inside the server**, as a first-class port -next to `tcp_port` and `http_port` — instead of (or alongside) the standalone `wsproxy` sidecar. -**Status: analysis only — nothing here is built yet.** See `TODO.md` (repo root) for the proxy's -status and `programs/wsproxy/CLOUDFLARE.md` for the sidecar/edge deployment story. +A WebSocket interface **inside the server**, as a first-class port next to `tcp_port` and +`http_port` — an alternative to (and complement of) the standalone `wsproxy` sidecar. See `TODO.md` +(repo root) for the proxy's status and `programs/wsproxy/CLOUDFLARE.md` for the sidecar/edge story. + +**Status: PROTOTYPE BUILT AND TESTED.** A `ws_port` is wired into `clickhouse-server`, reusing the +shared `WebSocketSession` bridge against an in-process `LocalConnection`. The same Node/vitest suite +that exercises the proxy runs against it (`npm run test:server`): **in-server 102/102, proxy 113/113** +(proxy-only suites — separate remote backend, proxy→backend TLS, `WSPROXY_BACKEND_COMPRESSION` — are +excluded as N/A in-server). What was built and the LocalConnection gotchas learned are at the end. ## Verdict @@ -93,6 +98,38 @@ valuable and arguably the "right" long-term home for a WebSocket interface. The iterate on the sidecar now (fast, edge-deployable, offloads CPU, compresses the WAN hop); propose the in-server `ws_port` upstream once the WS protocol/shape is proven. +## What was built (and the LocalConnection gotchas) + +- **Shared bridge.** `WebSocketFrames` + `WebSocketSession` live in `src/Server` (compiled into + `dbms`); `WebSocketSession::run(IServerConnection&)` is transport-agnostic. The proxy injects a + remote `Connection`; the server injects a `LocalConnection`. +- **`WSHandler`** (`src/Server/WSHandler.{h,cpp}`): RFC 6455 upgrade → authenticate a `Session` + (Basic / `X-ClickHouse-*` / `?user=&password=` / default) → `LocalConnection` over that session → + run the bridge. Registered as `WSHandler-factory` in `HTTPHandlerFactory`; wired as `ws_port` in + `Server.cpp` with a new `ServerType::WS`. + +Four behaviours differ between a remote `Connection` and an in-process `LocalConnection`, and each +needed a fix in the shared bridge (all also correct for the proxy): + +1. **Materialize before formatting.** `LocalConnection` hands blocks straight from the pipeline, so + `SELECT 1` yields a `ColumnConst`; row output formats mishandle that (it was a *fatal abort* in + `SerializationString::serializeTextJSON`). Fixed by `materializeBlock(...)` before `output->write`, + mirroring `ClientBase::onData`. Remote blocks are already materialized over the wire, so it is a + cheap no-op there. +2. **User comes from the session, not a `ClientInfo`.** Passing a fabricated `ClientInfo` to + `sendQuery` made `LocalConnection` build the query context from it, blanking `currentUser()`. Pass + no `ClientInfo` so the local path derives the user from its authenticated session (and the remote + server defaults the query kind). +3. **No external-tables handshake locally.** `LocalConnection::sendExternalTablesData` is + `NOT_IMPLEMENTED` (and unneeded — there is no wire handshake), so skip it for `Type::LOCAL`. +4. **Cancel cannot rely on `poll` timing.** `Connection::poll` blocks on the backend socket, which is + what let the old code interleave a client-socket check; `LocalConnection::poll` returns immediately + with data, so the check must run on *every* packet iteration, not only when `poll` times out. + +Also: `LocalConnection::sendQuery` **ignores the per-query `settings`** and reads them from its +context, so `WSHandler` applies `send_logs_level` to the session context (that is how `?logs=` still +pushes `Log` frames in-server). Any future client-controllable setting needs the same treatment. + ## Key references - `programs/server/Server.cpp` — `http_port` / `tcp_port` `createServer` blocks (the port pattern). diff --git a/programs/wsproxy/tests/README.md b/programs/wsproxy/tests/README.md index 6c6d4509f195..a050cd521389 100644 --- a/programs/wsproxy/tests/README.md +++ b/programs/wsproxy/tests/README.md @@ -32,6 +32,21 @@ not already listening, creates the shared `default.wsp_test` table, and tears do it started. Override binary/config locations with the `CLICKHOUSE_SERVER`, `CH_CONFIG`, `WSPROXY_BIN`, and `WSPROXY_URL` environment variables. +### Running against the in-server `ws_port` + +The same query/protocol tests also run against the **in-server** WebSocket endpoint (a +`clickhouse-server` built with a `ws_port`, see `programs/wsproxy/IN_SERVER.md`) — the bridge code +is shared, so behavior should match: + +```bash +npm run test:server # vitest run --config vitest.server.config.mjs +``` + +This spawns the freshly-built `build/programs/clickhouse server` with a `ws_port` on `:9010` +(override the binary with `WS_SERVER_BIN`). Proxy-only suites (a separate remote backend, +proxy→backend TLS, and the `WSPROXY_BACKEND_COMPRESSION` codec) are excluded — see the config. +Current status: proxy suite 113/113, in-server suite 102/102. + ## Protocol under test The client sends a SQL query as a **text** frame; results stream back as **binary** frames encoded diff --git a/programs/wsproxy/tests/package.json b/programs/wsproxy/tests/package.json index de514a0e5b89..a54bb4affd88 100644 --- a/programs/wsproxy/tests/package.json +++ b/programs/wsproxy/tests/package.json @@ -6,7 +6,8 @@ "description": "Integration tests for clickhouse-wsproxy (WebSocket -> native protocol proxy)", "scripts": { "test": "vitest run", - "watch": "vitest" + "watch": "vitest", + "test:server": "vitest run --config vitest.server.config.mjs" }, "devDependencies": { "vitest": "^2.1.9" diff --git a/programs/wsproxy/tests/test/proc.mjs b/programs/wsproxy/tests/test/proc.mjs index 8e0009a95b40..eb546625e2da 100644 --- a/programs/wsproxy/tests/test/proc.mjs +++ b/programs/wsproxy/tests/test/proc.mjs @@ -13,6 +13,9 @@ const repoRoot = resolve(testDir, "../../../.."); const WSPROXY_BIN = process.env.WSPROXY_BIN ?? resolve(repoRoot, "build/programs/wsproxy/clickhouse-wsproxy"); const CLICKHOUSE_SERVER = process.env.CLICKHOUSE_SERVER ?? "/usr/local/bin/clickhouse-server"; +// The freshly-built multi-call binary (run as `clickhouse server`) — this is the one that +// carries the new ws_port. Point WS_SERVER_BIN elsewhere to test a different build. +const WS_SERVER_BIN = process.env.WS_SERVER_BIN ?? resolve(repoRoot, "build/programs/clickhouse"); function portOpen(port, host = "127.0.0.1") { return new Promise((res) => { @@ -174,3 +177,68 @@ export async function spawnProxy({ }, }; } + +/// Spawn the freshly-built clickhouse-server with the in-server WebSocket endpoint +/// (`ws_port`) enabled, alongside a `tcp_port` (the server requires at least one +/// query port to start). Same default + wsp_user users as spawnBackend, so the +/// auth tests work unchanged. Returns { url } pointing at the ws_port. +export async function spawnServerWithWs({ wsPort, tcpPort }) { + const dir = fs.mkdtempSync(join(os.tmpdir(), "wsproxy-server-")); + const configPath = join(dir, "config.xml"); + fs.writeFileSync( + configPath, + ` + warning0 + ${dir}/server.log${dir}/server.err.log + ${tcpPort} + ${wsPort} + 127.0.0.1 + ${dir}/data/ + ${dir}/tmp/ + ${dir}/user_files/ + 536870912 + false + users.xml + default + default +`, + ); + fs.writeFileSync( + join(dir, "users.xml"), + ` + + + + ::/0 + defaultdefault + 1 + + + wsp_pass::/0 + defaultdefault + + + +`, + ); + + const proc = spawn(WS_SERVER_BIN, ["server", "--config-file", configPath], { stdio: "ignore" }); + await waitForPort(wsPort, 30_000); + return { + url: `ws://127.0.0.1:${wsPort}`, + dir, + stop() { + // Kill whatever holds the ws port (watchdog fork/re-parent shenanigans). + for (const port of [wsPort, tcpPort]) { + const r = spawnSync("lsof", ["-ti", `tcp:${port}`, "-sTCP:LISTEN"], { encoding: "utf8" }); + for (const pid of (r.stdout || "").split("\n").map((s) => s.trim()).filter(Boolean)) { + try { + process.kill(Number(pid), "SIGKILL"); + } catch { + /* already gone */ + } + } + } + }, + }; +} diff --git a/programs/wsproxy/tests/test/serverSetup.mjs b/programs/wsproxy/tests/test/serverSetup.mjs new file mode 100644 index 000000000000..ee8ae365278f --- /dev/null +++ b/programs/wsproxy/tests/test/serverSetup.mjs @@ -0,0 +1,50 @@ +// Vitest global setup for the IN-SERVER WebSocket endpoint (ws_port), as opposed +// to the standalone proxy. Spawns the freshly-built clickhouse-server with a +// ws_port on the same PROXY_PORT the helpers use, so the existing test files run +// unchanged against `ws://127.0.0.1:9010`. Only the query/protocol tests apply +// here; proxy-specific suites (its own backend/TLS/compression) are excluded by +// vitest.server.config.mjs. + +import net from "node:net"; +import { spawnServerWithWs } from "./proc.mjs"; + +const TCP_PORT = 19000; +const WS_PORT = 9010; // matches PROXY_URL default in helpers.mjs + +function portOpen(port, host = "127.0.0.1") { + return new Promise((resolve) => { + const socket = net.connect({ port, host }, () => { + socket.destroy(); + resolve(true); + }); + socket.on("error", () => resolve(false)); + }); +} + +export default async function setup() { + const stops = []; + + if (!(await portOpen(WS_PORT))) { + const server = await spawnServerWithWs({ wsPort: WS_PORT, tcpPort: TCP_PORT }); + stops.push(() => server.stop()); + } + + // Create the shared test table over the in-server WebSocket (DDL via the query path). + const { runQuery } = await import("./helpers.mjs"); + const ddl = await runQuery( + "CREATE TABLE IF NOT EXISTS default.wsp_test (a UInt32, b String) ENGINE = Memory", + ); + if (ddl.control.event !== "end") { + throw new Error(`failed to create test table: ${JSON.stringify(ddl.control)}`); + } + + return () => { + for (const stop of stops) { + try { + stop(); + } catch { + /* ignore */ + } + } + }; +} diff --git a/programs/wsproxy/tests/vitest.server.config.mjs b/programs/wsproxy/tests/vitest.server.config.mjs new file mode 100644 index 000000000000..1bf42ef1236a --- /dev/null +++ b/programs/wsproxy/tests/vitest.server.config.mjs @@ -0,0 +1,26 @@ +import { defineConfig } from "vitest/config"; + +// Runs the query/protocol test suite against the IN-SERVER WebSocket endpoint +// (clickhouse-server with a ws_port) instead of the standalone proxy. The bridge +// code is shared, so the same tests should pass. Proxy-specific suites are +// excluded: they spawn their own proxy and exercise features that only exist in +// the sidecar (a separate remote backend, proxy->backend TLS, and the +// WSPROXY_BACKEND_COMPRESSION codec). +// +// Run with: npx vitest run --config vitest.server.config.mjs +export default defineConfig({ + test: { + globalSetup: "./test/serverSetup.mjs", + fileParallelism: false, + testTimeout: 30_000, + hookTimeout: 60_000, + include: ["test/**/*.test.mjs"], + exclude: [ + "test/backend-failure.test.mjs", // kills a separate backend + "test/resilience.test.mjs", // points a proxy at a dead backend + "test/tls.test.mjs", // proxy->backend TLS (N/A in-server) + "test/backpressure.test.mjs", // spawns its own proxy with a send timeout + "test/compression.test.mjs", // WSPROXY_BACKEND_COMPRESSION (proxy-only) + ], + }, +}); From 0a7e4df016861054589355feda5ba4c0f835da59 Mon Sep 17 00:00:00 2001 From: Peter Leonov Date: Thu, 16 Jul 2026 18:36:01 +0200 Subject: [PATCH 45/46] Document ws_port in the default server config Add a commented entry (with a short protocol description) next to tcp_port in programs/server/config.xml, so the in-server WebSocket interface is discoverable. Disabled unless set. Co-Authored-By: Claude Opus 4.8 (1M context) --- programs/server/config.xml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/programs/server/config.xml b/programs/server/config.xml index 599cdb2d805c..ccf730fedcd7 100644 --- a/programs/server/config.xml +++ b/programs/server/config.xml @@ -182,6 +182,14 @@ --> 9000 + + + + + +