diff --git a/.gitignore b/.gitignore index 6f33e7e83db8..1204fd3b20d4 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,7 @@ ci/local.env .env /tests/integration/.env /ci/tmp +/tmp /tests/venv /obj-x86_64-linux-gnu/ diff --git a/TODO.md b/TODO.md new file mode 100644 index 000000000000..1bbaba8f9896 --- /dev/null +++ b/TODO.md @@ -0,0 +1,112 @@ +# WebSocket query interface + +## Selected scope + +The upstream target for this branch is the in-server `ws_port` in +`clickhouse-server`. It exposes the shared WebSocket query protocol through an +in-process `LocalConnection` and keeps authentication, query execution, and format +conversion inside the server. + +The standalone `clickhouse-wsproxy` binary remains in the branch as a compatible +prototype and test peer. It exercises the same `WebSocketFrames` and +`WebSocketSession` implementation through a remote `Connection`, but productionizing +or deploying that binary is not part of the current upstream scope. + +Cloudflare deployment is analysis only. Nothing under that deployment design has +been built; see `programs/wsproxy/CLOUDFLARE.md`. + +## Current implementation + +- `ws_port` is registered as a dedicated HTTP-upgrade listener in + `clickhouse-server`. +- `WSHandler` authenticates the upgrade request, creates a server session and a + `LocalConnection`, then invokes `WebSocketSession::run`. +- `WebSocketSession` supports query results, streamed inserts, progress, logs, + profile events, cancellation, output formats, optional SQL classification, + optional parallel formatting, and credit-based flow control. +- `WebSocketFrames` implements the RFC 6455 frame layer shared by the server and + standalone proxy. +- Browser upgrades with an `Origin` header are same-origin by default. + `ws_allowed_origins` configures a comma-separated allowlist for `ws_port`, and + `WSPROXY_ALLOWED_ORIGINS` provides the equivalent standalone-proxy setting. + Requests without `Origin` remain valid for non-browser clients. +- Explicit Basic authentication fails closed when its header is malformed. +- The initial `?flow=N` credit and later credit grants must be positive, fit in + `Int64`, and cannot overflow the accumulated credit. +- Client messages have a cumulative 16 MiB limit across fragments. Mid-query + frames must complete within one second after the socket becomes readable. +- Client write failures cancel the active query. The standalone proxy also bounds + stalled client writes with `WSPROXY_CLIENT_SEND_TIMEOUT_SEC`. +- Protocol violations, invalid close frames, oversized messages, and invalid + flow-control grants use RFC-appropriate close handling. +- The Node.js/Vitest suite runs the common protocol cases against both `ws_port` + and `clickhouse-wsproxy`. Adversarial cases cover fragmented message limits, + partial frames, interleaved control frames, invalid credit, origin policy, and + malformed credentials. Proxy-only cases cover the remote-backend boundary. + +`ws_port` currently speaks plaintext WebSocket (`ws://`). It does not provide a +`ws_port_secure` listener. Deployments that expose the port outside a trusted +network must terminate TLS at an ingress, load balancer, or other trusted proxy. + +## Protocol outline + +- A text frame containing SQL runs a normal query. Results are sent as binary + frames in the requested ClickHouse output format. +- A text control message with `{"cmd":"insert", ...}` starts a streamed insert; + binary frames carry input data and an empty binary frame ends the input. +- Text control frames report progress, logs, profile events, query classification, + and terminal `end`, `error`, or `cancelled` events. +- Closing the WebSocket while a query is running cancels the query. +- `?flow=N` enables frame-credit flow control. `?parallel=1` enables parallel + output formatting when flow control is disabled. + +The standalone proxy and `ws_port` intentionally use the same application +protocol. Features that depend on a remote native-protocol hop, such as backend +TLS and selecting its compression codec, apply only to `clickhouse-wsproxy`. + +## Work required before upstream review + +- Move the protocol coverage into repository-native ClickHouse CI. The current + Vitest suites are useful development coverage but are not yet CI gates. +- Define general connection, session, memory, and concurrency limits for + `ws_port` and the standalone proxy. +- Add graceful drain behavior so shutdown stops accepting new upgrades, cancels + or completes active work according to policy, and closes sessions predictably. +- Add configuration documentation for `ws_port`, its plaintext-only transport, + origin policy, resource limits, and TLS-termination expectations. +- Decide whether a native TLS listener is required for `ws_port`; until then, + document and validate the supported TLS-termination topology. +- Replace the standalone proxy's development-only invalid-certificate mode with + explicit CA-based backend verification before any production deployment. +- Decide whether the opt-in SQL classifier, parallel formatter, and application + flow-control extension belong in the first upstream version or a follow-up. + +## Deliberately out of scope + +- A production Cloudflare Workers/Containers deployment. +- Production hardening of `clickhouse-wsproxy`, including `BaseDaemon` + integration and a configuration file. +- A native TLS listener named `ws_port_secure`. +- Backend connection pooling or replica load balancing. The standalone proxy + keeps one remote `Connection` per WebSocket session. + +## Development validation + +The test harness is documented in `programs/wsproxy/tests/README.md`. The primary +command for the selected scope is: + +```bash +cd programs/wsproxy/tests +npm run test:server +``` + +The compatibility suite for the standalone prototype is: + +```bash +cd programs/wsproxy/tests +npm test +``` + +Historical benchmark notes remain under `programs/wsproxy/bench/`; they motivate +the sidecar experiment but are not acceptance criteria for the in-server +`ws_port`. diff --git a/programs/CMakeLists.txt b/programs/CMakeLists.txt index c44a71c1a85a..bd1f7fa9ad02 100644 --- a/programs/CMakeLists.txt +++ b/programs/CMakeLists.txt @@ -25,6 +25,8 @@ option (ENABLE_CLICKHOUSE_KEEPER_CONVERTER "Util allows to convert ZooKeeper log option (ENABLE_CLICKHOUSE_KEEPER_CLIENT "ClickHouse Keeper Client" ${ENABLE_CLICKHOUSE_ALL}) +option (ENABLE_CLICKHOUSE_WSPROXY "Standalone WebSocket proxy (native protocol <-> WebSocket, edge format conversion)" ${ENABLE_CLICKHOUSE_ALL}) + if (NOT ENABLE_NURAFT) # RECONFIGURE_MESSAGE_LEVEL should not be used here, # since ENABLE_NURAFT is set to OFF for FreeBSD and Darwin. @@ -126,6 +128,12 @@ if (ENABLE_CLICKHOUSE_KEEPER) add_subdirectory (keeper) endif() +# Standalone binary with its own `main` and `add_executable` (see wsproxy/CMakeLists.txt), +# so it is added here rather than via the multi-call `clickhouse_program_install` list below. +if (ENABLE_CLICKHOUSE_WSPROXY) + add_subdirectory (wsproxy) +endif() + if (ENABLE_CLICKHOUSE_SELF_EXTRACTING AND NOT ENABLE_DUMMY_LAUNCHERS) add_subdirectory (self-extracting) endif () diff --git a/programs/server/Server.cpp b/programs/server/Server.cpp index 1f4f9d6c2c6f..52b5d64ede7a 100644 --- a/programs/server/Server.cpp +++ b/programs/server/Server.cpp @@ -3653,6 +3653,29 @@ void Server::createServers( }); } + if (server_type.shouldStart(ServerType::Type::WS)) + { + /// WebSocket. An HTTP server whose handler upgrades the connection to a WebSocket and + /// bridges it to an in-process query connection (same wire protocol as clickhouse-wsproxy): + /// query as a text frame, results as binary frames in the ?format= output format, plus + /// mid-query progress/logs/profile-events and cancel-by-close. + port_name = "ws_port"; + createServer(config, listen_host, port_name, listen_try, start_servers, servers, [&](UInt16 port) -> ProtocolServerAdapter + { + Poco::Net::ServerSocket socket; + auto address = socketBindListen(server_settings, socket, listen_host, port); + socket.setReceiveTimeout(settings[Setting::http_receive_timeout]); + socket.setSendTimeout(settings[Setting::http_send_timeout]); + + return ProtocolServerAdapter( + listen_host, + port_name, + "websocket: " + address.toString(), + std::make_unique( + httpContext(), createHandlerFactory(*this, config, async_metrics, "WSHandler-factory"), server_pool, socket, http_params, connection_filter, ProfileEvents::InterfaceHTTPReceiveBytes, ProfileEvents::InterfaceHTTPSendBytes)); + }); + } + if (server_type.shouldStart(ServerType::Type::TCP)) { /// TCP diff --git a/programs/server/config.xml b/programs/server/config.xml index 599cdb2d805c..ef46a6ab8b73 100644 --- a/programs/server/config.xml +++ b/programs/server/config.xml @@ -182,6 +182,20 @@ --> 9000 + + + + + + Worker (true edge PoP) --CF backbone--> Container (regional) --native+zstd (WAN)--> ClickHouse Cloud + stateless router stateful proxy = a Durable Object +``` + +### A Container *is* a Durable Object (verified) + +The `Container` class from the [`@cloudflare/containers`](https://github.com/cloudflare/containers) +library **extends `DurableObject`**. Cloudflare's docs state it directly: *"a request passes through +a Durable Object instance (the Container class extends a Durable Object class)."* Each container +instance is backed by **exactly one** DO instance — the DO is the programmable sidecar that owns the +container's lifecycle and routing; the container is the workload attached to it. + +Consequences: + +- Your container subclass gets a free per-instance place for coordination logic — warm-up, health, + draining, `alarm`-based idle handling, small storage — with no extra moving part. +- The DO is single-threaded JS, but it is **not** in the per-frame data path: the client WebSocket + rides the container's exposed port via the stub's `fetch`; the DO only manages lifecycle. So + "DO is single-threaded" is **not** a throughput bottleneck for the proxy — as long as you forward + to the container port and do not proxy bytes through DO JS. +- **A DO and its container may run in different locations** (placement is optimized for routing and + startup). This feeds the placement nuance below. + +## Running many proxy containers (scaling out) + +The proxy's sessions are **interchangeable** — each is just a fresh native `Connection`, a stateless +pool — which is exactly the case the platform's helpers target: + +- `getRandom(env.WSPROXY, N)` — picks a random instance out of `N`. The stateless load balancer, and + the right fit here: each client WS upgrade → random container → new backend connection. The + WebSocket then naturally pins to that instance for its lifetime. +- `getContainer(env.WSPROXY, name)` — addresses a *specific* named instance by ID (sticky/stateful + routing). Probably unneeded for the proxy, but available. +- Multiple instances = multiple DO IDs; the library manages spin-up, pulling pre-fetched images at + other locations for fast cold starts. + +The Worker glue is small: + +```js +import { getRandom } from "@cloudflare/containers"; + +export default { + async fetch(request, env) { + // Optional: authenticate here, or let creds pass through to the container -> ClickHouse. + const instance = getRandom(env.WSPROXY, N); // N interchangeable proxy containers + return instance.fetch(request); // forwards the WS upgrade to the container port + }, +}; +``` + +**The catch — no autoscaling yet.** You pick and manage `N` yourself (a fixed fleet, or your own +logic). Cloudflare says built-in autoscaling is planned but not available today. Capacity ≈ +`N × sessions-per-container`, so size `N` against a per-container session cap (see the thread-per- +session note below). Until autoscaling lands, elasticity is on you — over-provision a fixed `N`, or +build a small coordinator (itself a DO) that adjusts routing under load. + +## Concrete work items + +1. **x86_64 Linux build** *(the long pole).* Everything so far was built on macOS/arm64; Cloudflare + Containers are **amd64 Linux only**. ClickHouse builds amd64 Linux routinely (CI does), so this is + well-trodden — just not yet done for this binary. +2. **Dockerfile.** A slim/distroless base + the ~306 MB binary + env; expose the listen port + (`WSPROXY_PORT`), set a `CMD`. Image ≈ 400-500 MB (under the platform's GB-scale image limits — + verify current limits). +3. **Worker + Durable Object + `wrangler` config.** ~50-150 lines: accept the WS upgrade, `getRandom` + to a container, forward. Secrets (backend password) via Worker/container secrets → container env. +4. **Secrets & egress.** Map `WSPROXY_BACKEND_*` into the container; settle ClickHouse-side source-IP + policy (see egress gotcha). + +No delivery estimate is asserted until the Linux image, Worker routing, current platform limits, +and end-to-end WebSocket behavior have been validated. + +## Frictions / gotchas (Cloudflare-specific) + +- **Thread-per-session vs small containers *(biggest one)*.** The proxy is thread-per-session blocking + IO with default ~8 MB stacks → ~100 sessions ≈ ~800 MB just in stacks, and container instances are + memory-capped. Sessions-per-container is therefore bounded; lean on `getRandom` across instances + and/or shrink the thread stack size (a small proxy change worth doing before serious fan-in). +- **CPU limits cap the conversion win.** Formatting is CPU-heavy and `?parallel` wants multiple cores, + but instance types meter vCPU. Right-size the instance; the parallel-formatting speedup is bounded + by the vCPU allotted. +- **Egress IP allowlisting.** The container's outbound to ClickHouse Cloud uses Cloudflare's + shared/dynamic egress IPs. If the service restricts source IPs, you need CH-side allow-all or + Cloudflare egress ranges — settle this early (security posture). +- **Cold starts.** Containers sleep on idle; the first connection pays container boot + proxy start + + backend handshake (seconds). Eager-connect adds to that. +- **Placement affects the compression benefit.** Containers run in *regional* locations, not literally + every edge PoP, and may not co-locate with their DO. A container near the ClickHouse region → short + compressed hop (smaller win); near the user → long compressed hop (bigger win). Less placement + control than k8s. +- **Keep the DO out of the per-frame data path** (forward to the container port; do not proxy frames + through DO JS). + +## Model shift: 1:1 sidecar → shared multi-tenant + +On k8s the proxy is a strict 1:1 sidecar (one proxy per app). On Cloudflare you would run a **shared +multi-tenant** pool (many sessions per container, the Worker fanning out across instances). The proxy +already supports this safely: it does **per-session credential pass-through** and keeps zero +cross-session state, so multi-tenant is fine as long as each session carries its own credentials. + +## Alternative considered — pure Worker (no container) + +Workers can open outbound TCP (`connect` from `cloudflare:sockets`), so in principle the proxy could +live entirely in a Worker. Rejected: it would require reimplementing the ClickHouse native protocol +**and** the full format matrix in JS/WASM. Compiling ClickHouse itself to WASM is not feasible. The +container is the right vehicle precisely because it runs the real C++ binary with full format +coverage. + +## Sources + +- [Lifecycle of a Container — architecture](https://developers.cloudflare.com/containers/platform-details/architecture/) +- [Scaling and Routing](https://developers.cloudflare.com/containers/platform-details/scaling-and-routing/) +- [`cloudflare/containers` library README](https://github.com/cloudflare/containers/blob/main/README.md) +- [Containers overview](https://developers.cloudflare.com/containers/) +- [Containers coming to Workers (announcement)](https://blog.cloudflare.com/cloudflare-containers-coming-2025/) + +*(Cloudflare Containers are a recently-GA product; exact limits and APIs move — verify image-size, +memory, vCPU, WS-to-container specifics, and egress ranges against the live docs before committing.)* diff --git a/programs/wsproxy/CMakeLists.txt b/programs/wsproxy/CMakeLists.txt new file mode 100644 index 000000000000..742921545f7d --- /dev/null +++ b/programs/wsproxy/CMakeLists.txt @@ -0,0 +1,33 @@ +set (CLICKHOUSE_WSPROXY_SOURCES + WsProxy.cpp + WsProxyHandler.cpp +) +# WebSocketFrames + WebSocketSession (the transport-agnostic WS<->native bridge) now live in +# src/Server (compiled into `dbms`), shared with the in-server WebSocket endpoint. This binary +# links `dbms`, so it picks them up automatically. + +set (CLICKHOUSE_WSPROXY_LINK + PRIVATE + clickhouse_aggregate_functions + clickhouse_common_config + clickhouse_common_io + clickhouse_functions + dbms +) + +# A standalone binary, deliberately NOT wired into the multi-call `clickhouse` +# dispatch: there is no `mainEntryClickHouseWsProxy` in programs/main.cpp and no +# `clickhouse_program_install` line for it. Omitting both is what keeps it a +# separate executable with its own `main`, which in turn keeps the diff to +# shared dispatch code at zero. This mirrors the standalone-keeper pattern +# (`BUILD_STANDALONE_KEEPER`), except we do not also produce a multi-call lib. +# +# Note: linking `dbms` is unavoidable (it is where `Connection`, `FormatFactory` +# and the HTTP server infrastructure live), so this binary is roughly the size +# of the full `clickhouse` binary. The separate target buys an independently +# deployable artifact, not a smaller one. +clickhouse_add_executable (clickhouse-wsproxy ${CLICKHOUSE_WSPROXY_SOURCES}) +target_link_libraries (clickhouse-wsproxy ${CLICKHOUSE_WSPROXY_LINK}) +# So the program's own headers resolve as angled includes (matches the keeper +# standalone pattern). +target_include_directories (clickhouse-wsproxy PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) diff --git a/programs/wsproxy/IN_SERVER.md b/programs/wsproxy/IN_SERVER.md new file mode 100644 index 000000000000..a2825e99cf0f --- /dev/null +++ b/programs/wsproxy/IN_SERVER.md @@ -0,0 +1,119 @@ +# In-server WebSocket port + +The primary upstream scope of this branch is a dedicated `ws_port` in +`clickhouse-server`. The standalone `clickhouse-wsproxy` binary remains a +compatible prototype that drives the same WebSocket session implementation over a +remote native-protocol `Connection`. + +## Status + +The prototype is implemented. `Server::main` creates an HTTP server for +`ws_port`, `HTTPHandlerFactory` routes every request on that listener to +`WSHandler`, and the handler runs a `WebSocketSession` over an in-process +`LocalConnection`. + +The shared Node.js/Vitest protocol suite can run against either implementation. +The in-server configuration excludes only tests that inherently require a +separate proxy or remote backend, such as proxy-to-backend TLS and backend codec +selection. See `tests/README.md` for the current commands and prerequisites. + +This remains a prototype rather than a production-ready interface. The branch now +includes origin and credential enforcement, bounded fragmented messages and +partial-frame reads, safe flow-control accounting, cancellation on client write +failure, RFC close handling, and adversarial tests. Remaining TLS, general +resource-limit, graceful-drain, and ClickHouse CI work is tracked in the +repository-level `TODO.md`. + +## Architecture + +`ws_port` is an HTTP listener because WebSocket connections begin with an HTTP +Upgrade request. Its request path is: + +```text +client + -> ws_port + -> WSHandler + -> authenticated Session + -> LocalConnection + -> WebSocketSession + -> query pipeline +``` + +The shared bridge keeps framing and application-protocol behavior aligned between +the two executables: + +- `WebSocketFrames` reads and writes RFC 6455 frames. +- `WebSocketSession::run` maps WebSocket messages to `IServerConnection` packets + and formats query results. +- `WSHandler` supplies an authenticated `LocalConnection`. +- `WsProxyHandler` supplies a remote `Connection` in the standalone prototype. + +The in-server path does not pass credentials to another service. It authenticates +the upgrade request through the server session and executes under that session's +user and settings. + +Browser requests that include an `Origin` header must be same-origin by default. +Set `ws_allowed_origins` to a comma-separated list of allowed HTTP or HTTPS origins +when clients are intentionally hosted elsewhere. Origins are normalized before +comparison. A missing `Origin` is accepted so non-browser WebSocket clients can +connect. + +## Transport security + +`ws_port` currently supports plaintext WebSocket (`ws://`) only. There is no +`ws_port_secure` implementation in this branch. Exposing the listener beyond a +trusted network therefore requires TLS termination at a trusted ingress, load +balancer, or reverse proxy, with the origin and forwarded-request policy configured +explicitly. + +The standalone proxy's backend TLS options protect a different connection: the +native-protocol hop from `clickhouse-wsproxy` to a remote ClickHouse server. They do +not add TLS to `ws_port`. + +## Differences between local and remote connections + +The bridge targets `IServerConnection`, but `LocalConnection` has several relevant +behavioral differences from a remote `Connection`: + +1. Local blocks must be materialized before row-format output. A local pipeline can + return a `ColumnConst`, while blocks decoded from the native protocol are already + materialized. +2. The authenticated user must come from the server session. Supplying a fabricated + `ClientInfo` to `LocalConnection::sendQuery` can replace the query context's user. +3. `LocalConnection` does not need the native protocol's external-table handshake; + `LocalConnection::sendExternalTablesData` is not implemented. +4. `LocalConnection::poll` yields after an executor timeout so the bridge can + inspect the WebSocket between packets. `LocalConnection::receivePacket` retains + blocking semantics for direct consumers. +5. Query settings used by `LocalConnection` must be applied to its context. For + example, `WSHandler` applies `send_logs_level` to the authenticated session + context. + +These are implementation constraints of the shared bridge, not alternate protocol +behaviors for clients. + +## Product boundary + +The in-server interface provides bidirectional streaming, cancellation by close, +and mid-query progress, log, and profile-event delivery without another deployed +service. Format conversion happens on the ClickHouse server. + +The standalone sidecar explores a different deployment tradeoff: it can perform +format conversion away from the server and use a compressed native-protocol hop to +a remote backend. That experiment is compatible with the in-server protocol but is +not the selected upstream deliverable. + +The Cloudflare Workers/Containers design in `CLOUDFLARE.md` is unbuilt analysis for +the sidecar and is not part of `ws_port`. + +## Key files + +- `programs/server/Server.cpp` registers the `ws_port` listener. +- `src/Server/ServerType.h` defines the `WS` server type. +- `src/Server/HTTPHandlerFactory.cpp` creates the `WSHandler` factory. +- `src/Server/WSHandler.{h,cpp}` authenticates and upgrades requests. +- `src/Server/WebSocketFrames.{h,cpp}` implements WebSocket framing. +- `src/Server/WebSocketSession.{h,cpp}` implements the shared query protocol. +- `src/Client/LocalConnection.h` provides in-process execution. +- `programs/wsproxy/WsProxyHandler.{h,cpp}` adapts the same session to a remote + backend for the standalone prototype. diff --git a/programs/wsproxy/WsProxy.cpp b/programs/wsproxy/WsProxy.cpp new file mode 100644 index 000000000000..a91188e3b021 --- /dev/null +++ b/programs/wsproxy/WsProxy.cpp @@ -0,0 +1,198 @@ +#include +#include + +#include + +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include +#include +#include + +#include + +#include + +#include + +#include +#include +#include + + +/// Standalone WebSocket proxy. +/// +/// Deliberately a separate binary, kept out of the multi-call `clickhouse` +/// dispatch (no `mainEntryClickHouseWsProxy`, no `clickhouse_program_install`). +/// +/// Step 2 scope: bring up the global `Context` and the format machinery, then +/// serve a real WebSocket endpoint (`WsProxyHandler`) that completes the RFC +/// 6455 handshake and echoes messages. Step 3 will replace the echo loop with a +/// native-protocol `Connection` to a backend server. +/// +/// Signal handling: we block SIGINT/SIGTERM in the main thread *before* the +/// server spawns any worker threads, so the workers inherit the blocked mask. +/// A process-directed signal then stays pending until `waitForTerminationRequest` +/// consumes it via `sigwait`, giving a clean shutdown instead of the default +/// "terminate" disposition firing on a worker thread. (The heavier alternative +/// is to base this on `BaseDaemon`, which also provides config/logging/crash +/// handling; deferred until the proxy needs those.) + +namespace DB +{ + +namespace +{ + +class WsProxyHandlerFactory : public HTTPRequestHandlerFactory +{ +public: + WsProxyHandlerFactory(ContextPtr context_, BackendParams backend_) + : context(std::move(context_)), backend(std::move(backend_)) + { + } + + std::unique_ptr createRequestHandler(const HTTPServerRequest &) override + { + return std::make_unique(context, backend); + } + +private: + ContextPtr context; + BackendParams backend; +}; + +/// Read a string environment variable, falling back to `def` when unset/empty. +std::string envOr(const char * name, const std::string & def) +{ + const char * value = std::getenv(name); + return (value && *value) ? std::string(value) : def; +} + +/// A boolean env var: true for "1"/"true"/"yes" (case-insensitive), else false. +bool envBool(const char * name) +{ + std::string v = envOr(name, ""); + std::transform(v.begin(), v.end(), v.begin(), ::tolower); + return v == "1" || v == "true" || v == "yes"; +} + +BackendParams backendParamsFromEnv() +{ + BackendParams params; + params.host = envOr("WSPROXY_BACKEND_HOST", params.host); + params.port = static_cast(std::stoul(envOr("WSPROXY_BACKEND_PORT", "9000"))); + params.user = envOr("WSPROXY_BACKEND_USER", params.user); + params.password = envOr("WSPROXY_BACKEND_PASSWORD", params.password); + params.database = envOr("WSPROXY_BACKEND_DATABASE", params.database); + params.secure = envBool("WSPROXY_BACKEND_SECURE"); + /// Native result-block compression codec: lz4 (default) | zstd | none. ZSTD trades backend + /// CPU for far fewer bytes on the wire — recommended for bandwidth-limited (WAN) deployments. + params.compression_method = envOr("WSPROXY_BACKEND_COMPRESSION", params.compression_method); + return params; +} + +/// Conservative HTTP limits/timeouts for the skeleton; a later revision should +/// source these from configuration. +class WsProxyHTTPContext : public IHTTPContext +{ +public: + uint64_t getMaxHstsAge() const override { return 0; } + uint64_t getMaxUriSize() const override { return 1024 * 1024; } + uint64_t getMaxFields() const override { return 1'000'000; } + uint64_t getMaxFieldNameSize() const override { return 128 * 1024; } + uint64_t getMaxFieldValueSize() const override { return 128 * 1024; } + uint64_t getMaxRequestHeaderSize() const override { return 8 * 1024 * 1024; } + Poco::Timespan getHeadersReadTimeout() const override { return {30, 0}; } + Poco::Timespan getReceiveTimeout() const override { return {30, 0}; } + Poco::Timespan getSendTimeout() const override { return {30, 0}; } +}; + +} + +class WsProxyServer : public Poco::Util::ServerApplication +{ +protected: + int main(const std::vector &) override + { + LoggerPtr log = getLogger("WsProxy"); + + /// Block termination signals before any worker threads are created, so + /// they inherit the mask and `waitForTerminationRequest` can consume the + /// signal cleanly. See the file header for the rationale. + sigset_t sigset{}; + sigemptyset(&sigset); + sigaddset(&sigset, SIGINT); + sigaddset(&sigset, SIGTERM); + pthread_sigmask(SIG_BLOCK, &sigset, nullptr); + + /// Minimal global state the format machinery needs. + shared_context = Context::createShared(); + global_context = Context::createGlobal(shared_context.get()); + global_context->makeGlobalContext(); + global_context->setApplicationType(Context::ApplicationType::SERVER); + + /// The same registrations clickhouse-client performs: aggregate functions + /// are required to (de)serialize `AggregateFunction`-typed columns off the + /// native wire, and functions back defaults/codecs. `registerFormats` alone + /// is not enough for full type coverage. + registerFunctions(); + registerAggregateFunctions(); + registerFormats(); + + const BackendParams backend = backendParamsFromEnv(); + + /// For a self-signed / dev backend certificate, configure the SSL client + /// context to accept invalid certificates (mirrors clickhouse-client's + /// --accept-invalid-certificate). Poco's SSLManager builds the default + /// client context lazily from this config on the first secure connect. + if (envBool("WSPROXY_BACKEND_ACCEPT_INVALID_CERT")) + { + config().setString("openSSL.client.invalidCertificateHandler.name", "AcceptCertificateHandler"); + config().setString("openSSL.client.verificationMode", "none"); + } + + const UInt16 port = static_cast(std::stoul(envOr("WSPROXY_PORT", "9010"))); + Poco::Net::ServerSocket socket(port); + Poco::ThreadPool server_pool(/* minCapacity= */ 1, /* maxCapacity= */ 16); + Poco::Net::HTTPServerParams::Ptr params(new Poco::Net::HTTPServerParams); + + HTTPServer server( + std::make_shared(), + std::make_shared(global_context, backend), + server_pool, + socket, + params); + + server.start(); + LOG_INFO(log, "clickhouse-wsproxy listening on port {}; backend {}:{}", port, backend.host, backend.port); + + waitForTerminationRequest(); + + LOG_INFO(log, "Shutting down"); + server.stop(); + return Application::EXIT_OK; + } + +private: + SharedContextHolder shared_context; + ContextMutablePtr global_context; +}; + +} + +int main(int argc, char ** argv) +{ + DB::WsProxyServer app; + return app.run(argc, argv); +} diff --git a/programs/wsproxy/WsProxyHandler.cpp b/programs/wsproxy/WsProxyHandler.cpp new file mode 100644 index 000000000000..8ee88f67636f --- /dev/null +++ b/programs/wsproxy/WsProxyHandler.cpp @@ -0,0 +1,521 @@ +#include + +#include +#include + +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include + +#include +#include + +#include +#include +#include + +#include + +#include +#include +#include +#include +#include +#include +#include + + +namespace DB +{ + +using namespace DB::WsProxy; + +namespace ErrorCodes +{ + extern const int BAD_ARGUMENTS; +} + +namespace +{ + +/// Minimal JSON string escaping for a short error message embedded in a control frame. +String jsonEscape(const String & s) +{ + String out; + out.reserve(s.size() + 2); + for (char c : s) + { + switch (c) + { + case '"': out += R"(\")"; break; + case '\\': out += R"(\\)"; break; + case '\n': out += R"(\n)"; break; + case '\r': out += R"(\r)"; break; + case '\t': out += R"(\t)"; break; + default: + if (static_cast(c) < 0x20) + { + static const char * hex = "0123456789abcdef"; + out += R"(\u00)"; + out += hex[(c >> 4) & 0xF]; + out += hex[c & 0xF]; + } + else + out += c; + } + } + return out; +} + +/// Per RFC 7230 the `Connection` header is a comma-separated token list. Match +/// the exact `upgrade` token rather than a substring (which would also accept +/// unrelated values containing those letters). +bool hasUpgradeToken(const String & connection_header) +{ + String value = connection_header; + std::transform(value.begin(), value.end(), value.begin(), ::tolower); + + size_t pos = 0; + while (pos <= value.size()) + { + size_t comma = value.find(',', pos); + size_t end_pos = (comma == String::npos) ? value.size() : comma; + size_t start = value.find_first_not_of(" \t", pos); + if (start != String::npos && start < end_pos) + { + size_t last = value.find_last_not_of(" \t", end_pos - 1); + if (value.substr(start, last - start + 1) == "upgrade") + return true; + } + if (comma == String::npos) + break; + pos = comma + 1; + } + return false; +} + +/// Read a query parameter from the WebSocket URL, returning `fallback` if absent. +String queryParam(const String & uri_string, const String & name, const String & fallback) +{ + Poco::URI uri(uri_string); + for (const auto & param : uri.getQueryParameters()) + { + if (param.first == name && !param.second.empty()) + return param.second; + } + return fallback; +} + +std::optional positiveInt64QueryParam(const String & uri_string, const String & name) +{ + Poco::URI uri(uri_string); + std::optional result; + for (const auto & param : uri.getQueryParameters()) + { + if (param.first != name) + continue; + + if (result) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Query parameter `{}` must not be repeated", name); + + Int64 value = 0; + const char * begin = param.second.data(); + const char * end = begin + param.second.size(); + const auto parse_result = std::from_chars(begin, end, value, 10); + if (param.second.empty() || parse_result.ec != std::errc{} || parse_result.ptr != end || value <= 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Query parameter `{}` must be a positive integer", name); + result = value; + } + return result; +} + +Int64 clientSendTimeoutSeconds() +{ + constexpr Int64 default_timeout = 30; + constexpr Int64 max_timeout = 86'400; + const char * value_string = std::getenv("WSPROXY_CLIENT_SEND_TIMEOUT_SEC"); + if (!value_string || !*value_string) + return default_timeout; + + Int64 value = 0; + const char * end = value_string + std::strlen(value_string); + const auto parse_result = std::from_chars(value_string, end, value, 10); + if (parse_result.ec != std::errc{} || parse_result.ptr != end || value <= 0 || value > max_timeout) + throw Exception( + ErrorCodes::BAD_ARGUMENTS, + "`WSPROXY_CLIENT_SEND_TIMEOUT_SEC` must be an integer between 1 and {}", + max_timeout); + return value; +} + +String normalizeOrigin(const String & origin) +{ + Poco::URI uri(origin); + String scheme = uri.getScheme(); + String host = uri.getHost(); + std::transform(scheme.begin(), scheme.end(), scheme.begin(), ::tolower); + std::transform(host.begin(), host.end(), host.begin(), ::tolower); + + if ((scheme != "http" && scheme != "https") || host.empty() || !uri.getUserInfo().empty() + || (!uri.getPath().empty() && uri.getPath() != "/") || !uri.getQuery().empty() || !uri.getFragment().empty()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Origin header"); + + const UInt16 port = uri.getPort(); + const UInt16 default_port = scheme == "https" ? 443 : 80; + if (host.find(':') != String::npos) + host = "[" + host + "]"; + return scheme + "://" + host + (port && port != default_port ? ":" + std::to_string(port) : ""); +} + +bool originAllowed(const HTTPServerRequest & request, const String & allowed_origins, LoggerPtr log) +{ + const String origin = request.get("Origin", ""); + if (origin.empty()) + return true; + + String normalized_origin; + try + { + normalized_origin = normalizeOrigin(origin); + } + catch (...) + { + LOG_WARNING(log, "WebSocket upgrade rejected: malformed Origin header"); + return false; + } + + if (!allowed_origins.empty()) + { + bool allowed = false; + size_t pos = 0; + while (pos <= allowed_origins.size()) + { + const size_t comma = allowed_origins.find(',', pos); + const size_t end_pos = comma == String::npos ? allowed_origins.size() : comma; + const size_t start = allowed_origins.find_first_not_of(" \t", pos); + if (start == String::npos || start >= end_pos) + { + LOG_WARNING(log, "WebSocket upgrade rejected: empty origin in `WSPROXY_ALLOWED_ORIGINS`"); + return false; + } + const size_t last = allowed_origins.find_last_not_of(" \t", end_pos - 1); + try + { + if (normalizeOrigin(allowed_origins.substr(start, last - start + 1)) == normalized_origin) + allowed = true; + } + catch (...) + { + LOG_WARNING(log, "WebSocket upgrade rejected: malformed origin in `WSPROXY_ALLOWED_ORIGINS`"); + return false; + } + if (comma == String::npos) + break; + pos = comma + 1; + } + return allowed; + } + + try + { + const String request_scheme = request.isSecure() ? "https" : "http"; + return normalizeOrigin(request_scheme + "://" + request.getHost()) == normalized_origin; + } + catch (...) + { + LOG_WARNING(log, "WebSocket upgrade rejected: malformed Host header"); + return false; + } +} + +/// Resolve backend credentials for this session (credential pass-through: the +/// backend performs authentication). Priority: `Authorization: Basic`, then +/// `X-ClickHouse-User`/`-Key` headers, then `?user=`/`?password=` URL params, +/// else the configured defaults already in `backend`. +void resolveCredentials(const HTTPServerRequest & request, const String & uri, BackendParams & backend) +{ + const String auth = request.get("Authorization", ""); + const size_t auth_scheme_end = auth.find_first_of(" \t"); + String auth_scheme = auth.substr(0, auth_scheme_end); + std::transform(auth_scheme.begin(), auth_scheme.end(), auth_scheme.begin(), ::tolower); + if (auth_scheme == "basic") + { + try + { + if (auth_scheme_end != 5 || auth.size() <= 6 || auth[5] != ' ') + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Basic Authorization header"); + const String encoded = auth.substr(6); + const size_t padding_pos = encoded.find('='); + const size_t data_end = padding_pos == String::npos ? encoded.size() : padding_pos; + if (encoded.size() % 4 != 0 || encoded.size() - data_end > 2) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Basic Authorization header"); + for (size_t i = 0; i < encoded.size(); ++i) + { + const unsigned char c = encoded[i]; + const bool is_base64_character + = (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '+' || c == '/'; + if (i < data_end ? !is_base64_character : c != '=') + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Basic Authorization header"); + } + const String decoded = base64Decode(encoded); + const size_t colon = decoded.find(':'); + if (colon == String::npos || colon == 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Basic Authorization header"); + backend.user = decoded.substr(0, colon); + backend.password = decoded.substr(colon + 1); + return; + } + catch (...) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Basic Authorization header"); + } + } + if (!auth.empty()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unsupported Authorization scheme"); + + const String header_user = request.get("X-ClickHouse-User", ""); + if (!header_user.empty()) + { + backend.user = header_user; + backend.password = request.get("X-ClickHouse-Key", ""); + return; + } + + const String param_user = queryParam(uri, "user", ""); + if (!param_user.empty()) + { + backend.user = param_user; + backend.password = queryParam(uri, "password", ""); + } +} + +} + +WsProxyHandler::WsProxyHandler(ContextPtr context_, BackendParams backend_) + : context(std::move(context_)), backend(std::move(backend_)) +{ +} + +void WsProxyHandler::serveInfo(HTTPServerRequest & request, HTTPServerResponse & response) +{ + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_OK); + response.setContentType("text/plain; charset=UTF-8"); + *response.send() + << "clickhouse-wsproxy\n" + << "Requested: " << request.getURI() << "\n" + << "Open a WebSocket connection to run queries. Send a query as a text\n" + << "frame; results stream back as binary frames in the output format\n" + << "(set via ?format=..., default JSONEachRow), followed by a JSON\n" + << "control frame ({\"event\":\"end\"} / \"error\" / \"cancelled\").\n"; +} + +void WsProxyHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResponse & response) +{ + LoggerPtr log = getLogger("WsProxyHandler"); + + if (request.getMethod() != HTTPRequest::HTTP_GET) + { + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_METHOD_NOT_ALLOWED); + *response.send() << "WebSocket upgrade requires GET method.\n"; + return; + } + + String ws_key = request.get("Sec-WebSocket-Key", ""); + if (!isValidWebSocketKey(ws_key)) + { + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_BAD_REQUEST); + *response.send() << "Invalid or missing Sec-WebSocket-Key.\n"; + return; + } + + /// RFC 6455 fixed the protocol at version 13; earlier drafts are obsolete. + if (request.get("Sec-WebSocket-Version", "") != "13") + { + response.set("Sec-WebSocket-Version", "13"); + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_BAD_REQUEST); + *response.send() << "Unsupported WebSocket version.\n"; + return; + } + + const char * allowed_origins_value = std::getenv("WSPROXY_ALLOWED_ORIGINS"); + const String allowed_origins = allowed_origins_value ? allowed_origins_value : ""; + if (!originAllowed(request, allowed_origins, log)) + { + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_FORBIDDEN); + *response.send() << "Origin not allowed.\n"; + return; + } + + const String & uri = request.getURI(); + String out_format; + String logs_level; + std::optional flow_credit_param; + String parse_param; + String parallel_param; + BackendParams session_backend = backend; + Int64 send_timeout_sec = 0; + try + { + out_format = queryParam(uri, "format", "JSONEachRow"); + logs_level = queryParam(uri, "logs", ""); + flow_credit_param = positiveInt64QueryParam(uri, "flow"); + parse_param = queryParam(uri, "parse", ""); + parallel_param = queryParam(uri, "parallel", ""); + resolveCredentials(request, uri, session_backend); + send_timeout_sec = clientSendTimeoutSeconds(); + } + catch (...) + { + LOG_DEBUG(log, "Invalid WebSocket request: {}", getCurrentExceptionMessage(false)); + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_BAD_REQUEST); + *response.send() << getCurrentExceptionMessage(false) << "\n"; + return; + } + + const bool flow_enabled = flow_credit_param.has_value(); + const Int64 flow_credit = flow_credit_param.value_or(0); + const bool parse_enabled = parse_param == "1" || parse_param == "true"; + const bool parallel_enabled = parallel_param == "1" || parallel_param == "true"; + + /// Complete the handshake by writing 101 directly to the socket; from here + /// on the stream is in WebSocket framing mode and we own the socket. + Poco::Net::StreamSocket & socket = response.getSocket(); + const String handshake + = "HTTP/1.1 101 Switching Protocols\r\n" + "Upgrade: websocket\r\n" + "Connection: Upgrade\r\n" + "Sec-WebSocket-Accept: " + + computeWebSocketAccept(ws_key) + "\r\n\r\n"; + + size_t sent_total = 0; + while (sent_total < handshake.size()) + { + int sent = socket.sendBytes(handshake.data() + sent_total, static_cast(handshake.size() - sent_total)); + if (sent <= 0) + { + LOG_DEBUG(log, "Failed to write WebSocket handshake"); + return; + } + sent_total += static_cast(sent); + } + + SCOPE_EXIT({ + try + { + socket.shutdown(); + } + catch (...) + { + /// The client may have already closed the socket; expected, not worth a stack trace. + LOG_DEBUG(log, "WebSocket socket shutdown: {}", getCurrentExceptionMessage(false)); + } + }); + + /// Optional: `?logs=` (e.g. information, trace) makes the backend push + /// server-side log lines for the session's queries. + /// Opt-in credit/window flow control: `?flow=N` enables it with N frames of + /// initial credit; absent = push mode (unbounded). The client then grants more + /// with {"cmd":"next","n":...} and can pause/resume. + /// Opt-in SQL parsing: `?parse=1` lets the proxy parse each query to auto-route + /// streamed-data INSERTs (no {"cmd":"insert"} needed) and report the parsed + /// verb + routing decision as a {"event":"query",...} frame. Off by default — + /// the proxy does not parse SQL unless the client explicitly asks it to. + /// Opt-in parallel output formatting: `?parallel=1` formats SELECT output on a thread pool + /// for higher conversion throughput, at the cost of coarser (batched) result frames. Ignored + /// when flow control is on. Default off preserves fine-grained one-frame-per-block streaming. + /// Credential pass-through: resolve this session's backend user/password from + /// the request (never mutate the shared default `backend`). + LOG_DEBUG(log, "WebSocket session established; format {}, backend user {}", out_format, session_backend.user); + + /// Bound each blocking WebSocket read so a stalled client cannot pin the + /// handler thread indefinitely between queries. + socket.setReceiveTimeout(Poco::Timespan(300, 0)); + + /// Send timeout: drop a client that stops reading entirely (its TCP window + /// stuck at 0) so a blocking write cannot pin a handler thread forever. The + /// resulting throw routes through WriteBufferToWebSocket (broken -> sendCancel) + /// to a clean teardown. Applies per blocking write, so a slow-but-progressing + /// client is unaffected; only genuine zero-progress stalls trip it. + /// Configurable (mainly for tests); default 30s. + socket.setSendTimeout(Poco::Timespan(send_timeout_sec * 1'000'000)); /// microseconds + + /// Open one native-protocol connection per session to the remote backend. + Connection connection( + session_backend.host, + session_backend.port, + session_backend.database, + session_backend.user, + session_backend.password, + /* proto_send_chunked_ */ "chunked_optional", + /* proto_recv_chunked_ */ "chunked_optional", + /* ssh_private_key_ */ SSHKey{}, + /* jwt_ */ "", + /* quota_key_ */ "", + /* cluster_ */ "", + /* cluster_secret_ */ "", + /* client_name_ */ "clickhouse-wsproxy", + session_backend.compression_method == "none" ? Protocol::Compression::Disable : Protocol::Compression::Enable, + session_backend.secure ? Protocol::Secure::Enable : Protocol::Secure::Disable, + /* tls_sni_override_ */ "", + /* bind_host_ */ ""); + + /// Establish the backend connection up front so authentication (during the + /// native handshake) is reported immediately rather than on the first query. + /// On failure, tell the client and close. + try + { + connection.forceConnected(ConnectionTimeouts::getTCPTimeoutsWithoutFailover(context->getSettingsRef())); + } + catch (...) + { + const String message = getCurrentExceptionMessage(false); + LOG_DEBUG(log, "Backend connect/auth failed: {}", message); + try + { + sendWebSocketText(socket, R"({"event":"error","message":")" + jsonEscape(message) + "\"}"); + sendWebSocketClose(socket, /* 1008 policy violation */ 1008, "Authentication failed"); + } + catch (...) + { + LOG_DEBUG(log, "Failed to deliver auth error to client (already gone)"); + } + return; + } + + WebSocketSession session( + socket, context, out_format, logs_level, flow_enabled, flow_credit, parse_enabled, + parallel_enabled, session_backend.compression_method); + session.run(connection); + + LOG_DEBUG(log, "WebSocket session closed"); +} + +void WsProxyHandler::handleRequest(HTTPServerRequest & request, HTTPServerResponse & response, const ProfileEvents::Event &) +{ + try + { + String upgrade = request.get("Upgrade", ""); + std::transform(upgrade.begin(), upgrade.end(), upgrade.begin(), ::tolower); + + if (upgrade == "websocket" && hasUpgradeToken(request.get("Connection", ""))) + handleWebSocket(request, response); + else + serveInfo(request, response); + } + catch (...) + { + tryLogCurrentException("WsProxyHandler"); + } +} + +} diff --git a/programs/wsproxy/WsProxyHandler.h b/programs/wsproxy/WsProxyHandler.h new file mode 100644 index 000000000000..bdc149219413 --- /dev/null +++ b/programs/wsproxy/WsProxyHandler.h @@ -0,0 +1,48 @@ +#pragma once + +#include +#include +#include + +namespace DB +{ + +/// Where the edge proxy forwards native-protocol queries (a remote backend). +struct BackendParams +{ + String host = "localhost"; + UInt16 port = 9000; + String user = "default"; + String password; + String database; + bool secure = false; /// Connect to the backend over TLS (native secure protocol). + /// Native-protocol compression codec for backend->proxy result blocks: "lz4" (fast, default), + /// "zstd" (higher ratio — fewer bytes on a bandwidth-limited WAN, at more backend CPU), or + /// "none". The proxy decompresses whatever the server sends regardless (codec is self-describing). + String compression_method = "lz4"; +}; + +/// HTTP entry point for the proxy. +/// +/// A plain HTTP request serves a short info page. A WebSocket upgrade completes +/// the RFC 6455 handshake after validating any browser `Origin`, then hands the +/// socket to a `WebSocketSession`, which bridges the WebSocket to a +/// native-protocol `Connection` against the backend. +/// The desired output format is taken from the `format` query parameter of the +/// WebSocket URL (default `JSONEachRow`). +class WsProxyHandler : public HTTPRequestHandler +{ +public: + WsProxyHandler(ContextPtr context_, BackendParams backend_); + + void handleRequest(HTTPServerRequest & request, HTTPServerResponse & response, const ProfileEvents::Event & write_event) override; + +private: + void handleWebSocket(HTTPServerRequest & request, HTTPServerResponse & response); + void serveInfo(HTTPServerRequest & request, HTTPServerResponse & response); + + ContextPtr context; + BackendParams backend; +}; + +} diff --git a/programs/wsproxy/bench/README.md b/programs/wsproxy/bench/README.md new file mode 100644 index 000000000000..31a0f61bf198 --- /dev/null +++ b/programs/wsproxy/bench/README.md @@ -0,0 +1,76 @@ +# clickhouse-wsproxy benchmarks + +Throughput benchmarks for fetching a query result as `JSONCompactEachRow`, comparing the +proxy against fetching the same bytes directly (native `clickhouse-client` and the HTTP +interface). Used to measure the cost of edge format conversion and to validate changes such +as parallel output formatting. + +The reference query is +`SELECT number AS n, number*2 AS d, toString(number) AS s FROM numbers(N)` — deterministic, +server-generated, and forces real formatting of two `UInt64` columns and one `String`. + +## Scripts + +- `bench.mjs` — drive the proxy over WebSocket. Opens a fresh session per iteration (each is a + new backend connection, so the cloud TLS handshake is *not* amortized — matching what + `clickhouse-client` pays per run), sends the query, counts the streamed bytes until the + terminal `end`, and reports min / median wall time plus throughput. One warm-up iteration is + discarded. + + ```bash + # against a proxy on ws://127.0.0.1:9010 (override with WSPROXY_URL) + node bench.mjs + node bench.mjs 3000000 JSONCompactEachRow 5 + ``` + +- `baselines.sh` — direct baselines for the same result, with byte-identical output (forces + `output_format_json_quote_64bit_integers=0` so the proxy's unquoted `UInt64` matches + `clickhouse-client`/HTTP, which quote 64-bit ints by default). Reports three paths: native + transport + client-side format, HTTP server-side format (plain), and HTTP + gzip. Reads the + cloud endpoint from the environment. + + ```bash + CLICKHOUSE_CLOUD_HOST=... CLICKHOUSE_CLOUD_PASSWORD=... bash baselines.sh + ``` + +- `tcp_ceiling.mjs` — diagnostic: measures Node's raw TCP loopback receive ceiling, to confirm + the JS client is not the bottleneck when attributing proxy overhead. + +## Running against a backend + +Start a proxy pointed at the backend, then run `bench.mjs`. + +```bash +# Cloud backend (native secure), credentials from the environment: +WSPROXY_BACKEND_HOST="$CLICKHOUSE_CLOUD_HOST" WSPROXY_BACKEND_PORT=9440 \ +WSPROXY_BACKEND_USER=default WSPROXY_BACKEND_PASSWORD="$CLICKHOUSE_CLOUD_PASSWORD" \ +WSPROXY_BACKEND_SECURE=1 WSPROXY_PORT=9010 \ + ../../../build/programs/wsproxy/clickhouse-wsproxy & + +node bench.mjs 3000000 JSONCompactEachRow 5 +CLICKHOUSE_CLOUD_HOST="$CLICKHOUSE_CLOUD_HOST" CLICKHOUSE_CLOUD_PASSWORD="$CLICKHOUSE_CLOUD_PASSWORD" \ + bash baselines.sh 3000000 3 +``` + +For an intrinsic (CPU-bound) measurement without the WAN, point the proxy at a local backend +(`WSPROXY_BACKEND_HOST=127.0.0.1 WSPROXY_BACKEND_PORT= WSPROXY_BACKEND_SECURE=0`) and run +the same commands against `127.0.0.1`. + +## Interpreting results + +- **Over a WAN** the result is network-bound, so the codec choice dominates. The proxy pulls + compressed native blocks and converts at the edge. With the default **lz4** it can lose to gzipped + HTTP on *highly compressible* data (lz4 trades ratio for speed: e.g. 36.5 MB on the wire vs gzip's + 21.3 MB for the sequential-integer query). Set `WSPROXY_BACKEND_COMPRESSION=zstd` and columnar + native drops to ~7-10 MB — 2-3× smaller than gzip-on-JSON — so the proxy wins outright (3M-row + cloud fetch ~2.5s vs gzip-HTTP ~3.5s). On realistic/high-entropy data columnar native beats row + JSON even with lz4. Compare with `baselines.sh` and re-run the proxy with different + `WSPROXY_BACKEND_COMPRESSION` values. + +The edge-conversion + compressed-wire model these numbers support is what a Cloudflare deployment +would exploit — see [`../CLOUDFLARE.md`](../CLOUDFLARE.md). +- **On loopback** the result is CPU-bound and exposes the proxy's conversion throughput. The + proxy formats output on a thread pool when `output_format_parallel_formatting` is enabled and + the format supports it; compare against `baselines.sh` (which formats in parallel by default) + and against a direct client run with `SETTINGS output_format_parallel_formatting=0` to see the + single-threaded floor. diff --git a/programs/wsproxy/bench/baselines.sh b/programs/wsproxy/bench/baselines.sh new file mode 100755 index 000000000000..5074e7723c86 --- /dev/null +++ b/programs/wsproxy/bench/baselines.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# Baseline fetch timings for a JSONCompactEachRow result, matched to the proxy's +# output (unquoted 64-bit ints) so byte counts are identical across paths. +set -uo pipefail +ROWS="${1:-1000000}" +ITERS="${2:-5}" +CH_HOST="$CLICKHOUSE_CLOUD_HOST"; CH_PW="$CLICKHOUSE_CLOUD_PASSWORD" +Q="SELECT number AS n, number*2 AS d, toString(number) AS s FROM numbers(${ROWS})" +S="SETTINGS output_format_json_quote_64bit_integers=0" +FQ="$Q $S FORMAT JSONCompactEachRow" + +median() { printf '%s\n' "$@" | sort -n | awk '{a[NR]=$1} END{print a[int((NR+1)/2)]}'; } + +# Run "$@" with up to 3 retries on transient failure; echoes stdout byte count. +retry_bytes() { local n=0; while :; do b=$("$@" | wc -c) && [ "$b" -gt 100 ] && { echo "$b"; return 0; }; n=$((n+1)); [ "$n" -ge 3 ] && { echo 0; return 1; }; sleep 2; done; } +nat() { clickhouse-client --host "$CH_HOST" --secure --user default --password "$CH_PW" --query "$FQ"; } +httpp() { curl -sS "https://${CH_HOST}:8443/" -H "X-ClickHouse-User: default" -H "X-ClickHouse-Key: ${CH_PW}" --data-binary "$FQ"; } +httpz() { curl -sS --compressed "https://${CH_HOST}:8443/?enable_http_compression=1" -H "X-ClickHouse-User: default" -H "X-ClickHouse-Key: ${CH_PW}" --data-binary "$FQ"; } + +echo "### rows=$ROWS iters=$ITERS" + +clickhouse-client --host "$CH_HOST" --secure --user default --password "$CH_PW" --query "SELECT 1" >/dev/null 2>&1 || true + +run_path() { # $1=label $2=fn + local label="$1" fn="$2" bytes=0 t=() s e b + for _ in $(seq "$ITERS"); do + s=$(date +%s.%N); b=$(retry_bytes "$fn"); e=$(date +%s.%N) + t+=("$(echo "$e - $s" | bc)"); bytes=$b + done + printf '%-40s median=%ss bytes=%s\n' "$label" "$(median "${t[@]}")" "$bytes" +} + +run_path "NATIVE (native lz4 -> client format)" nat +run_path "HTTP (server format, plain JSON)" httpp +run_path "HTTP-GZ (server format, gzip WAN)" httpz diff --git a/programs/wsproxy/bench/bench.mjs b/programs/wsproxy/bench/bench.mjs new file mode 100644 index 000000000000..9f3c5bacada4 --- /dev/null +++ b/programs/wsproxy/bench/bench.mjs @@ -0,0 +1,55 @@ +// Benchmark: fetch a query result through the wsproxy and measure wall time + +// bytes received. Usage: node bench.mjs +const ROWS = Number(process.argv[2] ?? 1_000_000); +const FORMAT = process.argv[3] ?? "JSONCompactEachRow"; +const ITERS = Number(process.argv[4] ?? 5); +const URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; +const PARALLEL = process.env.PARALLEL === "1"; // opt into ?parallel=1 (parallel output formatting) + +const QUERY = `SELECT number AS n, number*2 AS d, toString(number) AS s FROM numbers(${ROWS})`; + +function once() { + return new Promise((resolve, reject) => { + const ws = new WebSocket(`${URL}/?format=${FORMAT}${PARALLEL ? "¶llel=1" : ""}`); + ws.binaryType = "arraybuffer"; + let bytes = 0; + let t0 = 0; + ws.addEventListener("open", () => { + t0 = performance.now(); + ws.send(QUERY); + }); + ws.addEventListener("message", (ev) => { + if (typeof ev.data === "string") { + const msg = JSON.parse(ev.data); + if (msg.event === "end" || msg.event === "error" || msg.event === "cancelled") { + const ms = performance.now() - t0; + ws.close(); + if (msg.event === "end") resolve({ ms, bytes }); + else reject(new Error(`${msg.event}: ${msg.message ?? ""}`)); + } + } else { + bytes += ev.data.byteLength; + } + }); + ws.addEventListener("error", (e) => reject(new Error(`ws error: ${e?.message ?? e}`))); + }); +} + +const results = []; +// One warm-up iteration (not counted). +await once(); +for (let i = 0; i < ITERS; i++) { + const r = await once(); + results.push(r); + console.log(` iter ${i + 1}: ${r.ms.toFixed(0)} ms, ${(r.bytes / 1e6).toFixed(1)} MB`); +} +const times = results.map((r) => r.ms).sort((a, b) => a - b); +const median = times[Math.floor(times.length / 2)]; +const min = times[0]; +const bytes = results[0].bytes; +const mbps = bytes / 1e6 / (min / 1000); // best-case throughput +const rowsPerSec = ROWS / (min / 1000); +console.log( + `PROXY ${FORMAT} rows=${ROWS}: min=${min.toFixed(0)}ms median=${median.toFixed(0)}ms ` + + `bytes=${(bytes / 1e6).toFixed(1)}MB throughput=${mbps.toFixed(0)}MB/s (${(rowsPerSec / 1e6).toFixed(1)}M rows/s)`, +); diff --git a/programs/wsproxy/bench/tcp_ceiling.mjs b/programs/wsproxy/bench/tcp_ceiling.mjs new file mode 100644 index 000000000000..600ee3ef742f --- /dev/null +++ b/programs/wsproxy/bench/tcp_ceiling.mjs @@ -0,0 +1,25 @@ +import net from "node:net"; +const BYTES = 962_000_000; +const buf = Buffer.allocUnsafe(1 << 20); // 1MB chunks +const server = net.createServer((sock) => { + let sent = 0; + const pump = () => { + while (sent < BYTES) { + sent += buf.length; + if (!sock.write(buf)) { sock.once("drain", pump); return; } + } + sock.end(); + }; + pump(); +}); +server.listen(0, "127.0.0.1", () => { + const port = server.address().port; + let got = 0, t0 = performance.now(); + const c = net.connect(port, "127.0.0.1"); + c.on("data", (d) => { got += d.length; }); + c.on("end", () => { + const ms = performance.now() - t0; + console.log(`raw TCP recv: ${(got/1e6).toFixed(0)}MB in ${ms.toFixed(0)}ms = ${(got/1e6/(ms/1000)).toFixed(0)} MB/s`); + server.close(); + }); +}); diff --git a/programs/wsproxy/tests/.gitignore b/programs/wsproxy/tests/.gitignore new file mode 100644 index 000000000000..c2658d7d1b31 --- /dev/null +++ b/programs/wsproxy/tests/.gitignore @@ -0,0 +1 @@ +node_modules/ diff --git a/programs/wsproxy/tests/README.md b/programs/wsproxy/tests/README.md new file mode 100644 index 000000000000..8227a6464a74 --- /dev/null +++ b/programs/wsproxy/tests/README.md @@ -0,0 +1,110 @@ +# WebSocket integration tests + +This directory contains Node.js/Vitest integration tests for the shared WebSocket +query protocol. The primary target is the in-server `ws_port`; the same common +tests also validate the compatible `clickhouse-wsproxy` prototype. + +The suites cover query and format handling, streamed inserts, progress, logs, +profile events, cancellation, authentication, flow control, malformed and partial +frames, fragmented-message limits, interleaved control frames, origin policy, +concurrency, and session reuse. Tests that require a remote backend are proxy-only. + +## Requirements + +- Node.js 22 or newer. +- Dependencies installed with `npm install` in this directory. +- For `ws_port`, a built server at `build/programs/clickhouse` or a path supplied + through `WS_SERVER_BIN`. +- For the standalone suite, a built proxy at + `build/programs/wsproxy/clickhouse-wsproxy` and a `clickhouse-server` at + `/usr/local/bin/clickhouse-server`, or paths supplied through `WSPROXY_BIN` and + `CLICKHOUSE_SERVER`. + +The harness generates temporary ClickHouse configuration under its own temporary +directory, starts the required processes, and stops the processes it started. No +pre-existing ClickHouse configuration is required. + +## Run the primary in-server suite + +```bash +cd programs/wsproxy/tests +npm install +npm run test:server +``` + +`vitest.server.config.mjs` starts `clickhouse-server` with both `ws_port` and +`tcp_port`. It excludes cases that require the standalone proxy boundary, including +proxy-to-backend TLS, remote-backend failure, backend compression selection, and +proxy-specific send-timeout configuration. + +Origin-policy tests configure an allowlist and verify accepted and rejected browser +origins, plus clients without an `Origin` header. The implementation's empty-list +behavior is same-origin only. + +To select another server binary: + +```bash +WS_SERVER_BIN=/path/to/clickhouse npm run test:server +``` + +## Run the standalone compatibility suite + +```bash +cd programs/wsproxy/tests +npm install +npm test +``` + +The default configuration starts a ClickHouse backend and +`clickhouse-wsproxy`. Override binary and endpoint selection with +`CLICKHOUSE_SERVER`, `WSPROXY_BIN`, and `WSPROXY_URL` when needed. + +The proxy applies the same browser policy through `WSPROXY_ALLOWED_ORIGINS`: an +`Origin` header must be same-origin unless it appears in that comma-separated +allowlist, while requests without `Origin` are accepted. + +## Protocol exercised by the tests + +A client sends SQL in a text frame. Query data is returned in binary frames using +the output format selected by `?format=`. Progress, logs, profile events, query +classification, and terminal outcomes are JSON text frames. + +A streamed insert begins with a control message: + +```json +{"cmd":"insert","query":"INSERT INTO t FORMAT JSONEachRow","format":"JSONEachRow"} +``` + +The client then sends input data in binary frames and finishes with an empty binary +frame. Closing the WebSocket while a query or insert is active requests +cancellation. + +Optional protocol modes include: + +- `?flow=N` starts with `N` frames of output credit. A client grants more credit as + it consumes frames. A client at zero credit also delays ordered control events, + so applications should grant small increments instead of pausing indefinitely. +- `?parallel=1` requests parallel output formatting. It is disabled when flow + control is active. +- `?parse=1` asks the server to classify SQL and report its leading verb and route. + Classification failures return an error instead of silently selecting another + route. Explicit insert control messages remain the unambiguous streamed-insert + path. + +`test/helpers.mjs` contains the `Session`, `runQuery`, and `backendScalar` test +helpers. `test/raw.mjs` provides raw frame construction for adversarial cases. + +## Backpressure + +Browser-style event-based `WebSocket` APIs eagerly receive messages and do not +provide receive backpressure. Tests and applications that process large results +can use the `?flow=N` extension, a `WebSocketStream` reader where available, or a +transport that can pause socket reads. Plain browser clients should keep results +bounded when they cannot consume them promptly. + +The in-server `ws_port` currently uses plaintext `ws://`. TLS termination is an +external deployment responsibility; no `ws_port_secure` listener is built in this +branch. + +Cloudflare deployment described in `../CLOUDFLARE.md` is unbuilt analysis for the +standalone proxy and is not a test target. diff --git a/programs/wsproxy/tests/package-lock.json b/programs/wsproxy/tests/package-lock.json new file mode 100644 index 000000000000..7fdfcf87f116 --- /dev/null +++ b/programs/wsproxy/tests/package-lock.json @@ -0,0 +1,1460 @@ +{ + "name": "wsproxy-tests", + "version": "0.1.0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "wsproxy-tests", + "version": "0.1.0", + "devDependencies": { + "vitest": "^2.1.9" + } + }, + "node_modules/@esbuild/aix-ppc64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.21.5.tgz", + "integrity": "sha512-1SDgH6ZSPTlggy1yI6+Dbkiz8xzpHJEVAlF/AM1tHPLsf5STom9rwtjE4hKAF20FfXXNTFqEYXyJNWh1GiZedQ==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "aix" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-arm": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.21.5.tgz", + "integrity": "sha512-vCPvzSjpPHEi1siZdlvAlsPxXl7WbOVUBBAowWug4rJHb68Ox8KualB+1ocNvT5fjv6wpkX6o/iEpbDrf68zcg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.21.5.tgz", + "integrity": "sha512-c0uX9VAUBQ7dTDCjq+wdyGLowMdtR/GoC2U5IYk/7D1H1JYC0qseD7+11iMP2mRLN9RcCMRcjC4YMclCzGwS/A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/android-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.21.5.tgz", + "integrity": "sha512-D7aPRUUNHRBwHxzxRvp856rjUHRFW1SdQATKXH2hqA0kAZb1hKmi02OpYRacl0TxIGz/ZmXWlbZgjwWYaCakTA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/darwin-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.21.5.tgz", + "integrity": "sha512-DwqXqZyuk5AiWWf3UfLiRDJ5EDd49zg6O9wclZ7kUMv2WRFr4HKjXp/5t8JZ11QbQfUS6/cRCKGwYhtNAY88kQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/darwin-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.21.5.tgz", + "integrity": "sha512-se/JjF8NlmKVG4kNIuyWMV/22ZaerB+qaSi5MdrXtd6R08kvs2qCN4C09miupktDitvh8jRFflwGFBQcxZRjbw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/freebsd-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.21.5.tgz", + "integrity": "sha512-5JcRxxRDUJLX8JXp/wcBCy3pENnCgBR9bN6JsY4OmhfUtIHe3ZW0mawA7+RDAcMLrMIZaf03NlQiX9DGyB8h4g==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/freebsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.21.5.tgz", + "integrity": "sha512-J95kNBj1zkbMXtHVH29bBriQygMXqoVQOQYA+ISs0/2l3T9/kj42ow2mpqerRBxDJnmkUDCaQT/dfNXWX/ZZCQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-arm": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.21.5.tgz", + "integrity": "sha512-bPb5AHZtbeNGjCKVZ9UGqGwo8EUu4cLq68E95A53KlxAPRmUyYv2D6F0uUI65XisGOL1hBP5mTronbgo+0bFcA==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.21.5.tgz", + "integrity": "sha512-ibKvmyYzKsBeX8d8I7MH/TMfWDXBF3db4qM6sy+7re0YXya+K1cem3on9XgdT2EQGMu4hQyZhan7TeQ8XkGp4Q==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-ia32": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.21.5.tgz", + "integrity": "sha512-YvjXDqLRqPDl2dvRODYmmhz4rPeVKYvppfGYKSNGdyZkA01046pLWyRKKI3ax8fbJoK5QbxblURkwK/MWY18Tg==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-loong64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.21.5.tgz", + "integrity": "sha512-uHf1BmMG8qEvzdrzAqg2SIG/02+4/DHB6a9Kbya0XDvwDEKCoC8ZRWI5JJvNdUjtciBGFQ5PuBlpEOXQj+JQSg==", + "cpu": [ + "loong64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-mips64el": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.21.5.tgz", + "integrity": "sha512-IajOmO+KJK23bj52dFSNCMsz1QP1DqM6cwLUv3W1QwyxkyIWecfafnI555fvSGqEKwjMXVLokcV5ygHW5b3Jbg==", + "cpu": [ + "mips64el" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-ppc64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.21.5.tgz", + "integrity": "sha512-1hHV/Z4OEfMwpLO8rp7CvlhBDnjsC3CttJXIhBi+5Aj5r+MBvy4egg7wCbe//hSsT+RvDAG7s81tAvpL2XAE4w==", + "cpu": [ + "ppc64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-riscv64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.21.5.tgz", + "integrity": "sha512-2HdXDMd9GMgTGrPWnJzP2ALSokE/0O5HhTUvWIbD3YdjME8JwvSCnNGBnTThKGEB91OZhzrJ4qIIxk/SBmyDDA==", + "cpu": [ + "riscv64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-s390x": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.21.5.tgz", + "integrity": "sha512-zus5sxzqBJD3eXxwvjN1yQkRepANgxE9lgOW2qLnmr8ikMTphkjgXu1HR01K4FJg8h1kEEDAqDcZQtbrRnB41A==", + "cpu": [ + "s390x" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/linux-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.21.5.tgz", + "integrity": "sha512-1rYdTpyv03iycF1+BhzrzQJCdOuAOtaqHTWJZCWvijKD2N5Xu0TtVC8/+1faWqcP9iBCWOmjmhoH94dH82BxPQ==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/netbsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.21.5.tgz", + "integrity": "sha512-Woi2MXzXjMULccIwMnLciyZH4nCIMpWQAs049KEeMvOcNADVxo0UBIQPfSmxB3CWKedngg7sWZdLvLczpe0tLg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "netbsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/openbsd-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.21.5.tgz", + "integrity": "sha512-HLNNw99xsvx12lFBUwoT8EVCsSvRNDVxNpjZ7bPn947b8gJPzeHWyNVhFsaerc0n3TsbOINvRP2byTZ5LKezow==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/sunos-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.21.5.tgz", + "integrity": "sha512-6+gjmFpfy0BHU5Tpptkuh8+uw3mnrvgs+dSPQXQOv3ekbordwnzTVEb4qnIvQcYXq6gzkyTnoZ9dZG+D4garKg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "sunos" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-arm64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.21.5.tgz", + "integrity": "sha512-Z0gOTd75VvXqyq7nsl93zwahcTROgqvuAcYDUr+vOv8uHhNSKROyU961kgtCD1e95IqPKSQKH7tBTslnS3tA8A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-ia32": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.21.5.tgz", + "integrity": "sha512-SWXFF1CL2RVNMaVs+BBClwtfZSvDgtL//G/smwAc5oVK/UPu2Gu9tIaRgFmYFFKrmg3SyAjSrElf0TiJ1v8fYA==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@esbuild/win32-x64": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.21.5.tgz", + "integrity": "sha512-tQd/1efJuzPC6rCFwEvLtci/xNFcTZknmXs98FYDfGE4wP9ClFV98nyKrzJKVPMhdDnjzLhdUyMX4PsQAPjwIw==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">=12" + } + }, + "node_modules/@jridgewell/sourcemap-codec": { + "version": "1.5.5", + "resolved": "https://registry.npmjs.org/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz", + "integrity": "sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==", + "dev": true, + "license": "MIT" + }, + "node_modules/@rollup/rollup-android-arm-eabi": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm-eabi/-/rollup-android-arm-eabi-4.62.2.tgz", + "integrity": "sha512-6o7ZLZK+BeenkZCFNDXqpbjw9bD6nuWonvS/lwQJp7NoVVxm6p3qE7qQ5jGuBjiFsgvqjD8mZAU5oWxTmbOeOg==", + "cpu": [ + "arm" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-android-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-android-arm64/-/rollup-android-arm64-4.62.2.tgz", + "integrity": "sha512-BaH7BllCACHoH1LguOU56UItGfUWjujlO65kS9LAodViaN4bwIKd7oeW/ZHJ/4ljr/7MIiENnNy3HJ0zXv8Zkw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "android" + ] + }, + "node_modules/@rollup/rollup-darwin-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-arm64/-/rollup-darwin-arm64-4.62.2.tgz", + "integrity": "sha512-v39RCCvj4He82I9sFmk+M1VZ0PLM9sfsLVikjfx2hYBNALhrrOR2D3JjQA6AhlaSOgcR+RzrKY7e1+bT6SUO/A==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-darwin-x64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-darwin-x64/-/rollup-darwin-x64-4.62.2.tgz", + "integrity": "sha512-yl0y2vq3S3lHeuXhEdss6TWfKW8vkujImO12tn4ZkG/4oghr09LvdYm2RElVjokTQiUvDUGXLGsYeLqUMCKpGA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ] + }, + "node_modules/@rollup/rollup-freebsd-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-arm64/-/rollup-freebsd-arm64-4.62.2.tgz", + "integrity": "sha512-tT4pvt4qXD+vEoezupCWi+a1F0vvDiksiHc+PxRlYTOH1I6/X4id9jPxTP+Fg+545euaFT1jJVs4CEdHZAU1vw==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-freebsd-x64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-freebsd-x64/-/rollup-freebsd-x64-4.62.2.tgz", + "integrity": "sha512-6nU5F2wCW+qvCBhTn1pdIU3bzsIoF7EUwsCDRxilWGprQR6yd508YnH9+OKFCwpfS8pjZqDUmnCAr7exax0XCg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "freebsd" + ] + }, + "node_modules/@rollup/rollup-linux-arm-gnueabihf": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-gnueabihf/-/rollup-linux-arm-gnueabihf-4.62.2.tgz", + "integrity": "sha512-n1GJHPOvpIfhi3TmrCeh6S6URt9BFCt0KQE3qvexyGCTAKpR4Lg+eWvNZEqu7epxwus/8ElT3hacYEucm49SZg==", + "cpu": [ + "arm" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm-musleabihf": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm-musleabihf/-/rollup-linux-arm-musleabihf-4.62.2.tgz", + "integrity": "sha512-JqgflS8wEB+UXV/vS1RpRbifGBeN4D5lz8D8oOFbFZw4vedvdOgCFAjfBmIMdW3yL10XpQQ0Ambepw6MXrhOnA==", + "cpu": [ + "arm" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-gnu/-/rollup-linux-arm64-gnu-4.62.2.tgz", + "integrity": "sha512-wnFJkogWvN4jm/hQRF2UBaeUmk20j5+DmHvoyWii2b8HJDyvz1MF2OU/6ynXt2KR63rbZLWkFpoytpdc/yBuSA==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-arm64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-arm64-musl/-/rollup-linux-arm64-musl-4.62.2.tgz", + "integrity": "sha512-HVu2bp0zhvJ8xHEV9+UUs7S90VadmBSY3LcIMvozbPo4AuMGDWlz3ymHLHZPX4hR67TKTt8Qp5PJ5RBg/i+RMQ==", + "cpu": [ + "arm64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-gnu/-/rollup-linux-loong64-gnu-4.62.2.tgz", + "integrity": "sha512-mQqqAV8QaoSgr9I2fKDLY2BAVvmKjWoGiu/cSYQonsLvtqwEn1E4QYfnCOcp5zoEqNhsDYin1s6jx/VJmrxlZg==", + "cpu": [ + "loong64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-loong64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-loong64-musl/-/rollup-linux-loong64-musl-4.62.2.tgz", + "integrity": "sha512-IxKLoxCQ2IWi6bT2akyDUBGsOImDKB+sPp4EsTmwFQ/fMwpCKm8uLSSgP/Kx/QYUgKis6SEZ5/Nlhup0DIA0PQ==", + "cpu": [ + "loong64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-gnu/-/rollup-linux-ppc64-gnu-4.62.2.tgz", + "integrity": "sha512-Mk5ha2RQSgyFfmYYLkBpPnUk8D8FriBxesO1u9O75X0mHgXL1UQcH5Itl2lurWL2tj0RxV9b9tJgipac0hRY9A==", + "cpu": [ + "ppc64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-ppc64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-ppc64-musl/-/rollup-linux-ppc64-musl-4.62.2.tgz", + "integrity": "sha512-CjvEnqJL/0/TQ3TXX3OPIJ/kmBellrWd4heXUmHeJlTnmwjKpSJzoehLaL6Xk0ZnMHBu9dZuFADNOrtjF4v+2w==", + "cpu": [ + "ppc64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-gnu/-/rollup-linux-riscv64-gnu-4.62.2.tgz", + "integrity": "sha512-1SiZbzwdkaDURsew/tSOrooKiYy7EQGT6m8ufavAi9NEyQb/6VuIxFXAL1fqa4iZe3g4NbNk4P7J32z2tw5Mgg==", + "cpu": [ + "riscv64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-riscv64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-riscv64-musl/-/rollup-linux-riscv64-musl-4.62.2.tgz", + "integrity": "sha512-nQts12zJ3NQRoE6uYljOH89v7szzLDvG2JD/vsX+vGXU8w/At1GowTZ5/7qeFQ8m7L55rpR8Okugnuo5bgjy2Q==", + "cpu": [ + "riscv64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-s390x-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-s390x-gnu/-/rollup-linux-s390x-gnu-4.62.2.tgz", + "integrity": "sha512-E9/ll019jhPIJgpzfZoIkBGhcz+kKNgVWYRY0zr9srBdPPFVpvOKW8VaJKUbeK+eZXyQF9ltME+Kk6affeaPgg==", + "cpu": [ + "s390x" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-gnu/-/rollup-linux-x64-gnu-4.62.2.tgz", + "integrity": "sha512-5BqxR/pshjey51iliyzTD5Xi3EN0aLmQ2lZ3lvefVV9c82BvrLo2/6OT55iifpWBufs6kdwWbuOKS841DrmK9A==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "glibc" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-linux-x64-musl": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-linux-x64-musl/-/rollup-linux-x64-musl-4.62.2.tgz", + "integrity": "sha512-uNN83XxQrRAh/w0/pmAfibcwyb6YWt4gP+dpnQKPVJshAloQ785ii8CT8ZCIxkGg9opVsvAlGhFitSm6D1Jjpg==", + "cpu": [ + "x64" + ], + "dev": true, + "libc": [ + "musl" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ] + }, + "node_modules/@rollup/rollup-openbsd-x64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openbsd-x64/-/rollup-openbsd-x64-4.62.2.tgz", + "integrity": "sha512-srjEIxSH3LRnJN6THczDHWQplqEMFiAJrTab0msUryh9kwNpkICf3Ea6q6MN/2cZwRFUNx5w+h6Hpi4QuHS6Zg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openbsd" + ] + }, + "node_modules/@rollup/rollup-openharmony-arm64": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-openharmony-arm64/-/rollup-openharmony-arm64-4.62.2.tgz", + "integrity": "sha512-8hOJnxgbyObnCm5AlRA3A931xX19xq80RjVTKgJOvEKWqJruP/Uf12IbAOaDjjEXYRewwHLfmF0YRIdK3OwKWA==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "openharmony" + ] + }, + "node_modules/@rollup/rollup-win32-arm64-msvc": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-arm64-msvc/-/rollup-win32-arm64-msvc-4.62.2.tgz", + "integrity": "sha512-mmF4AY1i0hG/bLWUctUq59gtmgaSIRa3cu/A3JFRp/sCNEme2bgDEiDS22P9FbnJB8NJNF4jPJiSP5RHQpUTDg==", + "cpu": [ + "arm64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-ia32-msvc": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-ia32-msvc/-/rollup-win32-ia32-msvc-4.62.2.tgz", + "integrity": "sha512-DZgkknc6jhHrk46V25vbAM0zZkyP0nSDkJB8/dRkLTxv470dOmWDqGoEJl/9A0dFfS7yE3REOwNDxpHwSLSt0Q==", + "cpu": [ + "ia32" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-gnu": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-gnu/-/rollup-win32-x64-gnu-4.62.2.tgz", + "integrity": "sha512-T6xr6ucWSFto+VGajA8YH26LdpHRuP4YLHEKAtCWvJDOlnmWcDZVCI2Jmjr+IFHDlt2zRaTAKE4tfjTaWLgJBg==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@rollup/rollup-win32-x64-msvc": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/@rollup/rollup-win32-x64-msvc/-/rollup-win32-x64-msvc-4.62.2.tgz", + "integrity": "sha512-BfzEnDJOt9T8M989/lA37EcJgat01wLRnoi5dQf3QzOH7jzpqTAzdDbVfRljVr5r+jzKqpbHeyOfAaXxAd0PAA==", + "cpu": [ + "x64" + ], + "dev": true, + "license": "MIT", + "optional": true, + "os": [ + "win32" + ] + }, + "node_modules/@types/estree": { + "version": "1.0.9", + "resolved": "https://registry.npmjs.org/@types/estree/-/estree-1.0.9.tgz", + "integrity": "sha512-GhdPgy1el4/ImP05X05Uw4cw2/M93BCUmnEvWZNStlCzEKME4Fkk+YpoA5OiHNQmoS7Cafb8Xa3Pya8m1Qrzeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/@vitest/expect": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/expect/-/expect-2.1.9.tgz", + "integrity": "sha512-UJCIkTBenHeKT1TTlKMJWy1laZewsRIzYighyYiJKZreqtdxSos/S1t+ktRMQWu2CKqaarrkeszJx1cgC5tGZw==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/spy": "2.1.9", + "@vitest/utils": "2.1.9", + "chai": "^5.1.2", + "tinyrainbow": "^1.2.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/mocker": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/mocker/-/mocker-2.1.9.tgz", + "integrity": "sha512-tVL6uJgoUdi6icpxmdrn5YNo3g3Dxv+IHJBr0GXHaEdTcw3F+cPKnsXFhli6nO+f/6SDKPHEK1UN+k+TQv0Ehg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/spy": "2.1.9", + "estree-walker": "^3.0.3", + "magic-string": "^0.30.12" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "msw": "^2.4.9", + "vite": "^5.0.0" + }, + "peerDependenciesMeta": { + "msw": { + "optional": true + }, + "vite": { + "optional": true + } + } + }, + "node_modules/@vitest/pretty-format": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/pretty-format/-/pretty-format-2.1.9.tgz", + "integrity": "sha512-KhRIdGV2U9HOUzxfiHmY8IFHTdqtOhIzCpd8WRdJiE7D/HUcZVD0EgQCVjm+Q9gkUXWgBvMmTtZgIG48wq7sOQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyrainbow": "^1.2.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/runner": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/runner/-/runner-2.1.9.tgz", + "integrity": "sha512-ZXSSqTFIrzduD63btIfEyOmNcBmQvgOVsPNPe0jYtESiXkhd8u2erDLnMxmGrDCwHCCHE7hxwRDCT3pt0esT4g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/utils": "2.1.9", + "pathe": "^1.1.2" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/snapshot": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/snapshot/-/snapshot-2.1.9.tgz", + "integrity": "sha512-oBO82rEjsxLNJincVhLhaxxZdEtV0EFHMK5Kmx5sJ6H9L183dHECjiefOAdnqpIgT5eZwT04PoggUnW88vOBNQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "2.1.9", + "magic-string": "^0.30.12", + "pathe": "^1.1.2" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/spy": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/spy/-/spy-2.1.9.tgz", + "integrity": "sha512-E1B35FwzXXTs9FHNK6bDszs7mtydNi5MIfUWpceJ8Xbfb1gBMscAnwLbEu+B44ed6W3XjL9/ehLPHR1fkf1KLQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "tinyspy": "^3.0.2" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/@vitest/utils": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/@vitest/utils/-/utils-2.1.9.tgz", + "integrity": "sha512-v0psaMSkNJ3A2NMrUEHFRzJtDPFn+/VWZ5WxImB21T9fjucJRmS7xCS3ppEnARb9y11OAzaD+P2Ps+b+BGX5iQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/pretty-format": "2.1.9", + "loupe": "^3.1.2", + "tinyrainbow": "^1.2.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/assertion-error": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/assertion-error/-/assertion-error-2.0.1.tgz", + "integrity": "sha512-Izi8RQcffqCeNVgFigKli1ssklIbpHnCYc6AknXGYoB6grJqyeby7jv12JUQgmTAnIDnbck1uxksT4dzN3PWBA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=12" + } + }, + "node_modules/cac": { + "version": "6.7.14", + "resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz", + "integrity": "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/chai": { + "version": "5.3.3", + "resolved": "https://registry.npmjs.org/chai/-/chai-5.3.3.tgz", + "integrity": "sha512-4zNhdJD/iOjSH0A05ea+Ke6MU5mmpQcbQsSOkgdaUMJ9zTlDTD/GYlwohmIE2u0gaxHYiVHEn1Fw9mZ/ktJWgw==", + "dev": true, + "license": "MIT", + "dependencies": { + "assertion-error": "^2.0.1", + "check-error": "^2.1.1", + "deep-eql": "^5.0.1", + "loupe": "^3.1.0", + "pathval": "^2.0.0" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/check-error": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/check-error/-/check-error-2.1.3.tgz", + "integrity": "sha512-PAJdDJusoxnwm1VwW07VWwUN1sl7smmC3OKggvndJFadxxDRyFJBX/ggnu/KE4kQAB7a3Dp8f/YXC1FlUprWmA==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 16" + } + }, + "node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "dev": true, + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, + "node_modules/deep-eql": { + "version": "5.0.2", + "resolved": "https://registry.npmjs.org/deep-eql/-/deep-eql-5.0.2.tgz", + "integrity": "sha512-h5k/5U50IJJFpzfL6nO9jaaumfjO/f2NjK/oYB2Djzm4p9L+3T9qWpZqZ2hAbLPuuYq9wrU08WQyBTL5GbPk5Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/es-module-lexer": { + "version": "1.7.0", + "resolved": "https://registry.npmjs.org/es-module-lexer/-/es-module-lexer-1.7.0.tgz", + "integrity": "sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==", + "dev": true, + "license": "MIT" + }, + "node_modules/esbuild": { + "version": "0.21.5", + "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.21.5.tgz", + "integrity": "sha512-mg3OPMV4hXywwpoDxu3Qda5xCKQi+vCTZq8S9J/EpkhB2HzKXq4SNFZE3+NK93JYxc8VMSep+lOUSC/RVKaBqw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "bin": { + "esbuild": "bin/esbuild" + }, + "engines": { + "node": ">=12" + }, + "optionalDependencies": { + "@esbuild/aix-ppc64": "0.21.5", + "@esbuild/android-arm": "0.21.5", + "@esbuild/android-arm64": "0.21.5", + "@esbuild/android-x64": "0.21.5", + "@esbuild/darwin-arm64": "0.21.5", + "@esbuild/darwin-x64": "0.21.5", + "@esbuild/freebsd-arm64": "0.21.5", + "@esbuild/freebsd-x64": "0.21.5", + "@esbuild/linux-arm": "0.21.5", + "@esbuild/linux-arm64": "0.21.5", + "@esbuild/linux-ia32": "0.21.5", + "@esbuild/linux-loong64": "0.21.5", + "@esbuild/linux-mips64el": "0.21.5", + "@esbuild/linux-ppc64": "0.21.5", + "@esbuild/linux-riscv64": "0.21.5", + "@esbuild/linux-s390x": "0.21.5", + "@esbuild/linux-x64": "0.21.5", + "@esbuild/netbsd-x64": "0.21.5", + "@esbuild/openbsd-x64": "0.21.5", + "@esbuild/sunos-x64": "0.21.5", + "@esbuild/win32-arm64": "0.21.5", + "@esbuild/win32-ia32": "0.21.5", + "@esbuild/win32-x64": "0.21.5" + } + }, + "node_modules/estree-walker": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/estree-walker/-/estree-walker-3.0.3.tgz", + "integrity": "sha512-7RUKfXgSMMkzt6ZuXmqapOurLGPPfgj6l9uRZ7lRGolvk0y2yocc35LdcxKC5PQZdn2DMqioAQ2NoWcrTKmm6g==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "^1.0.0" + } + }, + "node_modules/expect-type": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.4.0.tgz", + "integrity": "sha512-KfYbmpRm0VbLjEvVa9yGwCi9GI34xvi7A/HXYWQO65CSD2u3MczUJSuwXKFIxlGsgBQizV9q5J9NHj4VG0n+pA==", + "dev": true, + "license": "Apache-2.0", + "engines": { + "node": ">=12.0.0" + } + }, + "node_modules/fsevents": { + "version": "2.3.3", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.3.tgz", + "integrity": "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw==", + "dev": true, + "hasInstallScript": true, + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": "^8.16.0 || ^10.6.0 || >=11.0.0" + } + }, + "node_modules/loupe": { + "version": "3.2.1", + "resolved": "https://registry.npmjs.org/loupe/-/loupe-3.2.1.tgz", + "integrity": "sha512-CdzqowRJCeLU72bHvWqwRBBlLcMEtIvGrlvef74kMnV2AolS9Y8xUv1I0U/MNAWMhBlKIoyuEgoJ0t/bbwHbLQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/magic-string": { + "version": "0.30.21", + "resolved": "https://registry.npmjs.org/magic-string/-/magic-string-0.30.21.tgz", + "integrity": "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@jridgewell/sourcemap-codec": "^1.5.5" + } + }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "dev": true, + "license": "MIT" + }, + "node_modules/nanoid": { + "version": "3.3.15", + "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.15.tgz", + "integrity": "sha512-y7Wygv/7mEOvxTuEQDB8StXdMRBWf1kR/tlhAzBRUFkB2jfcLOAxO/SHmOO2zgz1pVgK29/kyupn059/bCHdjA==", + "dev": true, + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "bin": { + "nanoid": "bin/nanoid.cjs" + }, + "engines": { + "node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1" + } + }, + "node_modules/pathe": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/pathe/-/pathe-1.1.2.tgz", + "integrity": "sha512-whLdWMYL2TwI08hn8/ZqAbrVemu0LNaNNJZX73O6qaIdCTfXutsLhMkjdENX0qhsQ9uIimo4/aQOmXkoon2nDQ==", + "dev": true, + "license": "MIT" + }, + "node_modules/pathval": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/pathval/-/pathval-2.0.1.tgz", + "integrity": "sha512-//nshmD55c46FuFw26xV/xFAaB5HF9Xdap7HJBBnrKdAd6/GxDBaNA1870O79+9ueg61cZLSVc+OaFlfmObYVQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">= 14.16" + } + }, + "node_modules/picocolors": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz", + "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", + "dev": true, + "license": "ISC" + }, + "node_modules/postcss": { + "version": "8.5.16", + "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.16.tgz", + "integrity": "sha512-vuwillviilfKZsg0VGj5R/YwwcHx4SLsIOI/7K6mQkWx+l5cUHTjj5g0AasTBcyXsbfTgrwsUNmVUb5xVwyPwg==", + "dev": true, + "funding": [ + { + "type": "opencollective", + "url": "https://opencollective.com/postcss/" + }, + { + "type": "tidelift", + "url": "https://tidelift.com/funding/github/npm/postcss" + }, + { + "type": "github", + "url": "https://github.com/sponsors/ai" + } + ], + "license": "MIT", + "dependencies": { + "nanoid": "^3.3.12", + "picocolors": "^1.1.1", + "source-map-js": "^1.2.1" + }, + "engines": { + "node": "^10 || ^12 || >=14" + } + }, + "node_modules/rollup": { + "version": "4.62.2", + "resolved": "https://registry.npmjs.org/rollup/-/rollup-4.62.2.tgz", + "integrity": "sha512-RFnrW4lhXA3s3eqHDZvN654g8OTjzRfqpIRJYczCGB6HzphckVAi/Qh4tbPUbRuDi7s1Llv8g/NspLkttY3gTA==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/estree": "1.0.9" + }, + "bin": { + "rollup": "dist/bin/rollup" + }, + "engines": { + "node": ">=18.0.0", + "npm": ">=8.0.0" + }, + "optionalDependencies": { + "@rollup/rollup-android-arm-eabi": "4.62.2", + "@rollup/rollup-android-arm64": "4.62.2", + "@rollup/rollup-darwin-arm64": "4.62.2", + "@rollup/rollup-darwin-x64": "4.62.2", + "@rollup/rollup-freebsd-arm64": "4.62.2", + "@rollup/rollup-freebsd-x64": "4.62.2", + "@rollup/rollup-linux-arm-gnueabihf": "4.62.2", + "@rollup/rollup-linux-arm-musleabihf": "4.62.2", + "@rollup/rollup-linux-arm64-gnu": "4.62.2", + "@rollup/rollup-linux-arm64-musl": "4.62.2", + "@rollup/rollup-linux-loong64-gnu": "4.62.2", + "@rollup/rollup-linux-loong64-musl": "4.62.2", + "@rollup/rollup-linux-ppc64-gnu": "4.62.2", + "@rollup/rollup-linux-ppc64-musl": "4.62.2", + "@rollup/rollup-linux-riscv64-gnu": "4.62.2", + "@rollup/rollup-linux-riscv64-musl": "4.62.2", + "@rollup/rollup-linux-s390x-gnu": "4.62.2", + "@rollup/rollup-linux-x64-gnu": "4.62.2", + "@rollup/rollup-linux-x64-musl": "4.62.2", + "@rollup/rollup-openbsd-x64": "4.62.2", + "@rollup/rollup-openharmony-arm64": "4.62.2", + "@rollup/rollup-win32-arm64-msvc": "4.62.2", + "@rollup/rollup-win32-ia32-msvc": "4.62.2", + "@rollup/rollup-win32-x64-gnu": "4.62.2", + "@rollup/rollup-win32-x64-msvc": "4.62.2", + "fsevents": "~2.3.2" + } + }, + "node_modules/siginfo": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/siginfo/-/siginfo-2.0.0.tgz", + "integrity": "sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==", + "dev": true, + "license": "ISC" + }, + "node_modules/source-map-js": { + "version": "1.2.1", + "resolved": "https://registry.npmjs.org/source-map-js/-/source-map-js-1.2.1.tgz", + "integrity": "sha512-UXWMKhLOwVKb728IUtQPXxfYU+usdybtUrK/8uGE8CQMvrhOpwvzDBwj0QhSL7MQc7vIsISBG8VQ8+IDQxpfQA==", + "dev": true, + "license": "BSD-3-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/stackback": { + "version": "0.0.2", + "resolved": "https://registry.npmjs.org/stackback/-/stackback-0.0.2.tgz", + "integrity": "sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==", + "dev": true, + "license": "MIT" + }, + "node_modules/std-env": { + "version": "3.10.0", + "resolved": "https://registry.npmjs.org/std-env/-/std-env-3.10.0.tgz", + "integrity": "sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinybench": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz", + "integrity": "sha512-0+DUvqWMValLmha6lr4kD8iAMK1HzV0/aKnCtWb9v9641TnP/MFb7Pc2bxoxQjTXAErryXVgUOfv2YqNllqGeg==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinyexec": { + "version": "0.3.2", + "resolved": "https://registry.npmjs.org/tinyexec/-/tinyexec-0.3.2.tgz", + "integrity": "sha512-KQQR9yN7R5+OSwaK0XQoj22pwHoTlgYqmUscPYoknOoWCWfj/5/ABTMRi69FrKU5ffPVh5QcFikpWJI/P1ocHA==", + "dev": true, + "license": "MIT" + }, + "node_modules/tinypool": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/tinypool/-/tinypool-1.1.1.tgz", + "integrity": "sha512-Zba82s87IFq9A9XmjiX5uZA/ARWDrB03OHlq+Vw1fSdt0I+4/Kutwy8BP4Y/y/aORMo61FQ0vIb5j44vSo5Pkg==", + "dev": true, + "license": "MIT", + "engines": { + "node": "^18.0.0 || >=20.0.0" + } + }, + "node_modules/tinyrainbow": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/tinyrainbow/-/tinyrainbow-1.2.0.tgz", + "integrity": "sha512-weEDEq7Z5eTHPDh4xjX789+fHfF+P8boiFB+0vbWzpbnbsEr/GRaohi/uMKxg8RZMXnl1ItAi/IUHWMsjDV7kQ==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/tinyspy": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/tinyspy/-/tinyspy-3.0.2.tgz", + "integrity": "sha512-n1cw8k1k0x4pgA2+9XrOkFydTerNcJ1zWCO5Nn9scWHTD+5tp8dghT2x1uduQePZTZgd3Tupf+x9BxJjeJi77Q==", + "dev": true, + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, + "node_modules/vite": { + "version": "5.4.21", + "resolved": "https://registry.npmjs.org/vite/-/vite-5.4.21.tgz", + "integrity": "sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==", + "dev": true, + "license": "MIT", + "dependencies": { + "esbuild": "^0.21.3", + "postcss": "^8.4.43", + "rollup": "^4.20.0" + }, + "bin": { + "vite": "bin/vite.js" + }, + "engines": { + "node": "^18.0.0 || >=20.0.0" + }, + "funding": { + "url": "https://github.com/vitejs/vite?sponsor=1" + }, + "optionalDependencies": { + "fsevents": "~2.3.3" + }, + "peerDependencies": { + "@types/node": "^18.0.0 || >=20.0.0", + "less": "*", + "lightningcss": "^1.21.0", + "sass": "*", + "sass-embedded": "*", + "stylus": "*", + "sugarss": "*", + "terser": "^5.4.0" + }, + "peerDependenciesMeta": { + "@types/node": { + "optional": true + }, + "less": { + "optional": true + }, + "lightningcss": { + "optional": true + }, + "sass": { + "optional": true + }, + "sass-embedded": { + "optional": true + }, + "stylus": { + "optional": true + }, + "sugarss": { + "optional": true + }, + "terser": { + "optional": true + } + } + }, + "node_modules/vite-node": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/vite-node/-/vite-node-2.1.9.tgz", + "integrity": "sha512-AM9aQ/IPrW/6ENLQg3AGY4K1N2TGZdR5e4gu/MmmR2xR3Ll1+dib+nook92g4TV3PXVyeyxdWwtaCAiUL0hMxA==", + "dev": true, + "license": "MIT", + "dependencies": { + "cac": "^6.7.14", + "debug": "^4.3.7", + "es-module-lexer": "^1.5.4", + "pathe": "^1.1.2", + "vite": "^5.0.0" + }, + "bin": { + "vite-node": "vite-node.mjs" + }, + "engines": { + "node": "^18.0.0 || >=20.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + } + }, + "node_modules/vitest": { + "version": "2.1.9", + "resolved": "https://registry.npmjs.org/vitest/-/vitest-2.1.9.tgz", + "integrity": "sha512-MSmPM9REYqDGBI8439mA4mWhV5sKmDlBKWIYbA3lRb2PTHACE0mgKwA8yQ2xq9vxDTuk4iPrECBAEW2aoFXY0Q==", + "dev": true, + "license": "MIT", + "dependencies": { + "@vitest/expect": "2.1.9", + "@vitest/mocker": "2.1.9", + "@vitest/pretty-format": "^2.1.9", + "@vitest/runner": "2.1.9", + "@vitest/snapshot": "2.1.9", + "@vitest/spy": "2.1.9", + "@vitest/utils": "2.1.9", + "chai": "^5.1.2", + "debug": "^4.3.7", + "expect-type": "^1.1.0", + "magic-string": "^0.30.12", + "pathe": "^1.1.2", + "std-env": "^3.8.0", + "tinybench": "^2.9.0", + "tinyexec": "^0.3.1", + "tinypool": "^1.0.1", + "tinyrainbow": "^1.2.0", + "vite": "^5.0.0", + "vite-node": "2.1.9", + "why-is-node-running": "^2.3.0" + }, + "bin": { + "vitest": "vitest.mjs" + }, + "engines": { + "node": "^18.0.0 || >=20.0.0" + }, + "funding": { + "url": "https://opencollective.com/vitest" + }, + "peerDependencies": { + "@edge-runtime/vm": "*", + "@types/node": "^18.0.0 || >=20.0.0", + "@vitest/browser": "2.1.9", + "@vitest/ui": "2.1.9", + "happy-dom": "*", + "jsdom": "*" + }, + "peerDependenciesMeta": { + "@edge-runtime/vm": { + "optional": true + }, + "@types/node": { + "optional": true + }, + "@vitest/browser": { + "optional": true + }, + "@vitest/ui": { + "optional": true + }, + "happy-dom": { + "optional": true + }, + "jsdom": { + "optional": true + } + } + }, + "node_modules/why-is-node-running": { + "version": "2.3.0", + "resolved": "https://registry.npmjs.org/why-is-node-running/-/why-is-node-running-2.3.0.tgz", + "integrity": "sha512-hUrmaWBdVDcxvYqnyh09zunKzROWjbZTiNy8dBEjkS7ehEDQibXJ7XvlmtbwuTclUiIyN+CyXQD4Vmko8fNm8w==", + "dev": true, + "license": "MIT", + "dependencies": { + "siginfo": "^2.0.0", + "stackback": "0.0.2" + }, + "bin": { + "why-is-node-running": "cli.js" + }, + "engines": { + "node": ">=8" + } + } + } +} diff --git a/programs/wsproxy/tests/package.json b/programs/wsproxy/tests/package.json new file mode 100644 index 000000000000..a54bb4affd88 --- /dev/null +++ b/programs/wsproxy/tests/package.json @@ -0,0 +1,15 @@ +{ + "name": "wsproxy-tests", + "version": "0.1.0", + "private": true, + "type": "module", + "description": "Integration tests for clickhouse-wsproxy (WebSocket -> native protocol proxy)", + "scripts": { + "test": "vitest run", + "watch": "vitest", + "test:server": "vitest run --config vitest.server.config.mjs" + }, + "devDependencies": { + "vitest": "^2.1.9" + } +} diff --git a/programs/wsproxy/tests/test/adversarial.test.mjs b/programs/wsproxy/tests/test/adversarial.test.mjs new file mode 100644 index 000000000000..1075df1e1b5a --- /dev/null +++ b/programs/wsproxy/tests/test/adversarial.test.mjs @@ -0,0 +1,430 @@ +// Adversarial WebSocket-frame tests for the clickhouse-wsproxy. +// +// These exercise the proxy's RFC 6455 hardening: client->server frames MUST be +// masked, reserved opcodes and RSV bits are protocol errors, over-large frames +// are rejected before allocation, control frames (ping) are answered with a +// pong, and — most importantly — none of the above takes down the proxy for +// other clients. We use the raw-TCP client because the native WebSocket cannot +// emit malformed frames. + +import { RawClient } from "./raw.mjs"; +import { backendScalar, runQuery, sleep } from "./helpers.mjs"; +import { describe, it, expect } from "vitest"; + +// A protocol violation should make the proxy tear the connection down. The spec +// allows either an outright TCP close or a close (0x8) control frame first, so +// we accept both: read frames until we either see a close-opcode frame or the +// socket goes away. +async function closedOrCloseFrame(c, timeoutMs = 10000) { + const deadline = Date.now() + timeoutMs; + for (;;) { + const remaining = deadline - Date.now(); + if (remaining <= 0) break; + if (await c.waitClose(0)) return true; + const f = await Promise.race([ + c.readFrame(), + new Promise((r) => setTimeout(() => r(null), Math.min(remaining, 300))), + ]); + if (f === null) continue; // timed out waiting for a frame; loop and re-check + if (f.closed) return true; + if (f.opcode === 0x8) return true; // close frame + // Anything else (e.g. a stray pong) is ignored; keep waiting for the close. + } + return c.closed; +} + +async function closeCode(c, timeoutMs = 10000) { + const deadline = Date.now() + timeoutMs; + for (;;) { + const remaining = deadline - Date.now(); + if (remaining <= 0) return null; + const frame = await Promise.race([ + c.readFrame(), + new Promise((resolve) => setTimeout(() => resolve(null), remaining)), + ]); + if (frame === null || frame.closed) return null; + if (frame.opcode === 0x8) { + return frame.payload.length >= 2 ? frame.payload.readUInt16BE(0) : null; + } + } +} + +describe("adversarial frames", () => { + it( + "rejects an unmasked client frame", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + // Client->server frames must be masked; an unmasked one is a violation. + c.sendFrame({ opcode: 0x1, payload: "SELECT 1", masked: false }); + expect(await closeCode(c)).toBe(1002); + c.close(); + }, + 10000, + ); + + it( + "rejects a reserved opcode (0x3)", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + // 0x3-0x7 are reserved non-control opcodes. + c.sendFrame({ opcode: 0x3, payload: "x" }); + expect(await closeCode(c)).toBe(1002); + c.close(); + }, + 10000, + ); + + it( + "rejects a frame with an RSV bit set", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + // No extension was negotiated, so any RSV bit is a protocol error. + c.sendFrame({ opcode: 0x1, payload: "SELECT 1", rsv: 1 }); + expect(await closeCode(c)).toBe(1002); + c.close(); + }, + 10000, + ); + + it("closes with 1002 for an invalid close status code", async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + const payload = Buffer.alloc(2); + payload.writeUInt16BE(1005); + c.sendFrame({ opcode: 0x8, payload }); + expect(await closeCode(c)).toBe(1002); + c.close(); + }); + + it("closes with 1007 for an invalid UTF-8 close reason", async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ opcode: 0x8, payload: Buffer.from([0x03, 0xe8, 0xc3, 0x28]) }); + expect(await closeCode(c)).toBe(1007); + c.close(); + }); + + it("closes with 1007 for an invalid UTF-8 text message", async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ opcode: 0x1, payload: Buffer.from([0xc3, 0x28]) }); + expect(await closeCode(c)).toBe(1007); + c.close(); + }); + + it.each([126, 127])("closes with 1002 for non-minimal %i-bit length encoding", async (encoding) => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ opcode: 0x1, payload: "x", lengthEncoding: encoding }); + expect(await closeCode(c)).toBe(1002); + c.close(); + }); + + it("closes with 1002 when the 64-bit payload length has its high bit set", async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.socket.write(Buffer.from([0x81, 0xff, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00])); + expect(await closeCode(c)).toBe(1002); + c.close(); + }); + + it( + "rejects an oversized frame before allocating", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + // Advertise 100 MiB in the header but send no payload bytes. The proxy + // must cap on the advertised length (> 16 MiB) and never allocate it. + c.sendFrame({ opcode: 0x1, payload: Buffer.alloc(0), advertisedLen: 100 * 1024 * 1024 }); + expect(await closeCode(c)).toBe(1009); + c.close(); + }, + 10000, + ); + + it( + "rejects a fragmented message whose aggregate payload is oversized", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + const chunk = Buffer.alloc(8 * 1024 * 1024, 0x20); + c.sendFrame({ opcode: 0x1, payload: chunk, fin: false }); + c.sendFrame({ opcode: 0x0, payload: chunk, fin: false }); + c.sendFrame({ opcode: 0x0, payload: "x", fin: true }); + expect(await closeCode(c, 20000)).toBe(1009); + c.close(); + }, + 30000, + ); + + it( + "rejects a continuation frame without a fragmented message", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ opcode: 0x0, payload: "SELECT 1" }); + expect(await closeCode(c)).toBe(1002); + c.close(); + }, + 10000, + ); + + it( + "rejects a new data frame during a fragmented message", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ opcode: 0x1, payload: "SELECT ", fin: false }); + c.sendFrame({ opcode: 0x1, payload: "1", fin: true }); + expect(await closeCode(c)).toBe(1002); + c.close(); + }, + 10000, + ); + + it( + "allows a ping interleaved with a fragmented query", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ opcode: 0x1, payload: "SELECT ", fin: false }); + c.sendFrame({ opcode: 0x9, payload: "hb" }); + c.sendFrame({ opcode: 0x0, payload: "1 AS n", fin: true }); + + const binary = []; + let sawPong = false; + let sawEnd = false; + for (;;) { + const f = await c.readFrame(); + if (f.closed) break; + if (f.opcode === 0xa) { + sawPong = f.payload.toString() === "hb"; + } else if (f.opcode === 0x2) { + binary.push(f.payload); + } else if (f.opcode === 0x1) { + const event = JSON.parse(f.payload.toString()); + if (event.event === "end") { + sawEnd = true; + break; + } + } + } + expect(sawPong).toBe(true); + expect(sawEnd).toBe(true); + expect(Buffer.concat(binary).toString()).toContain('{"n":1}'); + c.close(); + }, + 10000, + ); + + it( + "bounds a partial control frame received during a query", + async () => { + const marker = `PARTIAL_FRAME_${Date.now()}`; + const countSql = + `SELECT count() FROM system.processes WHERE query LIKE '%${marker}%' ` + + "AND query NOT LIKE '%system.processes%'"; + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ + opcode: 0x1, + payload: + `SELECT sleepEachRow(0.2), number FROM numbers(50) ` + + `SETTINGS max_block_size = 1 -- ${marker}`, + }); + + let streaming = false; + for (;;) { + const frame = await c.readFrame(); + if (frame.closed || frame.opcode === 0x8) break; + if (frame.opcode === 0x2) { + streaming = true; + break; + } + } + expect(streaming).toBe(true); + expect(await backendScalar(countSql)).not.toBe("0"); + + // Send only the first header byte. Mid-query polling must not block on + // the missing second byte for the socket's normal five-minute timeout. + c.socket.write(Buffer.from([0x81])); + expect(await closedOrCloseFrame(c, 5000)).toBe(true); + c.close(); + + let running = "1"; + for (let i = 0; i < 20; i++) { + running = await backendScalar(countSql); + if (running === "0") break; + await sleep(250); + } + expect(running).toBe("0"); + + const r = await runQuery("SELECT 1 AS n"); + expect(r.control.event).toBe("end"); + expect(JSON.parse(r.text.trim()).n).toBe(1); + }, + 15000, + ); + + it( + "bounds a partial control frame while waiting for flow credit", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow&flow=1"); + c.sendFrame({ + opcode: 0x1, + payload: "SELECT number FROM numbers(100) SETTINGS max_block_size = 1", + }); + + for (;;) { + const frame = await c.readFrame(); + if (frame.closed || frame.opcode === 0x8) throw new Error("query closed before using its initial credit"); + if (frame.opcode === 0x2) break; + } + + c.socket.write(Buffer.from([0x81])); + expect(await closedOrCloseFrame(c, 5000)).toBe(true); + c.close(); + }, + 10000, + ); + + it("rejects a continuation frame without a message during an insert", async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ + opcode: 0x1, + payload: JSON.stringify({ + cmd: "insert", + query: "INSERT INTO default.wsp_test FORMAT JSONEachRow", + format: "JSONEachRow", + }), + }); + await sleep(50); + c.sendFrame({ opcode: 0x0, payload: '{"a":1,"b":"x"}\n' }); + expect(await closeCode(c)).toBe(1002); + c.close(); + }); + + it("reassembles a fragmented binary message during an insert", async () => { + const marker = `fragmented-insert-${Date.now()}`; + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ + opcode: 0x1, + payload: JSON.stringify({ + cmd: "insert", + query: "INSERT INTO default.wsp_test FORMAT JSONEachRow", + format: "JSONEachRow", + }), + }); + const row = JSON.stringify({ a: 7, b: marker }) + "\n"; + const split = Math.floor(row.length / 2); + c.sendFrame({ opcode: 0x2, payload: row.slice(0, split), fin: false }); + c.sendFrame({ opcode: 0x9, payload: "insert-ping" }); + c.sendFrame({ opcode: 0x0, payload: row.slice(split), fin: true }); + c.sendFrame({ opcode: 0x2, payload: Buffer.alloc(0) }); + + let sawPong = false; + let sawEnd = false; + for (;;) { + const frame = await c.readFrame(); + if (frame.closed || frame.opcode === 0x8) break; + if (frame.opcode === 0xa) sawPong = frame.payload.toString() === "insert-ping"; + if (frame.opcode === 0x1 && JSON.parse(frame.payload.toString()).event === "end") { + sawEnd = true; + break; + } + } + c.close(); + + expect(sawPong).toBe(true); + expect(sawEnd).toBe(true); + expect(await backendScalar(`SELECT count() FROM default.wsp_test WHERE b = '${marker}'`)).toBe("1"); + }); + + it( + "answers a ping with a matching pong", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ opcode: 0x9, payload: "hb" }); + const f = await c.readFrame(); + expect(f.opcode).toBe(0xa); + expect(f.payload.toString()).toBe("hb"); + c.close(); + }, + 10000, + ); + + it( + "serves a valid masked query through the raw client", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + c.sendFrame({ opcode: 0x1, payload: "SELECT 1 AS n" }); + + const binary = []; + let sawEnd = false; + for (;;) { + const f = await c.readFrame(); + if (f.closed) break; + if (f.opcode === 0x2) { + // Binary frame: query result bytes. + binary.push(f.payload); + continue; + } + if (f.opcode === 0x1) { + // Text frame: a control event. + const msg = JSON.parse(f.payload.toString()); + // Non-terminal mid-query pushes: keep reading. + if (msg.event === "progress" || msg.event === "log" || msg.event === "profile_events") + continue; + if (msg.event === "end") { + sawEnd = true; + break; + } + // Any other control event (e.g. error) is unexpected here. + throw new Error(`unexpected control event: ${f.payload.toString()}`); + } + // Ignore stray control frames (ping/pong/close-with-data), keep reading. + } + + expect(sawEnd).toBe(true); + expect(Buffer.concat(binary).toString()).toContain('{"n":1}'); + c.close(); + }, + 10000, + ); + + it( + "does not throw on an idle connect-then-disconnect", + async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow"); + // Never send a query; just drop the connection. + c.close(); + + // The proxy must still serve new clients afterwards. + const r = await runQuery("SELECT 1 AS n"); + expect(r.control.event).toBe("end"); + expect(JSON.parse(r.text.trim()).n).toBe(1); + }, + 10000, + ); + + // Runs last: after all the malformed traffic above, a fresh normal client + // must still work, proving the proxy process survived every violation. + it( + "survives malformed input and keeps serving other clients", + async () => { + const r = await runQuery("SELECT 42 AS n"); + expect(r.control.event).toBe("end"); + expect(JSON.parse(r.text.trim()).n).toBe(42); + }, + 10000, + ); +}); diff --git a/programs/wsproxy/tests/test/auth.test.mjs b/programs/wsproxy/tests/test/auth.test.mjs new file mode 100644 index 000000000000..3d949d001da1 --- /dev/null +++ b/programs/wsproxy/tests/test/auth.test.mjs @@ -0,0 +1,151 @@ +import { describe, it, expect } from "vitest"; +import { runQuery, Session } from "./helpers.mjs"; +import { RawClient } from "./raw.mjs"; + +// The test backend (tmp/ch/users.xml) has a password-protected user: +const USER = "wsp_user"; +const PASS = "wsp_pass"; + +// Run `SELECT currentUser()` through a raw client that authenticates via the +// given HTTP request headers; returns the authenticated user name. +async function currentUserViaHeaders(headers) { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow", headers); + c.sendFrame({ opcode: 0x1, payload: "SELECT currentUser() AS u" }); + const chunks = []; + let control = null; + for (;;) { + const f = await c.readFrame(); + if (f.closed) break; + if (f.opcode === 0x2) chunks.push(f.payload); + else if (f.opcode === 0x1) { + const ev = JSON.parse(f.payload.toString()); + if (ev.event === "progress" || ev.event === "log" || ev.event === "profile_events") continue; + control = ev; + break; + } + } + c.close(); + const text = Buffer.concat(chunks).toString(); + return { control, user: text.trim() ? JSON.parse(text.trim()).u : null }; +} + +describe("auth (credential pass-through)", () => { + it("rejects a cross-origin browser handshake", async () => { + const c = new RawClient({ port: 9010 }); + const { head } = await c.handshakeResponse("/?format=JSONEachRow", { + Origin: "https://attacker.example", + }); + expect(head).toMatch(/^HTTP\/1\.1 403 /); + expect(head).not.toContain("101 Switching Protocols"); + c.close(); + }); + + it("allows a browser origin from the configured allowlist", async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow", { Origin: "https://trusted.example" }); + c.sendFrame({ opcode: 0x1, payload: "SELECT 1 AS n" }); + let sawEnd = false; + for (;;) { + const f = await c.readFrame(); + if (f.closed) break; + if (f.opcode === 0x1 && JSON.parse(f.payload.toString()).event === "end") { + sawEnd = true; + break; + } + } + expect(sawEnd).toBe(true); + c.close(); + }); + + it("allows a non-browser handshake without an Origin header", async () => { + const c = new RawClient({ port: 9010 }); + await c.handshake("/?format=JSONEachRow", { Origin: undefined }); + c.sendFrame({ opcode: 0x1, payload: "SELECT 1 AS n" }); + let sawEnd = false; + for (;;) { + const f = await c.readFrame(); + if (f.closed) break; + if (f.opcode === 0x1 && JSON.parse(f.payload.toString()).event === "end") { + sawEnd = true; + break; + } + } + expect(sawEnd).toBe(true); + c.close(); + }); + + it("rejects malformed explicit Basic credentials without falling back", async () => { + const c = new RawClient({ port: 9010 }); + const { head } = await c.handshakeResponse("/?format=JSONEachRow", { + Authorization: "Basic !!!not-base64!!!", + "X-ClickHouse-User": USER, + "X-ClickHouse-Key": PASS, + }); + expect(head).toMatch(/^HTTP\/1\.1 400 /); + expect(head).not.toContain("101 Switching Protocols"); + c.close(); + }); + + it("authenticates as the default user when no credentials are given", async () => { + const { text, control } = await runQuery("SELECT currentUser() AS u"); + expect(control.event).toBe("end"); + expect(JSON.parse(text.trim()).u).toBe("default"); + }); + + it("authenticates via ?user=/?password= URL params", async () => { + const { text, control } = await runQuery("SELECT currentUser() AS u", { + user: USER, + password: PASS, + }); + expect(control.event).toBe("end"); + expect(JSON.parse(text.trim()).u).toBe(USER); + }); + + it("rejects a wrong password (backend auth error passed through)", async () => { + const { control } = await runQuery("SELECT 1", { user: USER, password: "wrong" }); + expect(control.event).toBe("error"); + expect(control.message).toMatch(/auth|password|ACCESS|denied/i); + }); + + it("authenticates via Authorization: Basic header", async () => { + const basic = Buffer.from(`${USER}:${PASS}`).toString("base64"); + const { control, user } = await currentUserViaHeaders({ Authorization: `Basic ${basic}` }); + expect(control.event).toBe("end"); + expect(user).toBe(USER); + }); + + it("authenticates via X-ClickHouse-User / X-ClickHouse-Key headers", async () => { + const { control, user } = await currentUserViaHeaders({ + "X-ClickHouse-User": USER, + "X-ClickHouse-Key": PASS, + }); + expect(control.event).toBe("end"); + expect(user).toBe(USER); + }); + + it("rejects bad credentials eagerly, before any query is sent", async () => { + const s = new Session("JSONEachRow", { user: USER, password: "wrong" }); + await s.ready(); + // Do NOT send a query. Eager connect authenticates at session start, so the + // failure must arrive without the client sending anything. + let event = null; + for (let i = 0; i < 4; i++) { + const f = await s.nextFrame(); + if (f.type === "text") { + event = JSON.parse(f.data).event; + break; + } + if (f.type === "close") break; + } + s.close(); + expect(event).toBe("error"); + }, 10000); + + it("rejects a wrong password sent via Basic header", async () => { + const basic = Buffer.from(`${USER}:wrong`).toString("base64"); + const { control } = await currentUserViaHeaders({ Authorization: `Basic ${basic}` }); + expect(control.event).toBe("error"); + expect(control.message).toMatch(/auth|password|ACCESS|denied/i); + }); +}); diff --git a/programs/wsproxy/tests/test/backend-failure.test.mjs b/programs/wsproxy/tests/test/backend-failure.test.mjs new file mode 100644 index 000000000000..e72ed7c10354 --- /dev/null +++ b/programs/wsproxy/tests/test/backend-failure.test.mjs @@ -0,0 +1,67 @@ +import { describe, it, expect } from "vitest"; +import { Session, runQuery } from "./helpers.mjs"; +import { spawnBackend, spawnProxy } from "./proc.mjs"; + +// These tests run their OWN backend + proxy pair (on non-default ports) so they +// can kill the backend without disturbing the shared stack used by other files. +describe("backend failure", () => { + it("delivers an error (no hang or crash) when the backend drops mid-query", async () => { + const backend = await spawnBackend({ tcpPort: 9001 }); + const proxy = await spawnProxy({ listenPort: 9013, backendPort: 9001 }); + try { + const s = new Session("JSONEachRow", { baseUrl: proxy.url }); + await s.ready(); + // A long, slowly-streaming query so we can kill the backend mid-flight. + s.sendQuery("SELECT sleepEachRow(0.2), number FROM numbers(100) SETTINGS max_block_size = 1"); + + // Wait until the query is actually streaming (first binary frame). Skip + // non-terminal text frames (progress/log/profile_events); only stop early + // on a close or a terminal control event. + let streaming = false; + for (let i = 0; i < 200; i++) { + const f = await s.nextFrame(); + if (f.type === "binary") { + streaming = true; + break; + } + if (f.type === "close") break; + if (f.type === "text") { + const ev = JSON.parse(f.data).event; + if (ev === "end" || ev === "error" || ev === "cancelled") break; + } + } + expect(streaming).toBe(true); + + // Kill the backend out from under the running query. + backend.stop(); + + // The proxy must terminate the session cleanly (error or close), not hang. + const rest = await s.collect(); + expect(["error", "closed"]).toContain(rest.control.event); + s.close(); + } finally { + proxy.stop(); + backend.stop(); + } + }, 30000); + + it("keeps the proxy process alive after a backend failure", async () => { + const backend = await spawnBackend({ tcpPort: 9002 }); + const proxy = await spawnProxy({ listenPort: 9014, backendPort: 9002 }); + try { + // Sanity: works before the failure. + const before = await runQuery("SELECT 1 AS n", { baseUrl: proxy.url }); + expect(before.control.event).toBe("end"); + + backend.stop(); + + // After the backend is gone, new sessions get a clean error rather than a + // crash/hang — proving the proxy process survived. + const after = await runQuery("SELECT 1 AS n", { baseUrl: proxy.url }).catch(() => null); + expect(after === null || ["error", "closed"].includes(after.control.event)).toBe(true); + } finally { + proxy.stop(); + backend.stop(); + } + }, 30000); +}); diff --git a/programs/wsproxy/tests/test/backpressure.test.mjs b/programs/wsproxy/tests/test/backpressure.test.mjs new file mode 100644 index 000000000000..fc9fc63e635b --- /dev/null +++ b/programs/wsproxy/tests/test/backpressure.test.mjs @@ -0,0 +1,47 @@ +import { describe, it, expect } from "vitest"; +import { runQuery } from "./helpers.mjs"; +import { RawClient } from "./raw.mjs"; +import { spawnProxy } from "./proc.mjs"; + +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + +describe("backpressure / stalled client", () => { + it("drops a client that stops reading (send timeout) and stays healthy", async () => { + // Short send timeout so the test doesn't wait the 30s default. + const proxy = await spawnProxy({ listenPort: 9017, backendPort: 19000, sendTimeoutSec: 1 }); + try { + const c = new RawClient({ port: 9017 }); + await c.handshake("/?format=JSONEachRow"); + // A large result the client will refuse to read, so the proxy's blocking + // writes fill the socket buffers and then stall. + c.sendFrame({ + opcode: 0x1, + payload: "SELECT number, repeat('x', 1000) AS s FROM numbers(5000000)", + }); + c.pause(); // stop reading -> proxy writes block -> ~1s send timeout should fire + + // Wait past the send timeout, then resume and confirm the proxy dropped us + // (rather than blocking a handler thread forever). + await sleep(4000); + c.resume(); + const start = Date.now(); + let closed = false; + while (Date.now() - start < 10000) { + const f = await c.readFrame(); + if (f.closed) { + closed = true; + break; + } + } + c.close(); + expect(closed).toBe(true); + + // The proxy is still healthy for a fresh client (the stalled thread was released). + const ok = await runQuery("SELECT 1 AS n", { baseUrl: proxy.url }); + expect(ok.control.event).toBe("end"); + expect(JSON.parse(ok.text.trim()).n).toBe(1); + } finally { + proxy.stop(); + } + }, 30000); +}); diff --git a/programs/wsproxy/tests/test/cancel.test.mjs b/programs/wsproxy/tests/test/cancel.test.mjs new file mode 100644 index 000000000000..0cee09e5ac9a --- /dev/null +++ b/programs/wsproxy/tests/test/cancel.test.mjs @@ -0,0 +1,68 @@ +import { describe, it, expect } from "vitest"; +import { Session, backendScalar, sleep } from "./helpers.mjs"; + +describe("cancel", () => { + it( + "cancels a running query server-side", + async () => { + // Unique marker so we can find exactly this query in system.processes. + const marker = "CANCEL_MARKER_" + Date.now(); + + // A long (~10s) query that streams slowly, one row per block, with the + // marker in a trailing comment so it shows up in system.processes. + const sql = `SELECT sleepEachRow(0.2), number FROM numbers(50) SETTINGS max_block_size = 1 -- ${marker}`; + + // Query that counts how many instances of our marker query are running, + // excluding the counting query itself. + const countSql = `SELECT count() FROM system.processes WHERE query LIKE '%${marker}%' AND query NOT LIKE '%system.processes%'`; + + const s = new Session("JSONEachRow"); + await s.ready(); + s.sendQuery(sql); + + // Wait until the query is actually streaming: the first BINARY frame + // proves a block arrived and the query is running on the backend. + let streaming = false; + for (let i = 0; i < 50 && !streaming; i++) { + const frame = await s.nextFrame(); + if (frame.type === "binary") streaming = true; + else if (frame.type === "close") break; // socket died unexpectedly + } + expect(streaming).toBe(true); + + // Sanity check: the query really is visible on the backend. + const running = await backendScalar(countSql); + expect(running).not.toBe("0"); + + // Closing the socket cancels the running query. + const startedAt = Date.now(); + s.close(); + + // Poll for up to ~5s until the marker query is gone from the backend. + let finalCount = running; + for (let i = 0; i < 20; i++) { + finalCount = await backendScalar(countSql); + if (finalCount === "0") break; + await sleep(250); + } + const elapsed = Date.now() - startedAt; + + // The query was cancelled promptly, not left to finish naturally. + expect(finalCount).toBe("0"); + // ~10s natural completion; cancellation must be much faster. + expect(elapsed).toBeLessThan(6000); + }, + 20000, + ); + + it("a normal query is unaffected", async () => { + // The proxy should be perfectly healthy after a cancel. + const s = new Session("JSONEachRow"); + try { + const { control } = await s.run("SELECT 1"); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + }); +}); diff --git a/programs/wsproxy/tests/test/compression.test.mjs b/programs/wsproxy/tests/test/compression.test.mjs new file mode 100644 index 000000000000..5dbdb01ed329 --- /dev/null +++ b/programs/wsproxy/tests/test/compression.test.mjs @@ -0,0 +1,42 @@ +import { describe, it, expect, afterAll } from "vitest"; +import { spawnProxy } from "./proc.mjs"; +import { Session } from "./helpers.mjs"; + +// The native result-block codec (backend -> proxy) is configurable via +// WSPROXY_BACKEND_COMPRESSION (lz4 default | zstd | none). It only changes bytes +// on the wire and backend/proxy CPU; the data delivered to the client must be +// byte-identical across codecs. On a bandwidth-limited WAN, zstd is much smaller +// than lz4 (columnar native compresses far better than row JSON), which is what +// lets the proxy beat gzipped HTTP — see programs/wsproxy/bench/. +describe("backend compression codec", () => { + const BACKEND_PORT = 19000; + const QUERY = "SELECT number AS n, number*2 AS d, toString(number) AS s FROM numbers(50000)"; + const proxies = []; + + afterAll(() => proxies.forEach((p) => p.stop())); + + async function fetchVia(compression, listenPort) { + const proxy = await spawnProxy({ listenPort, backendPort: BACKEND_PORT, compression }); + proxies.push(proxy); + const s = new Session("JSONCompactEachRow", { baseUrl: proxy.url }); + try { + const { text, control } = await s.run(QUERY); + expect(control.event).toBe("end"); + return text; + } finally { + s.close(); + } + } + + it("delivers byte-identical output for lz4, zstd, and none", async () => { + const [lz4, zstd, none] = await Promise.all([ + fetchVia("lz4", 9031), + fetchVia("zstd", 9032), + fetchVia("none", 9033), + ]); + expect(zstd).toBe(lz4); + expect(none).toBe(lz4); + // sanity: the payload is non-trivial + expect(lz4.split("\n").filter((l) => l).length).toBe(50000); + }, 30000); +}); diff --git a/programs/wsproxy/tests/test/concurrency.test.mjs b/programs/wsproxy/tests/test/concurrency.test.mjs new file mode 100644 index 000000000000..e237c7712447 --- /dev/null +++ b/programs/wsproxy/tests/test/concurrency.test.mjs @@ -0,0 +1,72 @@ +import { describe, it, expect } from "vitest"; +import { Session, runQuery } from "./helpers.mjs"; + +// These tests stress the proxy with many simultaneous sessions, long-lived +// sessions, and large streamed results. The proxy uses one native Connection +// per WebSocket session and a Poco thread pool (capacity ~16), so this +// exercises thread-per-session behaviour under load and result streaming. + +const countLines = (text) => text.split("\n").filter((l) => l.length > 0).length; + +describe("concurrency", () => { + it("handles many concurrent independent sessions", async () => { + const N = 40; + const results = await Promise.all( + Array.from({ length: N }, (_, i) => runQuery(`SELECT ${i} AS n`)), + ); + + expect(results).toHaveLength(N); + results.forEach((res, i) => { + expect(res.control.event).toBe("end"); + expect(JSON.parse(res.text.trim()).n).toBe(i); + }); + }, 30000); + + it("handles concurrent slow queries through the thread pool", async () => { + const N = 12; + const sql = + "SELECT sleepEachRow(0.05), number FROM numbers(10) SETTINGS max_block_size = 1"; + const results = await Promise.all( + Array.from({ length: N }, () => runQuery(sql)), + ); + + expect(results).toHaveLength(N); + for (const res of results) { + expect(res.control.event).toBe("end"); + } + }, 30000); + + it("survives many sequential queries on one long-lived session", async () => { + const N = 100; + const s = new Session("JSONEachRow"); + try { + let last; + for (let i = 0; i < N; i++) { + last = await s.run(`SELECT ${i} AS n`); + expect(last.control.event).toBe("end"); + } + expect(JSON.parse(last.text.trim()).n).toBe(N - 1); + } finally { + s.close(); + } + }, 30000); + + it("streams a large result set without dropping rows", async () => { + const { text, control } = await runQuery("SELECT number FROM numbers(1000000)"); + expect(control.event).toBe("end"); + expect(countLines(text)).toBe(1000000); + }, 30000); + + it("handles concurrent large-ish result sets", async () => { + const N = 5; + const results = await Promise.all( + Array.from({ length: N }, () => runQuery("SELECT number FROM numbers(100000)")), + ); + + expect(results).toHaveLength(N); + for (const res of results) { + expect(res.control.event).toBe("end"); + expect(countLines(res.text)).toBe(100000); + } + }, 30000); +}); diff --git a/programs/wsproxy/tests/test/edge.test.mjs b/programs/wsproxy/tests/test/edge.test.mjs new file mode 100644 index 000000000000..88b858faa063 --- /dev/null +++ b/programs/wsproxy/tests/test/edge.test.mjs @@ -0,0 +1,53 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { runQuery, Session, backendScalar } from "./helpers.mjs"; + +describe("edge cases", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it("handles an empty result set", async () => { + const { text, control } = await runQuery("SELECT 1 AS n WHERE 0"); + expect(control.event).toBe("end"); + expect(text.split("\n").filter((l) => l.length > 0).length).toBe(0); + }); + + it("handles LIMIT 0 (header only, no rows)", async () => { + const { text, control } = await runQuery("SELECT number FROM numbers(100) LIMIT 0"); + expect(control.event).toBe("end"); + expect(text.split("\n").filter((l) => l.length > 0).length).toBe(0); + }); + + it("runs DDL and reports end with no data", async () => { + const create = await runQuery( + "CREATE TABLE IF NOT EXISTS default.wsp_edge_ddl (x UInt8) ENGINE = Memory", + ); + expect(create.control.event).toBe("end"); + expect(create.text.trim()).toBe(""); + const drop = await runQuery("DROP TABLE default.wsp_edge_ddl"); + expect(drop.control.event).toBe("end"); + }); + + it("handles many columns and NULLs in one row", async () => { + const { text, control } = await runQuery( + "SELECT 1 AS a, NULL AS b, 'x' AS c, [1,2] AS d, toFloat64(1.5) AS e", + ); + expect(control.event).toBe("end"); + const row = JSON.parse(text.trim()); + expect(row.b).toBe(null); + expect(row.c).toBe("x"); + expect(row.d).toEqual([1, 2]); + }); + + it("handles an empty string and unicode payload round-trip via INSERT", async () => { + const s = new Session("JSONEachRow"); + const ok = await s.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", [ + '{"a":1,"b":""}\n{"a":2,"b":"日本語"}\n', + ]); + s.close(); + expect(ok.control.event).toBe("end"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + expect(await backendScalar("SELECT b FROM default.wsp_test WHERE a = 2")).toBe("日本語"); + expect(await backendScalar("SELECT length(b) FROM default.wsp_test WHERE a = 1")).toBe("0"); + }); +}); diff --git a/programs/wsproxy/tests/test/exceptions.test.mjs b/programs/wsproxy/tests/test/exceptions.test.mjs new file mode 100644 index 000000000000..7e9c3b51b1aa --- /dev/null +++ b/programs/wsproxy/tests/test/exceptions.test.mjs @@ -0,0 +1,78 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { runQuery, Session, backendScalar } from "./helpers.mjs"; + +describe("exceptions", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it("streams partial results, then delivers the exception", async () => { + // throwIf fires at number = 5, so rows 0..4 stream first, then the error. + const { text, control } = await runQuery( + "SELECT number, throwIf(number = 5, 'boom') AS t FROM numbers(10) SETTINGS max_block_size = 1", + ); + expect(control.event).toBe("error"); + expect(control.message).toMatch(/boom/); + const lines = text.split("\n").filter((l) => l.length > 0); + expect(lines.length).toBe(5); // partial results were delivered before the error + // First streamed row is number 0 (quoting-agnostic: the proxy renders UInt64 unquoted). + expect(String(JSON.parse(lines[0]).number)).toBe("0"); + }); + + it("reuses the session after a mid-stream exception", async () => { + const s = new Session("JSONEachRow"); + const bad = await s.run( + "SELECT throwIf(number = 3, 'x') FROM numbers(10) SETTINGS max_block_size = 1", + ); + expect(bad.control.event).toBe("error"); + // The reused Connection must survive an exception that occurred mid-stream. + const good = await s.run("SELECT 1 AS n"); + expect(good.control.event).toBe("end"); + expect(JSON.parse(good.text.trim()).n).toBe(1); + s.close(); + }); + + it("aborts an INSERT on malformed data without committing partial rows", async () => { + const s = new Session("JSONEachRow"); + const { control } = await s.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", [ + '{"a":1,"b":"x"}\n', + "NOT JSON AT ALL\n", + ]); + s.close(); + expect(control.event).toBe("error"); + expect(control.message).toMatch(/Cannot parse|parse input|Code: 27/); + // The valid first row must NOT be committed — the insert aborted. + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("0"); + }); + + it("reports a type error in INSERT data", async () => { + const s = new Session("JSONEachRow"); + const { control } = await s.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", [ + '{"a":"notanumber","b":"x"}\n', + ]); + s.close(); + expect(control.event).toBe("error"); + expect(control.message).toMatch(/parse|Cannot|Code: 27/); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("0"); + }); + + it("preserves the ClickHouse error message and code", async () => { + const { control } = await runQuery("SELECT * FROM default.definitely_missing_tbl_xyz"); + expect(control.event).toBe("error"); + expect(control.message).toContain("definitely_missing_tbl_xyz"); + expect(control.message).toMatch(/Unknown table|UNKNOWN_TABLE|Code: 60|Code: 47/); + }); + + it("keeps the session healthy after an INSERT error", async () => { + const s = new Session("JSONEachRow"); + const bad = await s.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", ["garbage\n"]); + expect(bad.control.event).toBe("error"); + s.close(); + // A fresh session works, and a valid insert then commits. + const s2 = new Session("JSONEachRow"); + const ok = await s2.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", ['{"a":7,"b":"z"}\n']); + s2.close(); + expect(ok.control.event).toBe("end"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("1"); + }); +}); diff --git a/programs/wsproxy/tests/test/flow.test.mjs b/programs/wsproxy/tests/test/flow.test.mjs new file mode 100644 index 000000000000..8dca94276772 --- /dev/null +++ b/programs/wsproxy/tests/test/flow.test.mjs @@ -0,0 +1,119 @@ +import { describe, it, expect } from "vitest"; +import { Session } from "./helpers.mjs"; +import { RawClient } from "./raw.mjs"; + +// Collect frames until either a terminal control event or a quiet gap (no frame +// within `gapMs`, meaning the server is gated waiting for credit). Returns +// { binary, ended, cancelled }. +async function drainUntilGap(s, gapMs) { + let binary = 0; + let ended = false; + for (;;) { + const f = await s.nextFrame(gapMs); + if (f === null) break; // gated: nothing arrived within the gap + if (f.type === "binary") binary++; + else if (f.type === "text") { + const ev = JSON.parse(f.data).event; + if (ev === "end" || ev === "error" || ev === "cancelled") { + ended = true; + break; + } + // progress / log / profile_events are not gated — ignore + } else if (f.type === "close") { + break; + } + } + return { binary, ended }; +} + +describe("flow control (credit/window)", () => { + it.each(["0", "-1", "not-a-number"])("rejects invalid initial credit %s", async (credit) => { + const c = new RawClient({ port: 9010 }); + const { head } = await c.handshakeResponse(`/?format=JSONEachRow&flow=${credit}`); + expect(head).toMatch(/^HTTP\/1\.1 400 /); + expect(head).not.toContain("101 Switching Protocols"); + c.close(); + }); + + it("closes the session on a nonpositive credit grant", async () => { + const s = new Session("JSONEachRow", { flow: 1 }); + await s.ready(); + s.sendQuery("SELECT number FROM numbers(10) SETTINGS max_block_size = 1"); + expect((await drainUntilGap(s, 600)).binary).toBe(1); + s.next(-1); + const frame = await s.nextFrame(5000); + expect(frame?.type).toBe("close"); + expect(frame?.code).toBe(1008); + s.close(); + }, 10000); + + it("closes the session rather than overflowing accumulated credit", async () => { + const s = new Session("JSONEachRow", { flow: 1 }); + await s.ready(); + s.sendQuery("SELECT number FROM numbers(10) SETTINGS max_block_size = 1"); + s.ws.send('{"cmd":"next","n":9223372036854775807}'); + s.ws.send('{"cmd":"next","n":9223372036854775807}'); + let close = null; + for (;;) { + const frame = await s.nextFrame(5000); + if (frame === null || frame.type === "close") { + close = frame?.type === "close" ? frame : null; + break; + } + } + expect(close?.code).toBe(1008); + s.close(); + }, 10000); + + it("sends only up to the granted credit, then resumes on a next command", async () => { + // One row per block => one binary frame per row; initial credit = 3. + const s = new Session("JSONEachRow", { flow: 3 }); + await s.ready(); + s.sendQuery("SELECT number FROM numbers(20) SETTINGS max_block_size = 1"); + + // Initial window: exactly 3 data frames, then the stream should stall. + const phase1 = await drainUntilGap(s, 600); + expect(phase1.ended).toBe(false); + expect(phase1.binary).toBe(3); + + // Grant enough credit to finish; all remaining rows arrive and we reach end. + s.next(100); + const phase2 = await drainUntilGap(s, 3000); + s.close(); + expect(phase2.ended).toBe(true); + expect(phase1.binary + phase2.binary).toBe(20); + }, 20000); + + it("a pause command halts the stream and a resume command continues it", async () => { + const s = new Session("JSONEachRow", { flow: 2 }); + await s.ready(); + s.sendQuery("SELECT number FROM numbers(10) SETTINGS max_block_size = 1"); + + // Consume the initial 2, then pause before granting more. + const p1 = await drainUntilGap(s, 600); + expect(p1.binary).toBe(2); + s.pause(); + s.next(100); // credit is available, but paused must keep the stream halted + + const p2 = await drainUntilGap(s, 600); + expect(p2.binary).toBe(0); // paused: no frames despite credit + expect(p2.ended).toBe(false); + + // Resume: the remaining rows flow (credit already granted). + s.resume(); + const p3 = await drainUntilGap(s, 3000); + s.close(); + expect(p3.ended).toBe(true); + expect(p1.binary + p2.binary + p3.binary).toBe(10); + }, 20000); + + it("without ?flow, push mode is unbounded (no credit needed)", async () => { + const s = new Session("JSONEachRow"); // no flow param + await s.ready(); + s.sendQuery("SELECT number FROM numbers(50) SETTINGS max_block_size = 1"); + const r = await drainUntilGap(s, 3000); + s.close(); + expect(r.ended).toBe(true); + expect(r.binary).toBe(50); // all delivered without any next command + }, 20000); +}); diff --git a/programs/wsproxy/tests/test/formats.test.mjs b/programs/wsproxy/tests/test/formats.test.mjs new file mode 100644 index 000000000000..754afaf23fed --- /dev/null +++ b/programs/wsproxy/tests/test/formats.test.mjs @@ -0,0 +1,118 @@ +import { describe, it, expect } from "vitest"; +import { runQuery } from "./helpers.mjs"; + +// Converting between ClickHouse output formats is the proxy's whole purpose, +// so exercise a broad spread of them: fixed-byte binary formats, text formats +// with and without headers, structured JSON, and container formats with magic +// bytes (Parquet, Arrow). The row-per-line formats are covered elsewhere. +describe("output formats", () => { + it("RowBinary emits exact little-endian bytes", async () => { + const u32 = await runQuery("SELECT toUInt32(42) AS v", { format: "RowBinary" }); + expect(u32.control.event).toBe("end"); + expect(u32.data.equals(Buffer.from([42, 0, 0, 0]))).toBe(true); + + const two = await runQuery("SELECT toUInt8(1), toUInt8(2)", { format: "RowBinary" }); + expect(two.control.event).toBe("end"); + expect(two.data.equals(Buffer.from([1, 2]))).toBe(true); + }); + + it("TabSeparated / TSV separates columns with tabs", async () => { + const tsv = await runQuery("SELECT 1 AS a, 'x' AS b", { format: "TabSeparated" }); + expect(tsv.control.event).toBe("end"); + expect(tsv.text.trim()).toBe("1\tx"); + + const tsvAlias = await runQuery("SELECT 1 AS a, 'x' AS b", { format: "TSV" }); + expect(tsvAlias.control.event).toBe("end"); + expect(tsvAlias.text.trim()).toBe("1\tx"); + }); + + it("TSVWithNames prepends a header row", async () => { + const { text, control } = await runQuery("SELECT 1 AS a, 'x' AS b", { + format: "TSVWithNames", + }); + expect(control.event).toBe("end"); + const lines = text.split("\n"); + expect(lines[0]).toBe("a\tb"); + expect(lines[1]).toBe("1\tx"); + }); + + it("CSV quotes strings", async () => { + const { text, control } = await runQuery("SELECT 1, 'x'", { format: "CSV" }); + expect(control.event).toBe("end"); + expect(text).toContain('1,"x"'); + }); + + it("CSVWithNames prepends a header row", async () => { + const { text, control } = await runQuery("SELECT 1 AS a, 'x' AS b", { + format: "CSVWithNames", + }); + expect(control.event).toBe("end"); + const lines = text.split("\n").filter((l) => l.length > 0); + expect(lines[0]).toContain('"a"'); + expect(lines[0]).toContain('"b"'); + }); + + it("JSON produces a full object with meta and data", async () => { + const { text, control } = await runQuery("SELECT 1 AS a", { format: "JSON" }); + expect(control.event).toBe("end"); + const obj = JSON.parse(text); + expect(Array.isArray(obj.data)).toBe(true); + expect(Array.isArray(obj.meta)).toBe(true); + expect(obj.data[0].a).toBeDefined(); + }); + + it("JSONCompactEachRow emits an array per row", async () => { + const { text, control } = await runQuery("SELECT 1 AS a, 'x' AS b", { + format: "JSONCompactEachRow", + }); + expect(control.event).toBe("end"); + const line = text.split("\n").find((l) => l.length > 0); + expect(JSON.parse(line)).toEqual([1, "x"]); + }); + + it("Values renders a tuple literal", async () => { + const { text, control } = await runQuery("SELECT 1, 'x'", { format: "Values" }); + expect(control.event).toBe("end"); + expect(text.trim()).toBe("(1,'x')"); + }); + + it("Pretty renders a table (smoke)", async () => { + const { text, control } = await runQuery("SELECT 1 AS a", { format: "Pretty" }); + expect(control.event).toBe("end"); + expect(text.length).toBeGreaterThan(0); + // Box-drawing characters and/or the column name should be present. + expect(text.includes("┏") || text.includes("─") || text.includes("a")).toBe(true); + }); + + it("Native returns non-empty binary (smoke)", async () => { + const { data, control } = await runQuery("SELECT number FROM numbers(3)", { + format: "Native", + }); + expect(control.event).toBe("end"); + expect(data.length).toBeGreaterThan(0); + }); + + it("Parquet is framed by PAR1 magic bytes", async () => { + const { data, control } = await runQuery("SELECT number FROM numbers(10)", { + format: "Parquet", + }); + expect(control.event).toBe("end"); + expect(data.length).toBeGreaterThan(8); + expect(data.subarray(0, 4).toString("latin1")).toBe("PAR1"); + expect(data.subarray(-4).toString("latin1")).toBe("PAR1"); + }); + + it("Arrow starts with the ARROW1 file magic", async () => { + const { data, control } = await runQuery("SELECT number FROM numbers(10)", { + format: "Arrow", + }); + expect(control.event).toBe("end"); + expect(data.length).toBeGreaterThan(0); + expect(data.subarray(0, 6).toString("latin1")).toBe("ARROW1"); + }); + + it("an unknown format surfaces an error cleanly", async () => { + const { control } = await runQuery("SELECT 1", { format: "NoSuchFormat" }); + expect(control.event).toBe("error"); + }); +}); diff --git a/programs/wsproxy/tests/test/globalSetup.mjs b/programs/wsproxy/tests/test/globalSetup.mjs new file mode 100644 index 000000000000..ffa691894ac0 --- /dev/null +++ b/programs/wsproxy/tests/test/globalSetup.mjs @@ -0,0 +1,63 @@ +// Vitest global setup: ensure a ClickHouse backend (:9000) and the proxy (:9010) +// are running, create the shared test table, and tear down whatever we started. +// +// Self-contained: the backend config (including the auth-test user) is generated +// by spawnBackend, so no external config files are needed. Anything already +// listening is reused (so you can run against a stack you started by hand); only +// processes we spawn here are stopped on teardown. + +import net from "node:net"; +import { spawnBackend, spawnProxy } from "./proc.mjs"; + +// Use a high, private backend port so the suite is hermetic and does not +// accidentally reuse an unrelated ClickHouse (or other service) on 9000/9100. +const BACKEND_PORT = 19000; +const PROXY_PORT = 9010; + +function portOpen(port, host = "127.0.0.1") { + return new Promise((resolve) => { + const socket = net.connect({ port, host }, () => { + socket.destroy(); + resolve(true); + }); + socket.on("error", () => resolve(false)); + }); +} + +export default async function setup() { + const stops = []; + + if (!(await portOpen(BACKEND_PORT))) { + const backend = await spawnBackend({ tcpPort: BACKEND_PORT }); + stops.push(() => backend.stop()); + } + + if (!(await portOpen(PROXY_PORT))) { + const proxy = await spawnProxy({ + listenPort: PROXY_PORT, + backendPort: BACKEND_PORT, + allowedOrigins: "http://127.0.0.1, https://trusted.example", + }); + stops.push(() => proxy.stop()); + } + + // Create the shared test table via the proxy (DDL goes through the SELECT path). + const { runQuery } = await import("./helpers.mjs"); + const ddl = await runQuery( + "CREATE TABLE IF NOT EXISTS default.wsp_test (a UInt32, b String) ENGINE = Memory", + ); + if (ddl.control.event !== "end") { + throw new Error(`failed to create test table: ${JSON.stringify(ddl.control)}`); + } + + // Teardown: stop only what we started. + return () => { + for (const stop of stops) { + try { + stop(); + } catch { + /* ignore */ + } + } + }; +} diff --git a/programs/wsproxy/tests/test/helpers.mjs b/programs/wsproxy/tests/test/helpers.mjs new file mode 100644 index 000000000000..9d7d62392290 --- /dev/null +++ b/programs/wsproxy/tests/test/helpers.mjs @@ -0,0 +1,244 @@ +// WebSocket client helpers for the clickhouse-wsproxy integration tests. +// +// Protocol recap (proxy <-> client over one WebSocket): +// - The client sends a SQL query as a TEXT frame. +// - SELECT results stream back as BINARY frames (bytes of the chosen output +// format, set via the `?format=` query parameter of the WS URL). +// - Mid-query progress is pushed as TEXT frames: {"event":"progress",...}. +// - The query terminates with a TEXT control frame: +// {"event":"end"} | {"event":"error","message":...} | {"event":"cancelled"} +// - INSERT: send `INSERT INTO t [FORMAT X]` as a TEXT frame, then stream data +// as BINARY frames, then a zero-length BINARY frame to signal end of data. +// - Cancel a running query by CLOSING the socket. +// +// Uses Node's built-in global WebSocket (Node >= 22), so no dependencies. + +export const PROXY_URL = process.env.WSPROXY_URL ?? "ws://127.0.0.1:9010"; + +/** Build the WS URL for a format and options ({ logs, path, baseUrl, user, password, flow, parse }). */ +export function urlFor( + format = "JSONEachRow", + { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow, parse = false, parallel = false } = {}, +) { + const params = new URLSearchParams({ format }); + if (logs) params.set("logs", logs); + if (user) params.set("user", user); + if (password) params.set("password", password); + if (flow !== undefined) params.set("flow", String(flow)); // opt-in credit flow control + if (parse) params.set("parse", "1"); // opt-in SQL parsing (auto-route inserts, report kind) + if (parallel) params.set("parallel", "1"); // opt-in parallel output formatting (coarser frames) + return `${baseUrl}${path}?${params}`; +} + +/** + * A single proxy session over one WebSocket. Supports multiple sequential + * queries. Incoming frames are buffered into an async queue so callers can + * `await` them regardless of arrival timing. + */ +export class Session { + constructor( + format = "JSONEachRow", + { logs = "", path = "/", baseUrl = PROXY_URL, user = "", password = "", flow, parse = false, parallel = false } = {}, + ) { + this.ws = new WebSocket(urlFor(format, { logs, path, baseUrl, user, password, flow, parse, parallel })); + this.ws.binaryType = "arraybuffer"; + this._queue = []; + this._waiters = []; + this._closed = false; + + const push = (frame) => { + const waiter = this._waiters.shift(); + if (waiter) waiter(frame); + else this._queue.push(frame); + }; + + this.ws.addEventListener("message", (ev) => { + if (typeof ev.data === "string") push({ type: "text", data: ev.data }); + else push({ type: "binary", data: Buffer.from(ev.data) }); + }); + this.ws.addEventListener("close", (ev) => { + this._closed = true; + push({ type: "close", code: ev.code, reason: ev.reason }); + }); + + this.opened = new Promise((resolve, reject) => { + this.ws.addEventListener("open", () => resolve()); + this.ws.addEventListener("error", (ev) => + reject(new Error(`WebSocket error: ${ev?.message ?? ev}`)), + ); + }); + } + + /** Resolve once the socket is open. */ + ready() { + return this.opened; + } + + /** + * Next incoming frame: {type:'text',data:string} | {type:'binary',data:Buffer} | + * {type:'close',code:number,reason:string}. + * With `timeoutMs > 0`, resolves to null if no frame arrives in time (without + * losing a frame that arrives later — the waiter is removed on timeout). + */ + nextFrame(timeoutMs = 0) { + if (this._queue.length) return Promise.resolve(this._queue.shift()); + return new Promise((resolve) => { + let settled = false; + const waiter = (frame) => { + if (settled) { + this._queue.unshift(frame); // raced with the timeout; don't drop it + return; + } + settled = true; + resolve(frame); + }; + this._waiters.push(waiter); + if (timeoutMs > 0) { + setTimeout(() => { + if (settled) return; + settled = true; + const i = this._waiters.indexOf(waiter); + if (i >= 0) this._waiters.splice(i, 1); + resolve(null); + }, timeoutMs); + } + }); + } + + /** Flow-control: grant N more frames of credit. */ + next(n) { + this.ws.send(JSON.stringify({ cmd: "next", n })); + } + + /** Flow-control: pause the server push. */ + pause() { + this.ws.send(JSON.stringify({ cmd: "pause" })); + } + + /** Flow-control: resume after pause(). */ + resume() { + this.ws.send(JSON.stringify({ cmd: "resume" })); + } + + /** Send a SQL query (TEXT frame). */ + sendQuery(sql) { + this.ws.send(sql); + } + + /** Send INSERT data (BINARY frame). Accepts a string or a Buffer/Uint8Array. */ + sendData(chunk) { + this.ws.send(typeof chunk === "string" ? Buffer.from(chunk) : chunk); + } + + /** Signal end of INSERT data (zero-length BINARY frame). */ + endData() { + this.ws.send(new Uint8Array(0)); + } + + /** + * Consume frames until the terminal control event. + * Returns { data: Buffer, text: string, progress: object[], control: object }. + * Binary frames are concatenated into `data`; progress events collected. + */ + async collect() { + const chunks = []; + const progress = []; + const logs = []; + const profileEvents = []; + let queryInfo = null; // {"event":"query",...} sent in ?parse=1 mode + const done = (control) => { + const data = Buffer.concat(chunks); + return { data, text: data.toString("utf8"), progress, logs, profileEvents, queryInfo, control }; + }; + for (;;) { + const frame = await this.nextFrame(); + if (frame.type === "binary") { + chunks.push(frame.data); + } else if (frame.type === "text") { + const msg = JSON.parse(frame.data); + // Terminal events end the query; everything else is a mid-query push. + if (msg.event === "end" || msg.event === "error" || msg.event === "cancelled") { + return done(msg); + } + if (msg.event === "progress") progress.push(msg); + else if (msg.event === "log") logs.push(msg); + else if (msg.event === "profile_events") profileEvents.push(msg); + else if (msg.event === "query") queryInfo = msg; + // Unknown non-terminal events are ignored. + } else { + // Socket closed without a terminal control frame. + return done({ event: "closed" }); + } + } + } + + /** Send a query and collect its full result. */ + async run(sql) { + await this.ready(); + this.sendQuery(sql); + return this.collect(); + } + + /** + * Begin a streamed INSERT: declare it with a control message so the proxy + * opts into the data phase (the proxy never parses SQL). `format` is the input + * format of the data you'll stream (defaults to the session's format). + */ + beginInsert(query, { format } = {}) { + const cmd = { cmd: "insert", query }; + if (format) cmd.format = format; + this.ws.send(JSON.stringify(cmd)); + } + + /** + * Run a streamed INSERT: declare it, stream the data chunks, end, and collect + * the terminal control event. `chunks` is an array of strings/Buffers. + */ + async insert(query, chunks = [], opts = {}) { + await this.ready(); + this.beginInsert(query, opts); + for (const c of chunks) this.sendData(c); + this.endData(); + return this.collect(); + } + + /** Close the socket (also used to cancel a running query). */ + close() { + try { + this.ws.close(); + } catch { + /* already closing */ + } + } +} + +/** One-shot: open a session, run a query, close, return the result object. */ +export async function runQuery( + sql, + { format = "JSONEachRow", logs = "", baseUrl = PROXY_URL, user = "", password = "" } = {}, +) { + const s = new Session(format, { logs, baseUrl, user, password }); + try { + return await s.run(sql); + } finally { + s.close(); + } +} + +/** + * Query the backend through the proxy and return the single scalar/text result, + * trimmed. Handy for verifying server-side state (uses TSV so there is no JSON + * quoting to strip). + */ +export async function backendScalar(sql) { + const { text, control } = await runQuery(sql, { format: "TSV" }); + if (control.event !== "end") { + throw new Error(`backend query failed: ${control.event} ${control.message ?? ""}`); + } + return text.trim(); +} + +/** Sleep helper. */ +export function sleep(ms) { + return new Promise((r) => setTimeout(r, ms)); +} diff --git a/programs/wsproxy/tests/test/insert-flow.test.mjs b/programs/wsproxy/tests/test/insert-flow.test.mjs new file mode 100644 index 000000000000..72e2ec4f54f1 --- /dev/null +++ b/programs/wsproxy/tests/test/insert-flow.test.mjs @@ -0,0 +1,40 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { runQuery, Session, backendScalar, sleep } from "./helpers.mjs"; + +describe("insert flow", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it("aborts an INSERT (no commit) when the client closes mid-stream", async () => { + const s = new Session("JSONEachRow"); + await s.ready(); + s.beginInsert("INSERT INTO default.wsp_test FORMAT JSONEachRow"); + // Send some data but NEVER send the end marker; close the socket instead. + s.sendData('{"a":1,"b":"x"}\n{"a":2,"b":"y"}\n'); + await sleep(200); + s.close(); // Close mid-stream -> proxy calls sendCancel, backend rolls back. + + // Give the backend a moment to process the cancel, then verify nothing committed. + await sleep(500); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("0"); + }, 20000); + + it("commits a normal INSERT after a mid-stream abort (proxy stays healthy)", async () => { + // Abort one insert... + const a = new Session("JSONEachRow"); + await a.ready(); + a.beginInsert("INSERT INTO default.wsp_test FORMAT JSONEachRow"); + a.sendData('{"a":9,"b":"q"}\n'); + await sleep(150); + a.close(); + await sleep(300); + + // ...then a clean insert still works. + const b = new Session("JSONEachRow"); + const ok = await b.insert("INSERT INTO default.wsp_test FORMAT JSONEachRow", ['{"a":1,"b":"x"}\n']); + b.close(); + expect(ok.control.event).toBe("end"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("1"); + }, 20000); +}); diff --git a/programs/wsproxy/tests/test/insert.test.mjs b/programs/wsproxy/tests/test/insert.test.mjs new file mode 100644 index 000000000000..f28ff537b6ba --- /dev/null +++ b/programs/wsproxy/tests/test/insert.test.mjs @@ -0,0 +1,104 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { Session, runQuery, backendScalar } from "./helpers.mjs"; + +describe("INSERT", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + // These INSERTs carry their own data (SELECT source / inline VALUES), so they + // run as plain queries — no streamed data phase, and the proxy never parses SQL. + it("runs INSERT ... SELECT as a plain query (no data phase)", async () => { + const r = await runQuery("INSERT INTO default.wsp_test SELECT number, 'z' FROM numbers(4)"); + expect(r.control.event).toBe("end"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("4"); + }); + + it("runs inline INSERT ... VALUES as a plain query", async () => { + const r = await runQuery("INSERT INTO default.wsp_test VALUES (1, 'a'), (2, 'b')"); + expect(r.control.event).toBe("end"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + }); + + it("streams a JSONEachRow insert across multiple chunks", async () => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + "INSERT INTO default.wsp_test FORMAT JSONEachRow", + ['{"a":1,"b":"x"}\n{"a":2,"b":"y"}\n', '{"a":3,"b":"z"}\n'], + ); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("3"); + expect(await backendScalar("SELECT sum(a) FROM default.wsp_test")).toBe("6"); + }); + + it("handles an inline VALUES insert with an inferred format", async () => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + "INSERT INTO default.wsp_test VALUES (10,'v'),(11,'w')", + [], + ); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + expect(await backendScalar("SELECT sum(a) FROM default.wsp_test")).toBe("21"); + }); + + it("streams an insert in a different input format (TSV)", async () => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + "INSERT INTO default.wsp_test FORMAT TSV", + ["5\tp\n6\tq\n"], + { format: "TSV" }, // input format for the streamed data + ); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + }); + + it("reports an error when inserting into a missing table", async () => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + "INSERT INTO default.no_such_table_zz FORMAT JSONEachRow", + ['{"a":1,"b":"x"}\n'], + ); + expect(control.event).toBe("error"); + expect(control.message).toBeTruthy(); + expect(control.message).toMatch(/table/i); + } finally { + s.close(); + } + }); + + it("keeps the connection healthy for a SELECT after inserts", async () => { + const s = new Session("JSONEachRow"); + try { + const insertResult = await s.insert( + "INSERT INTO default.wsp_test FORMAT JSONEachRow", + ['{"a":1,"b":"x"}\n{"a":2,"b":"y"}\n'], + ); + expect(insertResult.control.event).toBe("end"); + } finally { + s.close(); + } + + const { text, control } = await runQuery( + "SELECT count() FROM default.wsp_test", + ); + expect(control.event).toBe("end"); + expect(text).toContain("2"); + }); +}); diff --git a/programs/wsproxy/tests/test/large-insert.test.mjs b/programs/wsproxy/tests/test/large-insert.test.mjs new file mode 100644 index 000000000000..66f6b0fcc64b --- /dev/null +++ b/programs/wsproxy/tests/test/large-insert.test.mjs @@ -0,0 +1,110 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { Session, runQuery, backendScalar } from "./helpers.mjs"; + +describe("large INSERT", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it( + "streams a JSONEachRow insert of 500,000 rows across multiple frames", + async () => { + const total = 500000; + const rowsPerChunk = 50000; + const chunks = []; + for (let start = 0; start < total; start += rowsPerChunk) { + const rows = []; + for (let i = start; i < start + rowsPerChunk; i++) { + rows.push(`{"a":${i},"b":"r${i}"}\n`); + } + chunks.push(rows.join("")); + } + expect(chunks.length).toBe(10); + + const s = new Session("JSONEachRow"); + let control; + try { + ({ control } = await s.insert( + "INSERT INTO default.wsp_test FORMAT JSONEachRow", + chunks, + )); + } finally { + s.close(); + } + expect(control.event).toBe("end"); + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe( + "500000", + ); + // Exact sum of 0..499999, computed with BigInt to avoid float precision issues. + const expectedSum = ((499999n * 500000n) / 2n).toString(); + expect(expectedSum).toBe("124999750000"); + expect(await backendScalar("SELECT sum(a) FROM default.wsp_test")).toBe( + expectedSum, + ); + }, + 60000, + ); + + it( + "handles a large INSERT in a single big frame", + async () => { + const total = 100000; + const rows = []; + for (let i = 0; i < total; i++) { + rows.push(`{"a":${i},"b":"r${i}"}\n`); + } + const chunk = rows.join(""); + + const s = new Session("JSONEachRow"); + let control; + try { + ({ control } = await s.insert( + "INSERT INTO default.wsp_test FORMAT JSONEachRow", + [chunk], + )); + } finally { + s.close(); + } + expect(control.event).toBe("end"); + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe( + "100000", + ); + }, + 60000, + ); + + it( + "round-trips integrity via a different input format (TSV)", + async () => { + const total = 100000; + const rows = []; + for (let i = 0; i < total; i++) { + rows.push(`${i}\tr${i}\n`); + } + const chunk = rows.join(""); + + const s = new Session("JSONEachRow"); + let control; + try { + ({ control } = await s.insert( + "INSERT INTO default.wsp_test FORMAT TSV", + [chunk], + { format: "TSV" }, // input format for the streamed data + )); + } finally { + s.close(); + } + expect(control.event).toBe("end"); + + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe( + "100000", + ); + expect(await backendScalar("SELECT max(a) FROM default.wsp_test")).toBe( + "99999", + ); + }, + 60000, + ); +}); diff --git a/programs/wsproxy/tests/test/logs.test.mjs b/programs/wsproxy/tests/test/logs.test.mjs new file mode 100644 index 000000000000..6a1a8fe6ce61 --- /dev/null +++ b/programs/wsproxy/tests/test/logs.test.mjs @@ -0,0 +1,39 @@ +import { describe, it, expect } from "vitest"; +import { runQuery } from "./helpers.mjs"; + +describe("logs and profile events", () => { + it("pushes server log lines when the client opts in via ?logs=", async () => { + const { logs, control } = await runQuery( + "SELECT count() FROM numbers(1000000)", + { logs: "trace" }, + ); + expect(control.event).toBe("end"); + expect(logs.length).toBeGreaterThanOrEqual(1); + + // Each log event carries an array of row objects with a `text` column. + const rows = logs.flatMap((e) => e.rows); + expect(rows.length).toBeGreaterThanOrEqual(1); + expect(typeof rows[0].text).toBe("string"); + expect(rows[0].text.length).toBeGreaterThan(0); + }, 20000); + + it("does not push logs at the default level", async () => { + const { logs, control } = await runQuery("SELECT 1"); + expect(control.event).toBe("end"); + expect(logs.length).toBe(0); + }); + + it("pushes profile events during a query", async () => { + // A slow query gives the server time to emit ProfileEvents packets. + const { profileEvents, control } = await runQuery( + "SELECT sleepEachRow(0.1), number FROM numbers(30) SETTINGS max_block_size = 1", + ); + expect(control.event).toBe("end"); + expect(profileEvents.length).toBeGreaterThanOrEqual(1); + + const rows = profileEvents.flatMap((e) => e.rows); + expect(rows.length).toBeGreaterThanOrEqual(1); + // ProfileEvents rows have `name` and `value` columns. + expect(rows.every((r) => typeof r.name === "string")).toBe(true); + }, 20000); +}); diff --git a/programs/wsproxy/tests/test/parallel.test.mjs b/programs/wsproxy/tests/test/parallel.test.mjs new file mode 100644 index 000000000000..b10bc4fe4a98 --- /dev/null +++ b/programs/wsproxy/tests/test/parallel.test.mjs @@ -0,0 +1,51 @@ +import { describe, it, expect } from "vitest"; +import { Session } from "./helpers.mjs"; + +// Opt-in parallel output formatting (`?parallel=1`). Formats SELECT output on a +// thread pool for throughput; result frames are coarser (blocks are batched) but +// the delivered bytes must be identical to the default single-threaded path. +describe("parallel output formatting (?parallel=1)", () => { + const QUERY = "SELECT number AS n, toString(number) AS s FROM numbers(200000)"; + + async function fetchText(opts) { + const s = new Session("JSONCompactEachRow", opts); + try { + const { text, control } = await s.run(QUERY); + expect(control.event).toBe("end"); + return text; + } finally { + s.close(); + } + } + + it("produces byte-identical output to the default path", async () => { + const [plain, parallel] = await Promise.all([fetchText({}), fetchText({ parallel: true })]); + expect(parallel.length).toBe(plain.length); + expect(parallel).toBe(plain); + }); + + it("delivers all rows", async () => { + const s = new Session("JSONCompactEachRow", { parallel: true }); + try { + const { text, control } = await s.run(QUERY); + expect(control.event).toBe("end"); + const lines = text.split("\n").filter((l) => l.length > 0); + expect(lines.length).toBe(200000); + expect(JSON.parse(lines[0])).toEqual([0, "0"]); + expect(JSON.parse(lines[199999])).toEqual([199999, "199999"]); + } finally { + s.close(); + } + }); + + it("propagates a backend exception as an error event", async () => { + const s = new Session("JSONCompactEachRow", { parallel: true }); + try { + const { control } = await s.run("SELECT throwIf(number = 5) FROM numbers(100)"); + expect(control.event).toBe("error"); + expect(control.message).toBeTruthy(); + } finally { + s.close(); + } + }); +}); diff --git a/programs/wsproxy/tests/test/parse.test.mjs b/programs/wsproxy/tests/test/parse.test.mjs new file mode 100644 index 000000000000..03affbe69033 --- /dev/null +++ b/programs/wsproxy/tests/test/parse.test.mjs @@ -0,0 +1,134 @@ +import { describe, it, expect, beforeEach } from "vitest"; +import { Session, runQuery, backendScalar } from "./helpers.mjs"; + +// Opt-in SQL parsing (`?parse=1`). The proxy never parses SQL by default; with +// this flag it parses each query to (a) report the leading verb + routing +// decision as a {"event":"query",...} frame and (b) auto-route a streamed-data +// INSERT without the client sending an explicit {"cmd":"insert",...} message. +describe("parse mode (?parse=1)", () => { + beforeEach(async () => { + await runQuery("TRUNCATE TABLE default.wsp_test"); + }); + + it("reports a SELECT as a plain query and streams results", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, text, control } = await s.run("SELECT 1 AS x"); + expect(control.event).toBe("end"); + expect(queryInfo).toEqual({ event: "query", kind: "query", verb: "SELECT" }); + expect(JSON.parse(text.trim())).toEqual({ x: 1 }); + } finally { + s.close(); + } + }); + + it("reports INSERT ... SELECT as a plain query and inserts server-side", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, control } = await s.run( + "INSERT INTO default.wsp_test SELECT number, 'z' FROM numbers(4)", + ); + expect(control.event).toBe("end"); + expect(queryInfo.kind).toBe("query"); // self-contained: no client data phase + expect(queryInfo.verb).toBe("INSERT"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("4"); + } finally { + s.close(); + } + }); + + it("reports inline INSERT ... VALUES as a plain query", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, control } = await s.run( + "INSERT INTO default.wsp_test VALUES (1, 'a'), (2, 'b')", + ); + expect(control.event).toBe("end"); + expect(queryInfo.kind).toBe("query"); // inline data: no client data phase + expect(queryInfo.verb).toBe("INSERT"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + } finally { + s.close(); + } + }); + + it("auto-routes a streamed INSERT (no control message needed)", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + await s.ready(); + // Send the INSERT as a plain text frame, then stream the rows. The proxy + // parses it, decides it needs client data, and reads the binary frames. + s.sendQuery("INSERT INTO default.wsp_test FORMAT JSONEachRow"); + s.sendData('{"a":10,"b":"p"}\n{"a":11,"b":"q"}\n'); + s.endData(); + const { queryInfo, control } = await s.collect(); + expect(control.event).toBe("end"); + expect(queryInfo).toEqual({ event: "query", kind: "insert", verb: "INSERT" }); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + } finally { + s.close(); + } + }); + + it("picks up the input format from the FORMAT clause for a streamed INSERT", async () => { + // Session output format is JSONEachRow, but the INSERT declares TSV — the + // proxy should feed the streamed data through TSV, not the session default. + const s = new Session("JSONEachRow", { parse: true }); + try { + await s.ready(); + s.sendQuery("INSERT INTO default.wsp_test FORMAT TSV"); + s.sendData("7\tp\n8\tq\n"); + s.endData(); + const { queryInfo, control } = await s.collect(); + expect(control.event).toBe("end"); + expect(queryInfo.kind).toBe("insert"); + expect(await backendScalar("SELECT count() FROM default.wsp_test")).toBe("2"); + } finally { + s.close(); + } + }); + + it("reports DDL with its leading verb", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, control } = await s.run( + "CREATE TABLE IF NOT EXISTS default.wsp_parse_ddl (x UInt8) ENGINE = Memory", + ); + expect(control.event).toBe("end"); + expect(queryInfo).toEqual({ event: "query", kind: "query", verb: "CREATE" }); + } finally { + s.close(); + await runQuery("DROP TABLE IF EXISTS default.wsp_parse_ddl"); + } + }); + + it("recovers the leading verb through leading comments", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, control } = await s.run("-- a leading comment\nSELECT 42 AS x"); + expect(control.event).toBe("end"); + expect(queryInfo.verb).toBe("SELECT"); + } finally { + s.close(); + } + }); + + it("fails closed when opted-in SQL classification cannot parse the query", async () => { + const s = new Session("JSONEachRow", { parse: true }); + try { + const { queryInfo, control } = await s.run("SELECT this is not valid sql ((("); + expect(control.event).toBe("error"); + expect(queryInfo).toBeNull(); + expect(control.message).toMatch(/syntax|parse|expected/i); + } finally { + s.close(); + } + }); + + it("does not send a query frame when parse mode is off", async () => { + // Default session: no ?parse=1, so no {"event":"query"} frame at all. + const { queryInfo, control } = await runQuery("SELECT 1 AS x"); + expect(control.event).toBe("end"); + expect(queryInfo).toBeNull(); + }); +}); diff --git a/programs/wsproxy/tests/test/proc.mjs b/programs/wsproxy/tests/test/proc.mjs new file mode 100644 index 000000000000..20ddcc4355ec --- /dev/null +++ b/programs/wsproxy/tests/test/proc.mjs @@ -0,0 +1,247 @@ +// Helper to spawn an extra proxy instance with a custom listen port and backend, +// used by resilience tests (e.g. pointing the proxy at a dead backend). + +import { spawn, spawnSync } from "node:child_process"; +import fs from "node:fs"; +import net from "node:net"; +import os from "node:os"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const testDir = dirname(fileURLToPath(import.meta.url)); +const repoRoot = resolve(testDir, "../../../.."); +const WSPROXY_BIN = + process.env.WSPROXY_BIN ?? resolve(repoRoot, "build/programs/wsproxy/clickhouse-wsproxy"); +const CLICKHOUSE_SERVER = process.env.CLICKHOUSE_SERVER ?? "/usr/local/bin/clickhouse-server"; +// The freshly-built multi-call binary (run as `clickhouse server`) — this is the one that +// carries the new ws_port. Point WS_SERVER_BIN elsewhere to test a different build. +const WS_SERVER_BIN = process.env.WS_SERVER_BIN ?? resolve(repoRoot, "build/programs/clickhouse"); + +function portOpen(port, host = "127.0.0.1") { + return new Promise((res) => { + const socket = net.connect({ port, host }, () => { + socket.destroy(); + res(true); + }); + socket.on("error", () => res(false)); + }); +} + +async function waitForPort(port, timeoutMs = 15_000) { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (await portOpen(port)) return; + await new Promise((r) => setTimeout(r, 100)); + } + throw new Error(`Timed out waiting for proxy port ${port}`); +} + +/** + * Spawn a disposable ClickHouse backend on `tcpPort` with a fresh temp data + * dir, so a test can kill it (e.g. backend-drops-mid-query). Returns + * { tcpPort, stop } where stop() kills the process and removes the temp dir. + */ +export async function spawnBackend({ tcpPort, securePort }) { + const dir = fs.mkdtempSync(join(os.tmpdir(), "wsproxy-ch-")); + const configPath = join(dir, "config.xml"); + + // Optional TLS: generate a self-signed cert and enable the secure native port. + let secureBlock = ""; + if (securePort) { + const crt = join(dir, "server.crt"); + const key = join(dir, "server.key"); + const gen = spawnSync("openssl", [ + "req", "-subj", "/CN=localhost", "-new", "-newkey", "rsa:2048", + "-days", "3650", "-nodes", "-x509", "-keyout", key, "-out", crt, + ]); + if (gen.status !== 0) throw new Error(`openssl cert generation failed: ${gen.stderr}`); + secureBlock = ` + ${securePort} + + ${crt} + ${key} + none + true + sslv2,sslv3 + true + `; + } + + fs.writeFileSync( + configPath, + ` + none0 + ${dir}/server.log${dir}/server.err.log + ${tcpPort}${secureBlock} + 127.0.0.1 + ${dir}/data/ + ${dir}/tmp/ + ${dir}/user_files/ + 536870912 + false + users.xml + default + default +`, + ); + fs.writeFileSync( + join(dir, "users.xml"), + ` + + + + ::/0 + defaultdefault + + + + wsp_pass::/0 + defaultdefault + + + +`, + ); + + // clickhouse-server forks a watchdog whose child re-parents into its own + // session, so signalling the spawned pid (or its group) leaves the real + // server running. Kill by matching the unique temp config path in argv, which + // reliably hits the actual server process (and the watchdog). + const proc = spawn(CLICKHOUSE_SERVER, ["--config-file", configPath], { stdio: "ignore" }); + await waitForPort(tcpPort, 30_000); + if (securePort) await waitForPort(securePort, 30_000); + return { + tcpPort, + securePort, + stop() { + // Kill whatever holds the TCP port — reliably the real server process, + // regardless of the watchdog fork/rename/re-parent shenanigans. + const r = spawnSync("lsof", ["-ti", `tcp:${tcpPort}`, "-sTCP:LISTEN"], { encoding: "utf8" }); + for (const pid of (r.stdout || "").split("\n").map((s) => s.trim()).filter(Boolean)) { + try { + process.kill(Number(pid), "SIGKILL"); + } catch { + /* already gone */ + } + } + spawnSync("pkill", ["-9", "-f", configPath]); + try { + proc.kill("SIGKILL"); + } catch { + /* already gone */ + } + try { + fs.rmSync(dir, { recursive: true, force: true }); + } catch { + /* ignore */ + } + }, + }; +} + +/** + * Spawn a proxy on `listenPort` pointing at `backendHost:backendPort`. + * Returns { url, stop } where stop() kills the process. + */ +export async function spawnProxy({ + listenPort, + backendHost = "127.0.0.1", + backendPort, + secure = false, + acceptInvalidCert = false, + sendTimeoutSec, + allowedOrigins, + compression, // native codec for backend->proxy blocks: "lz4" | "zstd" | "none" +}) { + const proc = spawn(WSPROXY_BIN, [], { + stdio: "ignore", + env: { + ...process.env, + WSPROXY_PORT: String(listenPort), + WSPROXY_BACKEND_HOST: backendHost, + WSPROXY_BACKEND_PORT: String(backendPort), + ...(secure ? { WSPROXY_BACKEND_SECURE: "1" } : {}), + ...(acceptInvalidCert ? { WSPROXY_BACKEND_ACCEPT_INVALID_CERT: "1" } : {}), + ...(sendTimeoutSec ? { WSPROXY_CLIENT_SEND_TIMEOUT_SEC: String(sendTimeoutSec) } : {}), + ...(allowedOrigins ? { WSPROXY_ALLOWED_ORIGINS: allowedOrigins } : {}), + ...(compression ? { WSPROXY_BACKEND_COMPRESSION: compression } : {}), + }, + }); + await waitForPort(listenPort); + return { + url: `ws://127.0.0.1:${listenPort}`, + stop() { + try { + proc.kill("SIGKILL"); + } catch { + /* already gone */ + } + }, + }; +} + +/// Spawn the freshly-built clickhouse-server with the in-server WebSocket endpoint +/// (`ws_port`) enabled, alongside a `tcp_port` (the server requires at least one +/// query port to start). Same default + wsp_user users as spawnBackend, so the +/// auth tests work unchanged. Returns { url } pointing at the ws_port. +export async function spawnServerWithWs({ wsPort, tcpPort, allowedOrigins }) { + const dir = fs.mkdtempSync(join(os.tmpdir(), "wsproxy-server-")); + const configPath = join(dir, "config.xml"); + fs.writeFileSync( + configPath, + ` + warning0 + ${dir}/server.log${dir}/server.err.log + ${tcpPort} + ${wsPort} + ${allowedOrigins ? `${allowedOrigins}` : ""} + 127.0.0.1 + ${dir}/data/ + ${dir}/tmp/ + ${dir}/user_files/ + 536870912 + false + users.xml + default + default +`, + ); + fs.writeFileSync( + join(dir, "users.xml"), + ` + + + + ::/0 + defaultdefault + 1 + + + wsp_pass::/0 + defaultdefault + + + +`, + ); + + const proc = spawn(WS_SERVER_BIN, ["server", "--config-file", configPath], { stdio: "ignore" }); + await waitForPort(wsPort, 30_000); + return { + url: `ws://127.0.0.1:${wsPort}`, + dir, + stop() { + // Kill whatever holds the ws port (watchdog fork/re-parent shenanigans). + for (const port of [wsPort, tcpPort]) { + const r = spawnSync("lsof", ["-ti", `tcp:${port}`, "-sTCP:LISTEN"], { encoding: "utf8" }); + for (const pid of (r.stdout || "").split("\n").map((s) => s.trim()).filter(Boolean)) { + try { + process.kill(Number(pid), "SIGKILL"); + } catch { + /* already gone */ + } + } + } + }, + }; +} diff --git a/programs/wsproxy/tests/test/progress.test.mjs b/programs/wsproxy/tests/test/progress.test.mjs new file mode 100644 index 000000000000..f78e2848846f --- /dev/null +++ b/programs/wsproxy/tests/test/progress.test.mjs @@ -0,0 +1,46 @@ +import { describe, it, expect } from "vitest"; +import { runQuery } from "./helpers.mjs"; + +/** True if the numbers are in non-decreasing order (compare against a sorted copy). */ +function isMonotonicNonDecreasing(values) { + const sorted = [...values].sort((a, b) => a - b); + return values.every((v, i) => v === sorted[i]); +} + +describe("progress", () => { + it( + "pushes progress events during a slow query", + async () => { + // ~3s query producing many small blocks so the server emits several + // progress updates while it runs. + const sql = + "SELECT sleepEachRow(0.1), number FROM numbers(30) SETTINGS max_block_size = 1"; + const { progress, control } = await runQuery(sql); + + expect(control.event).toBe("end"); + + // Multiple mid-query pushes, not just a single terminal one. + expect(progress.length).toBeGreaterThanOrEqual(2); + + for (const ev of progress) { + expect(typeof ev.read_rows).toBe("number"); + expect(typeof ev.read_bytes).toBe("number"); + expect(typeof ev.total_rows_to_read).toBe("number"); + } + + // Running totals never go backwards. + const readRows = progress.map((ev) => ev.read_rows); + expect(isMonotonicNonDecreasing(readRows)).toBe(true); + + const last = progress[progress.length - 1]; + expect(last.read_rows).toBe(30); + expect(last.total_rows_to_read).toBe(30); + }, + 20000, + ); + + it("still ends cleanly for an instant query", async () => { + const { control } = await runQuery("SELECT 1"); + expect(control.event).toBe("end"); + }); +}); diff --git a/programs/wsproxy/tests/test/raw.mjs b/programs/wsproxy/tests/test/raw.mjs new file mode 100644 index 000000000000..b7a6d2c98693 --- /dev/null +++ b/programs/wsproxy/tests/test/raw.mjs @@ -0,0 +1,189 @@ +// A raw-TCP WebSocket client for adversarial tests. Unlike the native global +// WebSocket, this lets us craft malformed frames (unmasked, reserved opcodes, +// oversized advertised lengths, bad fragmentation) to exercise the proxy's RFC +// 6455 hardening. + +import net from "node:net"; +import crypto from "node:crypto"; + +const GUID = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; + +export class RawClient { + constructor({ host = "127.0.0.1", port = 9010 } = {}) { + this.socket = net.connect({ host, port }); + this.socket.setNoDelay(true); + this.buf = Buffer.alloc(0); + this.closed = false; + this._wake = null; + this.socket.on("data", (chunk) => { + this.buf = Buffer.concat([this.buf, chunk]); + this._signal(); + }); + this.socket.on("close", () => { + this.closed = true; + this._signal(); + }); + this.socket.on("error", () => { + this.closed = true; + this._signal(); + }); + } + + _signal() { + if (this._wake) { + const w = this._wake; + this._wake = null; + w(); + } + } + + _waitData() { + return new Promise((resolve) => (this._wake = resolve)); + } + + connected() { + return new Promise((resolve, reject) => { + this.socket.once("connect", resolve); + this.socket.once("error", reject); + }); + } + + /** Send the WebSocket upgrade request and return its HTTP response headers. + * Raw clients identify as same-origin by default; pass `Origin: undefined` + * to exercise a non-browser client without an Origin header. */ + async handshakeResponse(path = "/?format=JSONEachRow", headers = {}) { + await this.connected(); + const key = crypto.randomBytes(16).toString("base64"); + const requestHeaders = { Origin: "http://127.0.0.1", ...headers }; + const extra = Object.entries(requestHeaders) + .filter(([, value]) => value !== undefined) + .map(([k, v]) => `${k}: ${v}\r\n`) + .join(""); + this.socket.write( + `GET ${path} HTTP/1.1\r\nHost: 127.0.0.1\r\n` + + "Upgrade: websocket\r\nConnection: Upgrade\r\n" + + extra + + `Sec-WebSocket-Key: ${key}\r\nSec-WebSocket-Version: 13\r\n\r\n`, + ); + while (this.buf.indexOf("\r\n\r\n") === -1 && !this.closed) await this._waitData(); + const idx = this.buf.indexOf("\r\n\r\n"); + if (idx === -1) throw new Error("connection closed during handshake"); + const head = this.buf.subarray(0, idx).toString("latin1"); + this.buf = this.buf.subarray(idx + 4); + return { head, key }; + } + + /** Perform the WebSocket upgrade handshake; resolves when the 101 is received. */ + async handshake(path = "/?format=JSONEachRow", headers = {}) { + const { head, key } = await this.handshakeResponse(path, headers); + if (!head.includes("101 Switching Protocols")) throw new Error(`no 101:\n${head}`); + const expect = crypto.createHash("sha1").update(key + GUID).digest("base64"); + if (!head.includes(`Sec-WebSocket-Accept: ${expect}`)) throw new Error("bad accept"); + return head; + } + + /** + * Send a WebSocket frame with full control over the header. + * opts: { opcode, payload (Buffer|string), fin=true, masked=true, rsv=0, + * advertisedLen (override the length field to lie about payload size), + * lengthEncoding (force 126 or 127 for non-minimal encoding tests) } + */ + sendFrame({ + opcode, + payload = Buffer.alloc(0), + fin = true, + masked = true, + rsv = 0, + advertisedLen, + lengthEncoding, + }) { + if (typeof payload === "string") payload = Buffer.from(payload); + const len = advertisedLen ?? payload.length; + const header = []; + header.push((fin ? 0x80 : 0) | (rsv << 4) | (opcode & 0x0f)); + let ext = Buffer.alloc(0); + const maskBit = masked ? 0x80 : 0; + if (lengthEncoding === 126) { + header.push(maskBit | 126); + ext = Buffer.alloc(2); + ext.writeUInt16BE(len); + } else if (lengthEncoding === 127) { + header.push(maskBit | 127); + ext = Buffer.alloc(8); + ext.writeBigUInt64BE(BigInt(len)); + } else if (len < 126) { + header.push(maskBit | len); + } else if (len < 65536) { + header.push(maskBit | 126); + ext = Buffer.alloc(2); + ext.writeUInt16BE(len); + } else { + header.push(maskBit | 127); + ext = Buffer.alloc(8); + ext.writeBigUInt64BE(BigInt(len)); + } + let body = payload; + let maskKey = Buffer.alloc(0); + if (masked) { + maskKey = crypto.randomBytes(4); + body = Buffer.from(payload); + for (let i = 0; i < body.length; i++) body[i] ^= maskKey[i % 4]; + } + this.socket.write(Buffer.concat([Buffer.from(header), ext, maskKey, body])); + } + + /** Read one server->client frame, or {closed:true} if the socket closes first. */ + async readFrame() { + const need = async (n) => { + while (this.buf.length < n && !this.closed) await this._waitData(); + return this.buf.length >= n; + }; + if (!(await need(2))) return { closed: true }; + const b0 = this.buf[0]; + const b1 = this.buf[1]; + const opcode = b0 & 0x0f; + const fin = (b0 & 0x80) !== 0; + let len = b1 & 0x7f; + let offset = 2; + if (len === 126) { + if (!(await need(4))) return { closed: true }; + len = this.buf.readUInt16BE(2); + offset = 4; + } else if (len === 127) { + if (!(await need(10))) return { closed: true }; + len = Number(this.buf.readBigUInt64BE(2)); + offset = 10; + } + if (!(await need(offset + len))) return { closed: true }; + const payload = this.buf.subarray(offset, offset + len); + this.buf = this.buf.subarray(offset + len); + return { opcode, fin, payload: Buffer.from(payload) }; + } + + /** Resolve once the socket has closed (with a timeout guard). */ + async waitClose(timeoutMs = 5000) { + const deadline = Date.now() + timeoutMs; + while (!this.closed && Date.now() < deadline) { + await Promise.race([this._waitData(), new Promise((r) => setTimeout(r, 200))]); + } + return this.closed; + } + + /** Stop reading from the socket (simulates a client that can't keep up). */ + pause() { + this.socket.pause(); + } + + /** Resume reading after `pause`. */ + resume() { + this.socket.resume(); + } + + close() { + try { + this.socket.destroy(); + } catch { + /* ignore */ + } + } +} diff --git a/programs/wsproxy/tests/test/resilience.test.mjs b/programs/wsproxy/tests/test/resilience.test.mjs new file mode 100644 index 000000000000..22cd9fabd0f7 --- /dev/null +++ b/programs/wsproxy/tests/test/resilience.test.mjs @@ -0,0 +1,71 @@ +import { describe, it, expect } from "vitest"; +import { runQuery, Session } from "./helpers.mjs"; +import { spawnProxy } from "./proc.mjs"; + +describe("resilience", () => { + it("reuses the session connection after a backend (semantic) error", async () => { + const s = new Session("JSONEachRow"); + const bad = await s.run("SELECT * FROM default.no_such_table_zz"); + expect(bad.control.event).toBe("error"); + + // The reused per-session Connection must still work for the next query. + const good = await s.run("SELECT 1 AS n"); + expect(good.control.event).toBe("end"); + expect(JSON.parse(good.text.trim()).n).toBe(1); + s.close(); + }); + + it("reuses the session after a syntax error", async () => { + const s = new Session("JSONEachRow"); + const bad = await s.run("SELCT 1"); // typo -> backend syntax error + expect(bad.control.event).toBe("error"); + + const good = await s.run("SELECT 2 AS n"); + expect(good.control.event).toBe("end"); + expect(JSON.parse(good.text.trim()).n).toBe(2); + s.close(); + }); + + it("stays healthy after a client cancels a query mid-stream", async () => { + // Session A: start a slow query, then cancel by closing the socket. + const a = new Session("JSONEachRow"); + await a.ready(); + a.sendQuery("SELECT sleepEachRow(0.2), number FROM numbers(50) SETTINGS max_block_size = 1"); + for (let i = 0; i < 50; i++) { + const f = await a.nextFrame(); + if (f.type === "binary" || f.type === "close") break; + } + a.close(); + + // Session B: the proxy still serves new sessions. + const b = await runQuery("SELECT 42 AS n"); + expect(b.control.event).toBe("end"); + expect(JSON.parse(b.text.trim()).n).toBe(42); + }, 20000); + + it("returns an error (no hang or crash) when the backend is unreachable", async () => { + // A second proxy instance pointed at a closed backend port. + const proxy = await spawnProxy({ listenPort: 9011, backendPort: 59999 }); + try { + const { control } = await runQuery("SELECT 1", { baseUrl: proxy.url }); + // Either a delivered error event or a clean socket close is acceptable; + // what matters is that it neither hangs nor takes down the proxy. + expect(["error", "closed"]).toContain(control.event); + } finally { + proxy.stop(); + } + }, 20000); + + it("keeps serving after an unreachable-backend session", async () => { + const proxy = await spawnProxy({ listenPort: 9012, backendPort: 59999 }); + try { + await runQuery("SELECT 1", { baseUrl: proxy.url }); // fails, but must not crash the proxy + const ok = await runQuery("SELECT 7 AS n", { baseUrl: proxy.url }).catch(() => null); + // The second connection still gets a response (another error), proving the + // proxy process survived the first failure. + expect(ok === null || ["error", "closed"].includes(ok.control.event)).toBe(true); + } finally { + proxy.stop(); + } + }, 20000); +}); diff --git a/programs/wsproxy/tests/test/select.test.mjs b/programs/wsproxy/tests/test/select.test.mjs new file mode 100644 index 000000000000..1c64106d2ac6 --- /dev/null +++ b/programs/wsproxy/tests/test/select.test.mjs @@ -0,0 +1,64 @@ +import { describe, it, expect } from "vitest"; +import { Session, runQuery } from "./helpers.mjs"; + +describe("SELECT", () => { + it("returns a scalar SELECT as JSONEachRow", async () => { + const { text, control } = await runQuery("SELECT 1 AS a, 'hello' AS b"); + expect(control.event).toBe("end"); + expect(text).toContain('{"a":1,"b":"hello"}'); + }); + + it("streams a small result set as one line per row", async () => { + const { text, control } = await runQuery("SELECT number FROM numbers(5)"); + expect(control.event).toBe("end"); + const lines = text.split("\n").filter((l) => l.length > 0); + expect(lines).toHaveLength(5); + }); + + it("respects the output format chosen via ?format=", async () => { + const sql = "SELECT 42, 'x'"; + + const tsv = await runQuery(sql, { format: "TSV" }); + expect(tsv.control.event).toBe("end"); + expect(tsv.text.trim()).toBe("42\tx"); + + const csv = await runQuery(sql, { format: "CSV" }); + expect(csv.control.event).toBe("end"); + expect(csv.text).toContain('42,"x"'); + + const json = await runQuery(sql, { format: "JSONEachRow" }); + expect(json.control.event).toBe("end"); + const line = json.text.split("\n").find((l) => l.length > 0); + expect(() => JSON.parse(line)).not.toThrow(); + }); + + it("propagates a backend error as an error control event", async () => { + const { control } = await runQuery("SELECT * FROM default.a_missing_table_zz"); + expect(control.event).toBe("error"); + expect(typeof control.message).toBe("string"); + expect(control.message.length).toBeGreaterThan(0); + expect(control.message).toMatch(/table|UNKNOWN_TABLE/i); + }); + + it("streams a large result set without dropping rows", async () => { + const { text, control } = await runQuery("SELECT number FROM numbers(10000)"); + expect(control.event).toBe("end"); + const lines = text.split("\n").filter((l) => l.length > 0); + expect(lines).toHaveLength(10000); + }); + + it("runs multiple queries over one persistent session", async () => { + const s = new Session("JSONEachRow"); + try { + const first = await s.run("SELECT 1 AS n"); + expect(first.control.event).toBe("end"); + expect(first.text).toContain('{"n":1}'); + + const second = await s.run("SELECT 2 AS n"); + expect(second.control.event).toBe("end"); + expect(second.text).toContain('{"n":2}'); + } finally { + s.close(); + } + }); +}); diff --git a/programs/wsproxy/tests/test/serverSetup.mjs b/programs/wsproxy/tests/test/serverSetup.mjs new file mode 100644 index 000000000000..1f24a8163f08 --- /dev/null +++ b/programs/wsproxy/tests/test/serverSetup.mjs @@ -0,0 +1,54 @@ +// Vitest global setup for the IN-SERVER WebSocket endpoint (ws_port), as opposed +// to the standalone proxy. Spawns the freshly-built clickhouse-server with a +// ws_port on the same PROXY_PORT the helpers use, so the existing test files run +// unchanged against `ws://127.0.0.1:9010`. Only the query/protocol tests apply +// here; proxy-specific suites (its own backend/TLS/compression) are excluded by +// vitest.server.config.mjs. + +import net from "node:net"; +import { spawnServerWithWs } from "./proc.mjs"; + +const TCP_PORT = 19000; +const WS_PORT = 9010; // matches PROXY_URL default in helpers.mjs + +function portOpen(port, host = "127.0.0.1") { + return new Promise((resolve) => { + const socket = net.connect({ port, host }, () => { + socket.destroy(); + resolve(true); + }); + socket.on("error", () => resolve(false)); + }); +} + +export default async function setup() { + const stops = []; + + if (!(await portOpen(WS_PORT))) { + const server = await spawnServerWithWs({ + wsPort: WS_PORT, + tcpPort: TCP_PORT, + allowedOrigins: "http://127.0.0.1, https://trusted.example", + }); + stops.push(() => server.stop()); + } + + // Create the shared test table over the in-server WebSocket (DDL via the query path). + const { runQuery } = await import("./helpers.mjs"); + const ddl = await runQuery( + "CREATE TABLE IF NOT EXISTS default.wsp_test (a UInt32, b String) ENGINE = Memory", + ); + if (ddl.control.event !== "end") { + throw new Error(`failed to create test table: ${JSON.stringify(ddl.control)}`); + } + + return () => { + for (const stop of stops) { + try { + stop(); + } catch { + /* ignore */ + } + } + }; +} diff --git a/programs/wsproxy/tests/test/tls.test.mjs b/programs/wsproxy/tests/test/tls.test.mjs new file mode 100644 index 000000000000..eeff89f4538b --- /dev/null +++ b/programs/wsproxy/tests/test/tls.test.mjs @@ -0,0 +1,60 @@ +import { describe, it, expect } from "vitest"; +import { runQuery, Session } from "./helpers.mjs"; +import { spawnBackend, spawnProxy } from "./proc.mjs"; + +// Stands up its own TLS-enabled ClickHouse (self-signed cert) plus a proxy +// configured to connect to it over the secure native protocol. +describe("proxy -> backend TLS", () => { + it("round-trips a query over a TLS backend connection", async () => { + const backend = await spawnBackend({ tcpPort: 9003, securePort: 9444 }); + const proxy = await spawnProxy({ + listenPort: 9015, + backendPort: 9444, + secure: true, + acceptInvalidCert: true, // self-signed cert + }); + try { + const { text, control } = await runQuery("SELECT 'tls' AS s, currentUser() AS u", { + baseUrl: proxy.url, + }); + expect(control.event).toBe("end"); + const row = JSON.parse(text.trim()); + expect(row.s).toBe("tls"); + expect(row.u).toBe("default"); + } finally { + proxy.stop(); + backend.stop(); + } + }, 60000); + + it("passes credentials through over TLS", async () => { + const backend = await spawnBackend({ tcpPort: 9004, securePort: 9445 }); + const proxy = await spawnProxy({ + listenPort: 9016, + backendPort: 9445, + secure: true, + acceptInvalidCert: true, + }); + try { + // Correct credentials over TLS. + const ok = await runQuery("SELECT currentUser() AS u", { + baseUrl: proxy.url, + user: "wsp_user", + password: "wsp_pass", + }); + expect(ok.control.event).toBe("end"); + expect(JSON.parse(ok.text.trim()).u).toBe("wsp_user"); + + // Wrong password is still rejected over TLS. + const bad = await runQuery("SELECT 1", { + baseUrl: proxy.url, + user: "wsp_user", + password: "nope", + }); + expect(bad.control.event).toBe("error"); + } finally { + proxy.stop(); + backend.stop(); + } + }, 60000); +}); diff --git a/programs/wsproxy/tests/test/types.test.mjs b/programs/wsproxy/tests/test/types.test.mjs new file mode 100644 index 000000000000..abff499f0f21 --- /dev/null +++ b/programs/wsproxy/tests/test/types.test.mjs @@ -0,0 +1,195 @@ +import { describe, it, expect } from "vitest"; +import { runQuery, backendScalar, Session } from "./helpers.mjs"; + +// Round-trip a typed literal through the proxy's format layer, rendering it as +// JSONEachRow, and return the parsed `v` value. This validates that the proxy +// faithfully carries ClickHouse's data types through the output format. +async function selectJSON(expr) { + const { text, control } = await runQuery(`SELECT ${expr} AS v`, { + format: "JSONEachRow", + }); + expect(control.event).toBe("end"); + return JSON.parse(text.trim()).v; +} + +describe("type coverage", () => { + it("round-trips integers and floats", async () => { + expect(await selectJSON("toInt64(-5)")).toBe(-5); + // 64-bit integers must survive intact. NOTE: the proxy currently renders + // UInt64 UNQUOTED in JSONEachRow, whereas clickhouse-client quotes by + // default (output_format_json_quote_64bit_integers) — a format-settings + // faithfulness gap for the productionization track (unquoted big ints lose + // precision when JSON.parsed in JS). Assert on the raw digits so this tests + // data integrity independent of quoting. + const u64 = await runQuery("SELECT toUInt64(18446744073709551615) AS v", { + format: "JSONEachRow", + }); + expect(u64.control.event).toBe("end"); + expect(u64.text).toContain("18446744073709551615"); + expect(await selectJSON("toFloat64(1.5)")).toBe(1.5); + }); + + it("round-trips a Decimal", async () => { + // Decimals render as JSON numbers in JSONEachRow. + expect(await selectJSON("toDecimal64(3.14, 2)")).toBe(3.14); + }); + + it("round-trips strings with unicode and escapes", async () => { + expect(await selectJSON("'a\\tb\\n\"c\"'")).toBe('a\tb\n"c"'); + expect(await selectJSON("'héllo'")).toBe("héllo"); + }); + + it("round-trips date and time types", async () => { + expect(await selectJSON("toDate('2020-01-02')")).toBe("2020-01-02"); + expect(await selectJSON("toDateTime('2020-01-02 03:04:05')")).toBe( + "2020-01-02 03:04:05", + ); + expect(await selectJSON("toDateTime64('2020-01-02 03:04:05.123', 3)")).toBe( + "2020-01-02 03:04:05.123", + ); + }); + + it("round-trips a UUID", async () => { + expect( + await selectJSON("toUUID('00000000-0000-0000-0000-000000000001')"), + ).toBe("00000000-0000-0000-0000-000000000001"); + }); + + it("round-trips IPv4 and IPv6", async () => { + expect(await selectJSON("toIPv4('1.2.3.4')")).toBe("1.2.3.4"); + expect(await selectJSON("toIPv6('::1')")).toBe("::1"); + }); + + it("round-trips an Enum", async () => { + expect(await selectJSON("CAST('a', 'Enum8(\\'a\\'=1,\\'b\\'=2)')")).toBe("a"); + }); + + it("round-trips Nullable (null and non-null)", async () => { + expect(await selectJSON("CAST(NULL, 'Nullable(String)')")).toBeNull(); + expect(await selectJSON("CAST('present', 'Nullable(String)')")).toBe( + "present", + ); + }); + + it("round-trips LowCardinality", async () => { + expect(await selectJSON("CAST('x', 'LowCardinality(String)')")).toBe("x"); + }); + + it("round-trips an Array", async () => { + expect(await selectJSON("[1,2,3]::Array(UInt32)")).toEqual([1, 2, 3]); + }); + + it("round-trips a Map (rendered as a JSON object)", async () => { + // UInt8 values fit in JS numbers, so no 64-bit quoting here. + expect(await selectJSON("map('k',1,'j',2)")).toEqual({ k: 1, j: 2 }); + }); + + it("round-trips a Tuple", async () => { + // JSONEachRow renders unnamed tuples as an object keyed by position + // ({"1":1,"2":"x"}); assert loosely that the values are present. + const v = await selectJSON("tuple(1,'x')"); + const values = Array.isArray(v) ? v : Object.values(v); + expect(values).toContain(1); + expect(values).toContain("x"); + }); +}); + +describe("advanced types", () => { + it("handles the JSON type", async () => { + const { text, control } = await runQuery( + `SELECT '{"a":1,"b":"x"}'::JSON AS v`, + { format: "JSONEachRow" }, + ); + expect(control.event).toBe("end"); + expect(text).toContain("a"); + expect(text).toContain("1"); + }); + + it("handles the Dynamic type", async () => { + const v = await selectJSON("42::Dynamic"); + // The value survives; may arrive as a number or a string depending on + // Dynamic's JSON rendering, so just assert it is present and equals 42. + expect(v).not.toBeUndefined(); + expect(v).not.toBeNull(); + expect(String(v)).toBe("42"); + }); + + it("handles the Variant type", async () => { + // Cast from a type that IS in the Variant (UInt64), not a bare UInt8 literal. + const { control, text } = await runQuery( + "SELECT CAST(toUInt64(42), 'Variant(UInt64, String)') AS v", + { format: "JSONEachRow" }, + ); + expect(control.event).toBe("end"); + expect(text).toContain("42"); + }); + + it("serializes an AggregateFunction state via a binary format", async () => { + // JSONEachRow cannot represent an aggregate state; RowBinary can. This + // proves the proxy streams aggregate-state bytes without error. + const { data, control } = await runQuery( + "SELECT sumState(number) AS v FROM numbers(10)", + { format: "RowBinary" }, + ); + expect(control.event).toBe("end"); + expect(data.length).toBeGreaterThan(0); + }); +}); + +describe("INSERT round-trip", () => { + // Prove the INPUT path handles complex types end-to-end: create a Memory + // table, insert one row of a complex type via JSONEachRow, then read it back. + async function withTable(columnDef, run) { + const table = `default.wsp_types_${Date.now()}_${Math.floor(Math.random() * 1e6)}`; + const create = await runQuery( + `CREATE TABLE ${table} (${columnDef}) ENGINE = Memory`, + ); + expect(create.control.event).toBe("end"); + try { + await run(table); + } finally { + await runQuery(`DROP TABLE IF EXISTS ${table}`); + } + } + + it("round-trips an Array(UInt32) column", async () => { + await withTable("v Array(UInt32)", async (table) => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + `INSERT INTO ${table} FORMAT JSONEachRow`, + ['{"v":[1,2,3]}\n'], + ); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + expect(await backendScalar(`SELECT v FROM ${table}`)).toBe("[1,2,3]"); + }); + }); + + it("round-trips a Nullable(String) column", async () => { + await withTable("v Nullable(String)", async (table) => { + const s = new Session("JSONEachRow"); + try { + const { control } = await s.insert( + `INSERT INTO ${table} FORMAT JSONEachRow`, + ['{"v":null}\n{"v":"hi"}\n'], + ); + expect(control.event).toBe("end"); + } finally { + s.close(); + } + // TSV renders SQL NULL as \N; the non-null value round-trips verbatim. + expect(await backendScalar(`SELECT count() FROM ${table}`)).toBe("2"); + expect( + await backendScalar(`SELECT v FROM ${table} WHERE v IS NOT NULL`), + ).toBe("hi"); + expect( + await backendScalar( + `SELECT count() FROM ${table} WHERE v IS NULL`, + ), + ).toBe("1"); + }); + }); +}); diff --git a/programs/wsproxy/tests/vitest.config.mjs b/programs/wsproxy/tests/vitest.config.mjs new file mode 100644 index 000000000000..775bd666550a --- /dev/null +++ b/programs/wsproxy/tests/vitest.config.mjs @@ -0,0 +1,13 @@ +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { + // Launches the backend ClickHouse server and the proxy (see test/globalSetup.mjs). + globalSetup: "./test/globalSetup.mjs", + // These are integration tests against a single shared backend, so run test + // files one at a time to avoid cross-file interference on shared tables. + fileParallelism: false, + testTimeout: 30_000, + hookTimeout: 60_000, + }, +}); diff --git a/programs/wsproxy/tests/vitest.server.config.mjs b/programs/wsproxy/tests/vitest.server.config.mjs new file mode 100644 index 000000000000..1bf42ef1236a --- /dev/null +++ b/programs/wsproxy/tests/vitest.server.config.mjs @@ -0,0 +1,26 @@ +import { defineConfig } from "vitest/config"; + +// Runs the query/protocol test suite against the IN-SERVER WebSocket endpoint +// (clickhouse-server with a ws_port) instead of the standalone proxy. The bridge +// code is shared, so the same tests should pass. Proxy-specific suites are +// excluded: they spawn their own proxy and exercise features that only exist in +// the sidecar (a separate remote backend, proxy->backend TLS, and the +// WSPROXY_BACKEND_COMPRESSION codec). +// +// Run with: npx vitest run --config vitest.server.config.mjs +export default defineConfig({ + test: { + globalSetup: "./test/serverSetup.mjs", + fileParallelism: false, + testTimeout: 30_000, + hookTimeout: 60_000, + include: ["test/**/*.test.mjs"], + exclude: [ + "test/backend-failure.test.mjs", // kills a separate backend + "test/resilience.test.mjs", // points a proxy at a dead backend + "test/tls.test.mjs", // proxy->backend TLS (N/A in-server) + "test/backpressure.test.mjs", // spawns its own proxy with a send timeout + "test/compression.test.mjs", // WSPROXY_BACKEND_COMPRESSION (proxy-only) + ], + }, +}); diff --git a/src/Client/LocalConnection.cpp b/src/Client/LocalConnection.cpp index 16cd7126ce16..69cb7c73acdc 100644 --- a/src/Client/LocalConnection.cpp +++ b/src/Client/LocalConnection.cpp @@ -486,15 +486,17 @@ bool LocalConnection::poll(size_t) try { - while (pollImpl()) + if (pollImpl()) { - LOG_TEST(&Poco::Logger::get("LocalConnection"), "Executor timeout encountered, will retry"); + LOG_TEST(&Poco::Logger::get("LocalConnection"), "Executor timeout encountered"); if (needSendProgressOrMetrics()) return true; if (needSendLogs()) return true; + + return false; } } catch (const Exception & e) @@ -709,15 +711,9 @@ Packet LocalConnection::receivePacket() return packet; } - if (!next_packet_type) + while (!next_packet_type) poll(0); - if (!next_packet_type) - { - packet.type = Protocol::Server::EndOfStream; - return packet; - } - packet.type = next_packet_type.value(); switch (next_packet_type.value()) { diff --git a/src/Server/HTTPHandlerFactory.cpp b/src/Server/HTTPHandlerFactory.cpp index 31c22da8256b..5d5454688736 100644 --- a/src/Server/HTTPHandlerFactory.cpp +++ b/src/Server/HTTPHandlerFactory.cpp @@ -10,6 +10,7 @@ #include #include #include +#include #if CLICKHOUSE_CLOUD #include #endif @@ -302,6 +303,17 @@ createHTTPHandlerFactory(IServer & server, const Poco::Util::AbstractConfigurati return factory; } +static inline HTTPRequestHandlerFactoryPtr createWSHandlerFactory(IServer & server, const std::string & name) +{ + /// The `ws_port` speaks WebSocket only: every request goes to the WSHandler, + /// which serves an info page for a plain GET and upgrades a WebSocket request. + auto factory = std::make_shared(name); + auto main_handler = std::make_shared>(server); + main_handler->allowGetAndHeadRequest(); + factory->addHandler(main_handler); + return factory; +} + static inline HTTPRequestHandlerFactoryPtr createInterserverHTTPHandlerFactory(IServer & server, const std::string & name, const Poco::Util::AbstractConfiguration & config) { auto factory = std::make_shared(name); @@ -320,6 +332,8 @@ HTTPRequestHandlerFactoryPtr createHandlerFactory(IServer & server, const Poco:: return createHTTPHandlerFactory(server, config, name, async_metrics, http_handlers_key.empty() ? "http_handlers" : http_handlers_key); if (name == "InterserverIOHTTPHandler-factory" || name == "InterserverIOHTTPSHandler-factory") return createInterserverHTTPHandlerFactory(server, name, config); + if (name == "WSHandler-factory") + return createWSHandlerFactory(server, name); if (name == "PrometheusHandler-factory") return createPrometheusHandlerFactory(server, config, async_metrics, name); if (name == "KeeperPrometheusHandler-factory") diff --git a/src/Server/ServerType.cpp b/src/Server/ServerType.cpp index 932894e06447..eea3835d5b9b 100644 --- a/src/Server/ServerType.cpp +++ b/src/Server/ServerType.cpp @@ -52,6 +52,7 @@ bool ServerType::shouldStart(Type server_type, const std::string & server_custom case Type::TCP_SECURE: case Type::HTTP: case Type::HTTPS: + case Type::WS: case Type::MYSQL: case Type::GRPC: case Type::POSTGRESQL: @@ -109,6 +110,9 @@ bool ServerType::shouldStop(const std::string & port_name) const else if (port_name == "https_port") port_type = Type::HTTPS; + else if (port_name == "ws_port") + port_type = Type::WS; + else if (port_name == "tcp_port") port_type = Type::TCP; diff --git a/src/Server/ServerType.h b/src/Server/ServerType.h index 2839ccfd99a3..0f5f53c0a73a 100644 --- a/src/Server/ServerType.h +++ b/src/Server/ServerType.h @@ -17,6 +17,7 @@ class ServerType TCP, HTTP, HTTPS, + WS, MYSQL, GRPC, ARROW_FLIGHT, diff --git a/src/Server/WSHandler.cpp b/src/Server/WSHandler.cpp new file mode 100644 index 000000000000..55b299293255 --- /dev/null +++ b/src/Server/WSHandler.cpp @@ -0,0 +1,462 @@ +#include + +#include +#include + +#include +#include +#include + +#include +#include +#include +#include + +#include +#include +#include + +#include + +#include +#include +#include +#include +#include + +#include + +#include +#include +#include +#include + +namespace DB +{ + +using namespace DB::WsProxy; + +namespace ErrorCodes +{ + extern const int BAD_ARGUMENTS; +} + +namespace +{ + +/// One resolved credential pair (delegated to ClickHouse for authN/authZ). +struct ResolvedCredentials +{ + String user = "default"; + String password; +}; + +String queryParam(const String & uri_string, const String & name, const String & fallback) +{ + Poco::URI uri(uri_string); + for (const auto & param : uri.getQueryParameters()) + if (param.first == name && !param.second.empty()) + return param.second; + return fallback; +} + +std::optional positiveInt64QueryParam(const String & uri_string, const String & name) +{ + Poco::URI uri(uri_string); + std::optional result; + for (const auto & param : uri.getQueryParameters()) + { + if (param.first != name) + continue; + + if (result) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Query parameter `{}` must not be repeated", name); + + Int64 value = 0; + const char * begin = param.second.data(); + const char * end = begin + param.second.size(); + const auto parse_result = std::from_chars(begin, end, value, 10); + if (param.second.empty() || parse_result.ec != std::errc{} || parse_result.ptr != end || value <= 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Query parameter `{}` must be a positive integer", name); + result = value; + } + return result; +} + +String normalizeOrigin(const String & origin) +{ + Poco::URI uri(origin); + String scheme = uri.getScheme(); + String host = uri.getHost(); + std::transform(scheme.begin(), scheme.end(), scheme.begin(), ::tolower); + std::transform(host.begin(), host.end(), host.begin(), ::tolower); + + if ((scheme != "http" && scheme != "https") || host.empty() || !uri.getUserInfo().empty() + || (!uri.getPath().empty() && uri.getPath() != "/") || !uri.getQuery().empty() || !uri.getFragment().empty()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Origin header"); + + const UInt16 port = uri.getPort(); + const UInt16 default_port = scheme == "https" ? 443 : 80; + if (host.find(':') != String::npos) + host = "[" + host + "]"; + return scheme + "://" + host + (port && port != default_port ? ":" + std::to_string(port) : ""); +} + +bool originAllowed(const HTTPServerRequest & request, const String & allowed_origins, LoggerPtr log) +{ + const String origin = request.get("Origin", ""); + if (origin.empty()) + return true; + + String normalized_origin; + try + { + normalized_origin = normalizeOrigin(origin); + } + catch (...) + { + LOG_WARNING(log, "WebSocket upgrade rejected: malformed Origin header"); + return false; + } + + if (!allowed_origins.empty()) + { + bool allowed = false; + size_t pos = 0; + while (pos <= allowed_origins.size()) + { + const size_t comma = allowed_origins.find(',', pos); + const size_t end_pos = comma == String::npos ? allowed_origins.size() : comma; + const size_t start = allowed_origins.find_first_not_of(" \t", pos); + if (start == String::npos || start >= end_pos) + { + LOG_WARNING(log, "WebSocket upgrade rejected: empty origin in `ws_allowed_origins`"); + return false; + } + const size_t last = allowed_origins.find_last_not_of(" \t", end_pos - 1); + try + { + if (normalizeOrigin(allowed_origins.substr(start, last - start + 1)) == normalized_origin) + allowed = true; + } + catch (...) + { + LOG_WARNING(log, "WebSocket upgrade rejected: malformed origin in `ws_allowed_origins`"); + return false; + } + if (comma == String::npos) + break; + pos = comma + 1; + } + return allowed; + } + + try + { + const String request_scheme = request.isSecure() ? "https" : "http"; + return normalizeOrigin(request_scheme + "://" + request.getHost()) == normalized_origin; + } + catch (...) + { + LOG_WARNING(log, "WebSocket upgrade rejected: malformed Host header"); + return false; + } +} + +/// Per RFC 7230 the `Connection` header is a comma-separated token list; match the +/// exact `upgrade` token rather than a substring. +bool hasUpgradeToken(const String & connection_header) +{ + String value = connection_header; + std::transform(value.begin(), value.end(), value.begin(), ::tolower); + size_t pos = 0; + while (pos <= value.size()) + { + size_t comma = value.find(',', pos); + size_t end_pos = (comma == String::npos) ? value.size() : comma; + size_t start = value.find_first_not_of(" \t", pos); + if (start != String::npos && start < end_pos) + { + size_t last = value.find_last_not_of(" \t", end_pos - 1); + if (value.substr(start, last - start + 1) == "upgrade") + return true; + } + if (comma == String::npos) + break; + pos = comma + 1; + } + return false; +} + +/// Minimal JSON string escaping for a short error message in a control frame. +String jsonEscape(const String & s) +{ + String out; + out.reserve(s.size() + 2); + for (char c : s) + { + switch (c) + { + case '"': out += R"(\")"; break; + case '\\': out += R"(\\)"; break; + case '\n': out += R"(\n)"; break; + case '\r': out += R"(\r)"; break; + case '\t': out += R"(\t)"; break; + default: + if (static_cast(c) < 0x20) + { + static const char * hex = "0123456789abcdef"; + out += R"(\u00)"; + out += hex[(c >> 4) & 0xF]; + out += hex[c & 0xF]; + } + else + out += c; + } + } + return out; +} + +/// Resolve credentials the same way clickhouse-wsproxy does, so the same clients +/// work against either deployment: Authorization: Basic > X-ClickHouse-User/-Key +/// headers > ?user=/?password= URL params > the `default` user. +ResolvedCredentials resolveCredentials(const HTTPServerRequest & request, const String & uri) +{ + ResolvedCredentials creds; + + const String auth = request.get("Authorization", ""); + const size_t auth_scheme_end = auth.find_first_of(" \t"); + String auth_scheme = auth.substr(0, auth_scheme_end); + std::transform(auth_scheme.begin(), auth_scheme.end(), auth_scheme.begin(), ::tolower); + if (auth_scheme == "basic") + { + try + { + if (auth_scheme_end != 5 || auth.size() <= 6 || auth[5] != ' ') + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Basic Authorization header"); + const String encoded = auth.substr(6); + const size_t padding_pos = encoded.find('='); + const size_t data_end = padding_pos == String::npos ? encoded.size() : padding_pos; + if (encoded.size() % 4 != 0 || encoded.size() - data_end > 2) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Basic Authorization header"); + for (size_t i = 0; i < encoded.size(); ++i) + { + const unsigned char c = encoded[i]; + const bool is_base64_character + = (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c == '+' || c == '/'; + if (i < data_end ? !is_base64_character : c != '=') + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Basic Authorization header"); + } + const String decoded = base64Decode(encoded); + const size_t colon = decoded.find(':'); + if (colon == String::npos || colon == 0) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Basic Authorization header"); + creds.user = decoded.substr(0, colon); + creds.password = decoded.substr(colon + 1); + return creds; + } + catch (...) + { + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Malformed Basic Authorization header"); + } + } + if (!auth.empty()) + throw Exception(ErrorCodes::BAD_ARGUMENTS, "Unsupported Authorization scheme"); + + const String header_user = request.get("X-ClickHouse-User", ""); + if (!header_user.empty()) + { + creds.user = header_user; + creds.password = request.get("X-ClickHouse-Key", ""); + return creds; + } + + const String param_user = queryParam(uri, "user", ""); + if (!param_user.empty()) + { + creds.user = param_user; + creds.password = queryParam(uri, "password", ""); + } + return creds; +} + +} + +WSHandler::WSHandler(IServer & server_) : server(server_) +{ +} + +void WSHandler::handleRequest(HTTPServerRequest & request, HTTPServerResponse & response, const ProfileEvents::Event &) +{ + try + { + String upgrade = request.get("Upgrade", ""); + std::transform(upgrade.begin(), upgrade.end(), upgrade.begin(), ::tolower); + + if (upgrade == "websocket" && hasUpgradeToken(request.get("Connection", ""))) + handleWebSocket(request, response); + else + serveInfo(request, response); + } + catch (...) + { + tryLogCurrentException("WSHandler"); + } +} + +void WSHandler::serveInfo(HTTPServerRequest & request, HTTPServerResponse & response) +{ + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_OK); + response.setContentType("text/plain; charset=UTF-8"); + *response.send() + << "ClickHouse WebSocket endpoint\n" + << "Requested: " << request.getURI() << "\n" + << "Open a WebSocket connection to run queries. Send a query as a text\n" + << "frame; results stream back as binary frames in the output format\n" + << "(set via ?format=..., default JSONEachRow), followed by a JSON\n" + << "control frame ({\"event\":\"end\"} / \"error\" / \"cancelled\").\n"; +} + +void WSHandler::handleWebSocket(HTTPServerRequest & request, HTTPServerResponse & response) +{ + LoggerPtr log = getLogger("WSHandler"); + + if (request.getMethod() != Poco::Net::HTTPRequest::HTTP_GET) + { + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_METHOD_NOT_ALLOWED); + *response.send() << "WebSocket upgrade requires GET method.\n"; + return; + } + + String ws_key = request.get("Sec-WebSocket-Key", ""); + if (!isValidWebSocketKey(ws_key)) + { + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_BAD_REQUEST); + *response.send() << "Invalid or missing Sec-WebSocket-Key.\n"; + return; + } + if (request.get("Sec-WebSocket-Version", "") != "13") + { + response.set("Sec-WebSocket-Version", "13"); + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_BAD_REQUEST); + *response.send() << "Unsupported WebSocket version.\n"; + return; + } + + if (!originAllowed(request, server.config().getString("ws_allowed_origins", ""), log)) + { + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_FORBIDDEN); + *response.send() << "Origin not allowed.\n"; + return; + } + + const String & uri = request.getURI(); + String out_format; + String logs_level; + std::optional flow_credit_param; + String parse_param; + String parallel_param; + ResolvedCredentials creds; + try + { + out_format = queryParam(uri, "format", "JSONEachRow"); + logs_level = queryParam(uri, "logs", ""); + flow_credit_param = positiveInt64QueryParam(uri, "flow"); + parse_param = queryParam(uri, "parse", ""); + parallel_param = queryParam(uri, "parallel", ""); + creds = resolveCredentials(request, uri); + } + catch (...) + { + LOG_DEBUG(log, "Invalid WebSocket request: {}", getCurrentExceptionMessage(false)); + response.setStatusAndReason(Poco::Net::HTTPResponse::HTTP_BAD_REQUEST); + *response.send() << getCurrentExceptionMessage(false) << "\n"; + return; + } + + const bool flow_enabled = flow_credit_param.has_value(); + const Int64 flow_credit = flow_credit_param.value_or(0); + const bool parse_enabled = parse_param == "1" || parse_param == "true"; + const bool parallel_enabled = parallel_param == "1" || parallel_param == "true"; + + /// Complete the handshake by writing 101 directly to the socket; from here on + /// the stream is in WebSocket framing mode and we own the socket. + Poco::Net::StreamSocket & socket = response.getSocket(); + const String handshake + = "HTTP/1.1 101 Switching Protocols\r\n" + "Upgrade: websocket\r\n" + "Connection: Upgrade\r\n" + "Sec-WebSocket-Accept: " + + computeWebSocketAccept(ws_key) + "\r\n\r\n"; + + size_t sent_total = 0; + while (sent_total < handshake.size()) + { + int sent = socket.sendBytes(handshake.data() + sent_total, static_cast(handshake.size() - sent_total)); + if (sent <= 0) + { + LOG_DEBUG(log, "Failed to write WebSocket handshake"); + return; + } + sent_total += static_cast(sent); + } + + SCOPE_EXIT({ + try + { + socket.shutdown(); + } + catch (...) + { + LOG_DEBUG(log, "WebSocket socket shutdown: {}", getCurrentExceptionMessage(false)); + } + }); + + /// Bound each blocking read; bound blocking writes so a client that stops + /// reading cannot pin a handler thread (mirrors the standalone proxy). + socket.setReceiveTimeout(Poco::Timespan(300, 0)); + socket.setSendTimeout(Poco::Timespan(30, 0)); + + /// Authenticate against the server (ClickHouse performs authN/authZ) and + /// build an in-process connection bound to that session. + ContextMutablePtr session_context; + std::unique_ptr connection; + try + { + auto session = std::make_unique(server.context(), ClientInfo::Interface::HTTP, request.isSecure()); + session->authenticate(BasicCredentials(creds.user, creds.password), request.clientAddress()); + session_context = session->makeSessionContext(); + /// LocalConnection ignores the per-query settings passed to sendQuery and reads them from + /// its context, so apply the session-level knobs here. send_logs_level drives whether the + /// in-process query pushes Log packets (which WebSocketSession forwards as `log` frames). + if (!logs_level.empty()) + session_context->setSetting("send_logs_level", logs_level); + connection = std::make_unique( + std::move(session), /* in */ nullptr, /* send_progress */ true, /* send_profile_events */ true, /* server_display_name */ ""); + } + catch (...) + { + const String message = getCurrentExceptionMessage(false); + LOG_DEBUG(log, "WebSocket authentication failed: {}", message); + try + { + sendWebSocketText(socket, R"({"event":"error","message":")" + jsonEscape(message) + "\"}"); + sendWebSocketClose(socket, /* 1008 policy violation */ 1008, "Authentication failed"); + } + catch (...) + { + LOG_DEBUG(log, "Failed to deliver auth error to client (already gone)"); + } + return; + } + + LOG_DEBUG(log, "WebSocket session established for user {}; format {}", creds.user, out_format); + + WebSocketSession session( + socket, session_context, out_format, logs_level, flow_enabled, flow_credit, parse_enabled, parallel_enabled); + session.run(*connection); + + LOG_DEBUG(log, "WebSocket session closed"); +} + +} diff --git a/src/Server/WSHandler.h b/src/Server/WSHandler.h new file mode 100644 index 000000000000..0d5aaa01c2b8 --- /dev/null +++ b/src/Server/WSHandler.h @@ -0,0 +1,34 @@ +#pragma once + +#include + +namespace DB +{ + +class IServer; + +/// In-server WebSocket query endpoint (the `ws_port`). +/// +/// A plain HTTP request serves a short info page. A WebSocket upgrade completes +/// the RFC 6455 handshake after validating any browser `Origin`, authenticates +/// via the server's own session auth (Basic / X-ClickHouse headers / +/// ?user=&password= / default), and then bridges +/// the socket to an in-process `LocalConnection` using the shared +/// `WebSocketSession`. The wire protocol is identical to `clickhouse-wsproxy`: +/// query as a text frame, results as binary frames in the `?format=` output +/// format, JSON control frames, and the `?logs`/`?flow`/`?parse`/`?parallel` knobs. +class WSHandler : public HTTPRequestHandler +{ +public: + explicit WSHandler(IServer & server_); + + void handleRequest(HTTPServerRequest & request, HTTPServerResponse & response, const ProfileEvents::Event & write_event) override; + +private: + void handleWebSocket(HTTPServerRequest & request, HTTPServerResponse & response); + void serveInfo(HTTPServerRequest & request, HTTPServerResponse & response); + + IServer & server; +}; + +} diff --git a/src/Server/WebSocketFrames.cpp b/src/Server/WebSocketFrames.cpp new file mode 100644 index 000000000000..861b339f3743 --- /dev/null +++ b/src/Server/WebSocketFrames.cpp @@ -0,0 +1,310 @@ +#include + +#include +#include + +#include +#include +#include + +#include +#include +#include + + +namespace DB::WsProxy +{ + +namespace +{ + +/// Monotonic clock in nanoseconds, self-contained so the framing layer has no +/// dependency beyond libc. Used only to enforce absolute read deadlines. +UInt64 monotonicNs() +{ + struct timespec ts + { + }; + clock_gettime(CLOCK_MONOTONIC, &ts); + return static_cast(ts.tv_sec) * 1'000'000'000ULL + static_cast(ts.tv_nsec); +} + +/// Send all bytes to the socket, handling partial writes. +void sendAllBytes(Poco::Net::StreamSocket & socket, const char * data, size_t len) +{ + size_t total = 0; + while (total < len) + { + const int chunk_size = static_cast(std::min(len - total, std::numeric_limits::max())); + int sent = socket.sendBytes(data + total, chunk_size); + if (sent <= 0) + throw Poco::IOException("Failed to send bytes to WebSocket"); + total += static_cast(sent); + } +} + +/// Read exactly `n` bytes. If `deadline_ns` is non-zero, abort when the +/// monotonic clock passes it. +bool readExact(Poco::Net::StreamSocket & socket, char * buf, size_t n, UInt64 deadline_ns) +{ + size_t total = 0; + while (total < n) + { + if (deadline_ns) + { + const UInt64 now_ns = monotonicNs(); + if (now_ns >= deadline_ns) + return false; + + const UInt64 remaining_ns = deadline_ns - now_ns; + const UInt64 remaining_us = remaining_ns / 1'000 + (remaining_ns % 1'000 != 0); + const auto poll_timeout_us = static_cast(std::min( + remaining_us, static_cast(std::numeric_limits::max()))); + if (!socket.poll(Poco::Timespan(poll_timeout_us), Poco::Net::Socket::SELECT_READ)) + return false; + } + + const int chunk_size = static_cast(std::min(n - total, std::numeric_limits::max())); + int received = socket.receiveBytes(buf + total, chunk_size); + if (received <= 0) + return false; + total += static_cast(received); + } + return true; +} + +} + +String computeWebSocketAccept(const String & key) +{ + const String magic = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; + Poco::SHA1Engine sha1; + sha1.update(key + magic); + auto digest = sha1.digest(); + return base64Encode(String(reinterpret_cast(digest.data()), digest.size())); +} + +bool isValidWebSocketKey(const String & key) +{ + if (key.empty() || key.size() > 128) + return false; + try + { + return base64Decode(key).size() == 16; + } + catch (...) /// Ok: malformed base64 is just an invalid key; the caller maps it to a 400. + { + return false; + } +} + +void sendWebSocketFrame(Poco::Net::StreamSocket & socket, uint8_t opcode, const char * data, size_t len) +{ + uint8_t header[10]; + size_t header_len = 2; + + header[0] = 0x80 | opcode; /// FIN + opcode + + if (len < 126) + { + header[1] = static_cast(len); + } + else if (len < 65536) + { + header[1] = 126; + header[2] = static_cast((len >> 8) & 0xFF); + header[3] = static_cast(len & 0xFF); + header_len = 4; + } + else + { + header[1] = 127; + for (int i = 0; i < 8; ++i) + header[2 + i] = static_cast((len >> (56 - 8 * i)) & 0xFF); + header_len = 10; + } + + /// Send the header then the payload. The caller serializes whole-frame sends + /// (see ProxySession's write mutex), so these two writes cannot interleave with + /// another frame; sending the payload directly avoids copying it into a + /// combined buffer (which, for large result frames, is a significant memcpy). + sendAllBytes(socket, reinterpret_cast(header), header_len); + if (len > 0) + sendAllBytes(socket, data, len); +} + +void sendWebSocketBinary(Poco::Net::StreamSocket & socket, const char * data, size_t len) +{ + sendWebSocketFrame(socket, Opcode::Binary, data, len); +} + +void sendWebSocketText(Poco::Net::StreamSocket & socket, const String & text) +{ + sendWebSocketFrame(socket, Opcode::Text, text.data(), text.size()); +} + +void sendWebSocketClose(Poco::Net::StreamSocket & socket, uint16_t code, const String & reason) +{ + if (reason.size() > 123) + throw Poco::InvalidArgumentException("WebSocket close reason exceeds 123 bytes"); + + String payload; + payload.push_back(static_cast((code >> 8) & 0xFF)); + payload.push_back(static_cast(code & 0xFF)); + payload.append(reason); + sendWebSocketFrame(socket, Opcode::Close, payload.data(), payload.size()); +} + +WebSocketFrame readWebSocketFrame( + Poco::Net::StreamSocket & socket, + UInt64 deadline_ns, + uint64_t max_payload_size, + UInt64 completion_timeout_ns) +{ + WebSocketFrame frame; + uint8_t header[2]; + + /// Waiting for the first byte is session idle time and is intentionally not + /// bounded by `completion_timeout_ns`. Once it arrives, cap the rest of the + /// frame so a client cannot retain a handler by trickling bytes indefinitely. + if (!readExact(socket, reinterpret_cast(header), 1, deadline_ns)) + return frame; + + if (completion_timeout_ns) + { + const UInt64 completion_deadline = monotonicNs() + completion_timeout_ns; + if (!deadline_ns || completion_deadline < deadline_ns) + deadline_ns = completion_deadline; + } + + if (!readExact(socket, reinterpret_cast(header + 1), 1, deadline_ns)) + return frame; + + frame.fin = (header[0] & 0x80) != 0; + frame.opcode = header[0] & 0x0F; + + /// RSV1/RSV2/RSV3 must be zero (no extensions negotiated). + if (header[0] & 0x70) + { + frame.protocol_error = true; + return frame; + } + + /// Reject reserved opcodes per RFC 6455 section 5.2. + if ((frame.opcode >= 0x03 && frame.opcode <= 0x07) || frame.opcode >= 0x0B) + { + frame.protocol_error = true; + return frame; + } + + bool masked = (header[1] & 0x80) != 0; + uint64_t payload_len = header[1] & 0x7F; + + /// RFC 6455: client-to-server frames MUST be masked. + if (!masked) + { + frame.protocol_error = true; + return frame; + } + + /// Control frames (opcode >= 0x08) must have FIN set and payload <= 125. + if (frame.opcode >= 0x08) + { + if (!frame.fin || payload_len > 125) + { + frame.protocol_error = true; + return frame; + } + } + + if (payload_len == 126) + { + uint8_t ext[2]; + if (!readExact(socket, reinterpret_cast(ext), 2, deadline_ns)) + return frame; + payload_len = (static_cast(ext[0]) << 8) | ext[1]; + if (payload_len < 126) + { + frame.protocol_error = true; + return frame; + } + } + else if (payload_len == 127) + { + uint8_t ext[8]; + if (!readExact(socket, reinterpret_cast(ext), 8, deadline_ns)) + return frame; + if (ext[0] & 0x80) + { + frame.protocol_error = true; + return frame; + } + payload_len = 0; + for (const auto & byte : ext) + payload_len = (payload_len << 8) | byte; + if (payload_len < 65536) + { + frame.protocol_error = true; + return frame; + } + } + + /// Cap the payload before allocating, so a crafted length header cannot + /// force a huge allocation. Distinct from `protocol_error` so the caller + /// can close with the dedicated 1009 code. + if (payload_len > max_payload_size) + { + frame.message_too_big = true; + return frame; + } + + uint8_t mask_key[4] = {}; + if (!readExact(socket, reinterpret_cast(mask_key), 4, deadline_ns)) + return frame; + + frame.payload.resize(payload_len); + if (payload_len > 0) + { + if (!readExact(socket, frame.payload.data(), payload_len, deadline_ns)) + return frame; + + for (uint64_t i = 0; i < payload_len; ++i) + frame.payload[i] ^= static_cast(mask_key[i % 4]); + } + + if (frame.opcode == Opcode::Close) + { + if (payload_len == 1) + { + frame.protocol_error = true; + return frame; + } + + if (payload_len >= 2) + { + const auto code = static_cast( + (static_cast(frame.payload[0]) << 8) | static_cast(frame.payload[1])); + const bool standard_code = code >= 1000 && code <= 1014 + && code != 1004 && code != 1005 && code != 1006; + const bool application_code = code >= 3000 && code < 5000; + if (!standard_code && !application_code) + { + frame.protocol_error = true; + return frame; + } + + if (payload_len > 2 + && !UTF8::isValidUTF8( + reinterpret_cast(frame.payload.data() + 2), frame.payload.size() - 2)) + { + frame.invalid_utf8 = true; + return frame; + } + } + } + + frame.valid = true; + return frame; +} + +} diff --git a/src/Server/WebSocketFrames.h b/src/Server/WebSocketFrames.h new file mode 100644 index 000000000000..3c2d7a25004b --- /dev/null +++ b/src/Server/WebSocketFrames.h @@ -0,0 +1,66 @@ +#pragma once + +#include + +#include + +/// Low-level RFC 6455 WebSocket framing over a raw stream socket. +/// +/// Lifted and adapted from `src/Server/WebTerminalRequestHandler.cpp`, factored +/// into a reusable module because the proxy session loop (native protocol <-> +/// WebSocket) will drive these primitives directly. Server-to-client frames are +/// never masked; client-to-server frames MUST be masked (enforced on read). + +namespace DB::WsProxy +{ + +/// WebSocket opcodes (RFC 6455 section 5.2). +namespace Opcode +{ + static constexpr uint8_t Continuation = 0x0; + static constexpr uint8_t Text = 0x1; + static constexpr uint8_t Binary = 0x2; + static constexpr uint8_t Close = 0x8; + static constexpr uint8_t Ping = 0x9; + static constexpr uint8_t Pong = 0xA; +} + +struct WebSocketFrame +{ + uint8_t opcode = 0; + bool fin = false; + String payload; + /// A fully and correctly read frame. + bool valid = false; + /// Set on an RFC 6455 protocol violation; caller closes with 1002. + bool protocol_error = false; + /// Set when the advertised payload exceeds `max_payload_size`; caller closes with 1009. + bool message_too_big = false; + /// Set when a close reason is not valid UTF-8; caller closes with 1007. + bool invalid_utf8 = false; +}; + +/// Compute the `Sec-WebSocket-Accept` response value per RFC 6455 section 4.2.2. +String computeWebSocketAccept(const String & key); + +/// Validate that `Sec-WebSocket-Key` is a base64-encoded 16-byte nonce. +bool isValidWebSocketKey(const String & key); + +/// Send a single unmasked frame with the given opcode. +void sendWebSocketFrame(Poco::Net::StreamSocket & socket, uint8_t opcode, const char * data, size_t len); +void sendWebSocketBinary(Poco::Net::StreamSocket & socket, const char * data, size_t len); +void sendWebSocketText(Poco::Net::StreamSocket & socket, const String & text); +void sendWebSocketClose(Poco::Net::StreamSocket & socket, uint16_t code, const String & reason); + +/// Read a single frame. `deadline_ns == 0` means no absolute deadline (per-read +/// timeouts are still governed by the socket's receive timeout). `max_payload_size` +/// caps the advertised payload length before any allocation. If +/// `completion_timeout_ns` is non-zero, it starts after the first byte arrives and +/// limits the remaining frame read without limiting idle time between frames. +WebSocketFrame readWebSocketFrame( + Poco::Net::StreamSocket & socket, + UInt64 deadline_ns = 0, + uint64_t max_payload_size = 16 * 1024 * 1024, + UInt64 completion_timeout_ns = 0); + +} diff --git a/src/Server/WebSocketSession.cpp b/src/Server/WebSocketSession.cpp new file mode 100644 index 000000000000..03fee33a6758 --- /dev/null +++ b/src/Server/WebSocketSession.cpp @@ -0,0 +1,1276 @@ +#include +#include + +#include + +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include + + +namespace DB +{ + +using namespace DB::WsProxy; + +namespace +{ + +/// A parsed `{"cmd":"insert",...}` control message (a streamed data INSERT). +struct InsertCommand +{ + bool is_insert = false; + String query; + String format; /// Input format for the client's streamed data; empty = use session default. +}; + +/// Detect and parse a streamed-insert control message WITHOUT touching SQL. Only +/// a top-level JSON object with `"cmd":"insert"` is an insert; anything else is a +/// plain query. (SQL never starts with '{', so the check is unambiguous.) +InsertCommand parseInsertCommand(const String & message) +{ + InsertCommand cmd; + const size_t first = message.find_first_not_of(" \t\r\n"); + if (first == String::npos || message[first] != '{') + return cmd; /// Not JSON -> a plain query. + try + { + Poco::JSON::Parser parser; + const auto obj = parser.parse(message).extract(); + if (obj->optValue("cmd", "") != "insert") + return cmd; + cmd.is_insert = true; + cmd.query = obj->optValue("query", ""); + cmd.format = obj->optValue("format", ""); + } + catch (...) + { + cmd.is_insert = false; /// Malformed -> treat as a plain query; the backend reports errors. + } + return cmd; +} + +/// The leading SQL keyword of a query (e.g. `SELECT`, `INSERT`, `CREATE`), +/// uppercased. Uses the SQL lexer so leading comments/whitespace are skipped +/// correctly. Clients frequently want this to drive their own logic without +/// re-implementing a tokenizer. Returns empty if there is no significant token. +String leadingVerb(const String & query) +{ + Lexer lexer(query.data(), query.data() + query.size()); + for (Token token = lexer.nextToken(); !token.isEnd() && !token.isError(); token = lexer.nextToken()) + { + if (!token.isSignificant()) + continue; + String verb(token.begin, token.size()); + for (char & c : verb) + if (c >= 'a' && c <= 'z') + c = static_cast(c - ('a' - 'A')); + return verb; + } + return ""; +} + +/// The proxy's classification of a query, reported to the client in ?parse=1 mode. +struct QueryClass +{ + String verb; /// Leading SQL keyword, uppercased. + bool streamed_insert = false; /// True if it is an INSERT that expects client-streamed data. + String insert_format; /// The FORMAT clause of such an INSERT (empty = session default). +}; + +/// OPT-IN only (?parse=1): parse the query so the proxy can (a) report the leading +/// verb + routing decision to the client and (b) auto-route a streamed-data INSERT +/// without the client sending an explicit {"cmd":"insert",...} envelope. +/// +/// A query needs the client-streamed data phase iff it is an `INSERT` with no +/// SELECT source, no INFILE, and no inline data (`INSERT INTO t [FORMAT X]` with +/// the rows still to come). Everything else (SELECT / INSERT-SELECT / inline +/// VALUES / DDL) is a plain query. +/// +/// This is the ONLY place the proxy parses SQL, and only when the client asks for +/// it. A parse failure is returned to the client instead of silently changing the +/// routing decision. +QueryClass classifyQuery(const String & query) +{ + QueryClass result; + result.verb = leadingVerb(query); + + ParserQuery parser(query.data() + query.size()); + ASTPtr ast = parseQuery( + parser, + query, + /* max_query_size */ 0, /// 0 = unlimited; the backend enforces the real limit. + /* max_parser_depth */ 1000, + /* max_parser_backtracks */ 1'000'000); + + if (const auto * insert = ast->as()) + { + const bool has_inline_data = insert->data != nullptr && insert->data != insert->end; + result.streamed_insert = !insert->select && !insert->infile && !has_inline_data; + result.insert_format = insert->format; + } + return result; +} + +constexpr UInt64 MAX_CLIENT_MESSAGE_SIZE = 16 * 1024 * 1024; +constexpr UInt64 MID_QUERY_FRAME_READ_TIMEOUT_NS = 1'000'000'000; +constexpr UInt64 CLIENT_FRAME_READ_TIMEOUT_NS = 30'000'000'000; + +bool isValidUTF8(const String & value) +{ + return UTF8::isValidUTF8(reinterpret_cast(value.data()), value.size()); +} + +/// Opt-in credit/window flow control for the SELECT push direction. The client +/// grants credit in whole frames (`{"cmd":"next","n":N}`) and can `pause`/`resume`; +/// the proxy sends a data frame only while `!paused && credit > 0`. This lets an +/// event-based JS client (which cannot apply receive backpressure) bound how much +/// it must buffer. `client_gone` latches a Close/broken read. +struct FlowControl +{ + bool enabled = false; + Int64 credit = 0; + bool paused = false; + std::atomic client_gone = false; + bool fragmented_message = false; + uint8_t fragmented_opcode = 0; + String fragmented_payload; +}; + +UInt64 monotonicNs() +{ + struct timespec ts + { + }; + clock_gettime(CLOCK_MONOTONIC, &ts); + return static_cast(ts.tv_sec) * 1'000'000'000ULL + static_cast(ts.tv_nsec); +} + +/// Apply one client control frame to the flow state (also answers pings). Used +/// both by the between-packets poll and by the credit gate, so `next`/`pause`/ +/// `resume`/close are handled consistently wherever the client frame is read. +/// `write_mutex` (if given) serializes the Pong send against other socket writers +/// (the parallel-format collector thread). +void applyControlFrame( + Poco::Net::StreamSocket & socket, const WebSocketFrame & frame, FlowControl & fc, std::mutex * write_mutex = nullptr) +{ + auto send_close = [&](uint16_t code, const String & reason) + { + fc.client_gone = true; + try + { + std::unique_lock lock; + if (write_mutex) + lock = std::unique_lock(*write_mutex); + sendWebSocketClose(socket, code, reason); + } + catch (...) + { + } + }; + + if (frame.protocol_error) + { + send_close(1002, "Protocol error"); + return; + } + if (frame.message_too_big) + { + send_close(1009, "Message too big"); + return; + } + if (frame.invalid_utf8) + { + send_close(1007, "Invalid UTF-8"); + return; + } + if (!frame.valid) + { + fc.client_gone = true; + return; + } + if (frame.opcode == Opcode::Close) + { + fc.client_gone = true; + try + { + std::unique_lock lock; + if (write_mutex) + lock = std::unique_lock(*write_mutex); + sendWebSocketFrame(socket, Opcode::Close, frame.payload.data(), frame.payload.size()); + } + catch (...) + { + } + return; + } + if (frame.opcode == Opcode::Ping) + { + try + { + std::unique_lock lock; + if (write_mutex) + lock = std::unique_lock(*write_mutex); + sendWebSocketFrame(socket, Opcode::Pong, frame.payload.data(), frame.payload.size()); + } + catch (...) + { + fc.client_gone = true; + } + return; + } + + uint8_t message_opcode = frame.opcode; + String message_payload; + if (frame.opcode == Opcode::Continuation) + { + if (!fc.fragmented_message) + { + send_close(1002, "Protocol error"); + return; + } + if (frame.payload.size() > MAX_CLIENT_MESSAGE_SIZE - fc.fragmented_payload.size()) + { + send_close(1009, "Message too big"); + return; + } + fc.fragmented_payload.append(frame.payload); + if (!frame.fin) + return; + + message_opcode = fc.fragmented_opcode; + message_payload = std::move(fc.fragmented_payload); + fc.fragmented_message = false; + fc.fragmented_opcode = 0; + } + else + { + if (fc.fragmented_message) + { + send_close(1002, "Protocol error"); + return; + } + if (!frame.fin) + { + fc.fragmented_message = true; + fc.fragmented_opcode = frame.opcode; + fc.fragmented_payload = frame.payload; + return; + } + message_payload = frame.payload; + } + + if (message_opcode != Opcode::Text) + return; /// Ignore stray binary messages during a SELECT. + if (!isValidUTF8(message_payload)) + { + send_close(1007, "Invalid UTF-8"); + return; + } + + bool next_command = false; + try + { + Poco::JSON::Parser parser; + const auto obj = parser.parse(message_payload).extract(); + const String cmd = obj->optValue("cmd", ""); + if (cmd == "next") + { + next_command = true; + const Int64 grant = obj->optValue("n", 0); + if (grant <= 0 || fc.credit < 0 || grant > std::numeric_limits::max() - fc.credit) + { + send_close(1008, "Invalid flow-control credit"); + return; + } + fc.credit += grant; + } + else if (cmd == "pause") + fc.paused = true; + else if (cmd == "resume") + fc.paused = false; + } + catch (...) + { + if (next_command) + { + send_close(1008, "Invalid flow-control credit"); + return; + } + /// Malformed control frame: ignore (the query stream is unaffected). + LOG_DEBUG(getLogger("WebSocketSession"), "Ignoring malformed control frame: {}", getCurrentExceptionMessage(false)); + } +} + +/// A `WriteBuffer` whose flushes are emitted as WebSocket binary frames. The +/// output format writes into it directly, so calling `flush` after each result +/// block streams that block to the client as its own frame (mid-query push). +/// +/// Send failures are swallowed and latched into `broken` rather than thrown: +/// once the client is gone there is nothing to do but stop, and a throwing +/// `nextImpl` would otherwise leave the buffer unfinalized (and could throw +/// from `finalize`). Callers check `isBroken` to notice the client left. +class WriteBufferToWebSocket : public BufferWithOwnMemory +{ +public: + explicit WriteBufferToWebSocket( + Poco::Net::StreamSocket & socket_, + FlowControl * flow_ = nullptr, + std::mutex * write_mutex_ = nullptr, + size_t size = DBMS_DEFAULT_BUFFER_SIZE) + : BufferWithOwnMemory(size) + , socket(socket_) + , flow(flow_) + , write_mutex(write_mutex_) + { + } + + ~WriteBufferToWebSocket() override = default; + + bool isBroken() const { return broken; } + +private: + void nextImpl() override + { + if (broken || offset() == 0) + return; + + /// Flow control: block this data frame until the client has granted credit + /// (and is not paused). While blocked we read the client's control frames + /// (next/pause/resume, or a Close), which also stops us reading the backend + /// -> TCP backpressure to the server, paced by the client's consumption. + if (flow && flow->enabled) + { + while (!flow->client_gone && (flow->paused || flow->credit <= 0)) + { + WebSocketFrame frame; + try + { + frame = readWebSocketFrame( + socket, /* deadline_ns */ 0, MAX_CLIENT_MESSAGE_SIZE, MID_QUERY_FRAME_READ_TIMEOUT_NS); + } + catch (...) + { + flow->client_gone = true; + break; + } + applyControlFrame(socket, frame, *flow, write_mutex); + } + if (flow->client_gone) + { + broken = true; + return; + } + } + + try + { + /// The send must be atomic against the session thread's control-frame + /// writes (used when parallel formatting runs this on a collector thread). + std::unique_lock lock; + if (write_mutex) + lock = std::unique_lock(*write_mutex); + if (flow && flow->client_gone) + { + broken = true; + return; + } + sendWebSocketBinary(socket, working_buffer.begin(), offset()); + } + catch (...) + { + broken = true; + return; + } + + if (flow && flow->enabled) + --flow->credit; + } + + Poco::Net::StreamSocket & socket; + FlowControl * flow; + std::mutex * write_mutex; + std::atomic broken = false; +}; + +/// A `ReadBuffer` that yields the payloads of incoming WebSocket binary frames, +/// used to feed the input format when parsing client-supplied INSERT data. +/// +/// End of data is a zero-length binary frame or a text frame (clean). A Close +/// frame or read error is an abort (client disconnected mid-insert): `wasAborted` +/// lets the caller cancel the backend query instead of committing partial data. +class ReadBufferFromWebSocket : public ReadBuffer +{ +public: + explicit ReadBufferFromWebSocket(Poco::Net::StreamSocket & socket_) + : ReadBuffer(nullptr, 0), socket(socket_) + { + } + + bool wasAborted() const { return aborted; } + +private: + bool fail(uint16_t code, const String & reason) + { + aborted = true; + try + { + sendWebSocketClose(socket, code, reason); + } + catch (...) + { + } + return false; + } + + bool nextImpl() override + { + /// `ReadBuffer::next` may call `nextImpl` again after a false return + /// (there is no permanent EOF latch in the base class), and some input + /// formats do a trailing read. Latch the end so we never block on a + /// frame that will not arrive. + if (finished || aborted) + return false; + + while (true) + { + WebSocketFrame frame; + try + { + frame = readWebSocketFrame( + socket, /* deadline_ns */ 0, MAX_CLIENT_MESSAGE_SIZE, CLIENT_FRAME_READ_TIMEOUT_NS); + } + catch (...) + { + aborted = true; + return false; + } + + if (frame.protocol_error) + return fail(1002, "Protocol error"); + if (frame.message_too_big) + return fail(1009, "Message too big"); + if (frame.invalid_utf8) + return fail(1007, "Invalid UTF-8"); + if (!frame.valid) + { + aborted = true; + return false; + } + if (frame.opcode == Opcode::Close) + { + aborted = true; + try + { + sendWebSocketFrame(socket, Opcode::Close, frame.payload.data(), frame.payload.size()); + } + catch (...) + { + } + return false; + } + if (frame.opcode == Opcode::Ping) + { + try + { + sendWebSocketFrame(socket, Opcode::Pong, frame.payload.data(), frame.payload.size()); + } + catch (...) + { + aborted = true; + return false; + } + continue; + } + + uint8_t message_opcode = frame.opcode; + String message_payload; + if (frame.opcode == Opcode::Continuation) + { + if (!fragmented_message) + return fail(1002, "Protocol error"); + if (frame.payload.size() > MAX_CLIENT_MESSAGE_SIZE - fragmented_payload.size()) + return fail(1009, "Message too big"); + fragmented_payload.append(frame.payload); + if (!frame.fin) + continue; + + message_opcode = fragmented_opcode; + message_payload = std::move(fragmented_payload); + fragmented_message = false; + fragmented_opcode = 0; + } + else + { + if (fragmented_message) + return fail(1002, "Protocol error"); + if (!frame.fin) + { + fragmented_message = true; + fragmented_opcode = frame.opcode; + fragmented_payload = frame.payload; + continue; + } + message_payload = std::move(frame.payload); + } + + if (message_opcode == Opcode::Text) + { + if (!isValidUTF8(message_payload)) + return fail(1007, "Invalid UTF-8"); + finished = true; /// Clean end-of-data control frame. + return false; + } + if (message_opcode != Opcode::Binary) + return fail(1002, "Protocol error"); + + /// An empty binary message is the clean end marker. + if (message_payload.empty()) + { + finished = true; + return false; + } + + current_frame = std::move(message_payload); + BufferBase::set(current_frame.data(), current_frame.size(), 0); + return true; + } + } + + Poco::Net::StreamSocket & socket; + String current_frame; + String fragmented_payload; + uint8_t fragmented_opcode = 0; + bool fragmented_message = false; + bool aborted = false; + bool finished = false; +}; + +/// Minimal JSON string escaping for the small control-frame payloads. +String escapeJSON(const String & s) +{ + String out; + out.reserve(s.size() + 2); + for (char c : s) + { + switch (c) + { + case '"': out += R"(\")"; break; + case '\\': out += R"(\\)"; break; + case '\n': out += R"(\n)"; break; + case '\r': out += R"(\r)"; break; + case '\t': out += R"(\t)"; break; + default: + if (static_cast(c) < 0x20) + { + static const char * hex = "0123456789abcdef"; + out += R"(\u00)"; + out += hex[(c >> 4) & 0xF]; + out += hex[c & 0xF]; + } + else + out += c; + } + } + return out; +} + +/// Report the proxy's parse decision to the client (only in ?parse=1 mode), as a +/// non-terminal control frame preceding the result. `kind` is the routing +/// decision: "insert" = the proxy will read client-streamed data; "query" = the +/// plain path (results / self-contained statement). `verb` is the leading keyword. +void sendQueryInfoFrame(Poco::Net::StreamSocket & socket, const String & verb, const String & kind) +{ + sendWebSocketText(socket, R"({"event":"query","kind":")" + kind + R"(","verb":")" + escapeJSON(verb) + R"("})"); +} + +} + +WebSocketSession::WebSocketSession( + Poco::Net::StreamSocket & socket_, + ContextPtr context_, + String format_, + String logs_level_, + bool flow_enabled_, + Int64 flow_initial_credit_, + bool parse_enabled_, + bool parallel_enabled_, + String compression_method_) + : socket(socket_) + , context(std::move(context_)) + , format(std::move(format_)) + , logs_level(std::move(logs_level_)) + , flow_enabled(flow_enabled_) + , flow_initial_credit(flow_initial_credit_) + , parse_enabled(parse_enabled_) + , parallel_enabled(parallel_enabled_) + , compression_method(std::move(compression_method_)) +{ +} + +void WebSocketSession::sendControlEvent(const String & event, const String & message) +{ + String json = R"({"event":")" + event + "\""; + if (!message.empty()) + json += R"(,"message":")" + escapeJSON(message) + "\""; + json += "}"; + std::lock_guard lock(ws_write_mutex); + sendWebSocketText(socket, json); +} + +void WebSocketSession::sendBlockEvent(const String & event, const Block & block) +{ + if (block.rows() == 0) + return; + + /// Serialize the block to JSONEachRow (one JSON object per row), then splice + /// the rows into a JSON array so the whole batch is one valid control frame: + /// {"event":"log","rows":[{...},{...}]}. Reusing the output format keeps this + /// correct for whatever columns/types the server sends. + WriteBufferFromOwnString buf; + auto out = FormatFactory::instance().getOutputFormat("JSONEachRow", buf, block.cloneEmpty(), context); + out->write(materializeBlock(block, !out->supportsSpecialSerializationKinds())); + out->finalize(); + const String & rows_text = buf.str(); + + String rows_array; + size_t start = 0; + bool first = true; + while (start < rows_text.size()) + { + const size_t newline = rows_text.find('\n', start); + const size_t end = (newline == String::npos) ? rows_text.size() : newline; + if (end > start) + { + if (!first) + rows_array += ','; + rows_array.append(rows_text, start, end - start); + first = false; + } + if (newline == String::npos) + break; + start = newline + 1; + } + + std::lock_guard lock(ws_write_mutex); + sendWebSocketText(socket, R"({"event":")" + event + R"(","rows":[)" + rows_array + "]}"); +} + +void WebSocketSession::sendBackendQuery(IServerConnection & connection, const String & query, bool with_pending_data) +{ + /// Copy the settings so we can opt into server log delivery. The backend + /// attaches its log queue based on `send_logs_level` in the query packet's + /// settings (a query-text SETTINGS clause is too late), so it must be set here. + Settings settings = context->getSettingsRef(); + if (!logs_level.empty()) + settings.set("send_logs_level", logs_level); + + /// The server compresses the result blocks it sends using the client's + /// `network_compression_method`. On a bandwidth-limited WAN this codec choice + /// dominates end-to-end time; ZSTD's higher ratio (columnar native compresses + /// far better than row JSON) makes the proxy beat gzipped HTTP. Empty = leave + /// the backend/connection default (LZ4). "none" disables block compression via + /// the connection's compression flag, so nothing to set here. + if (compression_method != "none" && !compression_method.empty()) + settings.set("network_compression_method", compression_method); + + auto timeouts = ConnectionTimeouts::getTCPTimeoutsWithoutFailover(settings); + + /// Pass no ClientInfo: a remote Connection then lets the server default the + /// query kind for a direct client, and an in-process LocalConnection derives + /// the query context (and current user) from its authenticated session rather + /// than being clobbered by an empty ClientInfo. + /// `with_pending_data` must be true for INSERTs so the connection enters the + /// send-data phase and replies with the sample/header block. + connection.sendQuery( + timeouts, + query, + /* query_parameters */ {}, + /* query_id */ "", + QueryProcessingStage::Complete, + &settings, + /* client_info */ nullptr, + with_pending_data, + /* external_roles */ {}, + /* process_progress_callback */ {}); +} + +void WebSocketSession::drainUntilEndOfStream(IServerConnection & connection) +{ + try + { + while (true) + { + Packet packet = connection.receivePacket(); + if (packet.type == Protocol::Server::EndOfStream || packet.type == Protocol::Server::Exception) + return; + } + } + catch (...) + { + /// The connection may be broken after a cancel; nothing more to drain. + LOG_DEBUG(getLogger("WebSocketSession"), "drain after cancel: {}", getCurrentExceptionMessage(false)); + } +} + +std::optional WebSocketSession::readClientMessage() +{ + String buffer; + bool in_fragmented_message = false; + uint8_t message_opcode = 0; + + auto send_close = [&](uint16_t code, const String & reason) + { + try + { + std::lock_guard lock(ws_write_mutex); + sendWebSocketClose(socket, code, reason); + } + catch (...) + { + } + }; + + while (true) + { + WebSocketFrame frame; + try + { + frame = readWebSocketFrame( + socket, /* deadline_ns */ 0, MAX_CLIENT_MESSAGE_SIZE, CLIENT_FRAME_READ_TIMEOUT_NS); + } + catch (...) + { + return std::nullopt; + } + + if (frame.protocol_error) + { + send_close(1002, "Protocol error"); + return std::nullopt; + } + if (frame.message_too_big) + { + send_close(1009, "Message too big"); + return std::nullopt; + } + if (frame.invalid_utf8) + { + send_close(1007, "Invalid UTF-8"); + return std::nullopt; + } + if (!frame.valid) + return std::nullopt; + + /// Control frames may interleave with data frames. + if (frame.opcode >= 0x08) + { + if (frame.opcode == Opcode::Close) + { + try + { + std::lock_guard lock(ws_write_mutex); + sendWebSocketFrame(socket, Opcode::Close, frame.payload.data(), frame.payload.size()); + } + catch (...) + { + } + return std::nullopt; + } + if (frame.opcode == Opcode::Ping) + { + try + { + std::lock_guard lock(ws_write_mutex); + sendWebSocketFrame(socket, Opcode::Pong, frame.payload.data(), frame.payload.size()); + } + catch (...) + { + return std::nullopt; + } + } + continue; + } + + if (frame.opcode != Opcode::Continuation) + { + if (in_fragmented_message) + { + send_close(1002, "Protocol error"); + return std::nullopt; /// New data frame during fragmentation. + } + buffer = std::move(frame.payload); + message_opcode = frame.opcode; + in_fragmented_message = !frame.fin; + } + else + { + if (!in_fragmented_message) + { + send_close(1002, "Protocol error"); + return std::nullopt; /// Continuation without a start. + } + if (frame.payload.size() > MAX_CLIENT_MESSAGE_SIZE - buffer.size()) + { + send_close(1009, "Message too big"); + return std::nullopt; + } + buffer.append(frame.payload); + if (frame.fin) + in_fragmented_message = false; + } + + if (frame.fin) + { + if (message_opcode == Opcode::Text && !isValidUTF8(buffer)) + { + send_close(1007, "Invalid UTF-8"); + return std::nullopt; + } + return buffer; + } + } +} + +bool WebSocketSession::executeSelect(IServerConnection & connection, const String & query) +{ + LoggerPtr log = getLogger("WebSocketSession"); + + sendBackendQuery(connection, query); + + FlowControl fc; + fc.enabled = flow_enabled; + fc.credit = flow_initial_credit; + + /// Parallel output formatting (opt-in, ?parallel=1) spreads the otherwise single-threaded + /// (~500 MB/s) format work across cores. It runs a collector thread that writes result frames, + /// so all frame sends are serialized behind `ws_write_mutex`. Trade-off: result frames are + /// coarser (blocks are batched, not one frame per block). It is incompatible with flow control + /// (whose credit gate reads the client socket, which must stay on the single session thread), + /// so flow control wins when both are requested. `getOutputFormatParallelIfPossible` still + /// honours the `output_format_parallel_formatting` setting and the format's own support. + const bool use_parallel = parallel_enabled && !flow_enabled; + + WriteBufferToWebSocket out_buf(socket, &fc, &ws_write_mutex); + OutputFormatPtr output; + bool cancelled = false; + bool client_gone = false; + + auto note_client_gone = [&] + { + client_gone = true; + if (!cancelled) + { + connection.sendCancel(); + cancelled = true; + } + }; + + /// Send a JSON control frame, tolerating a client that has already left. + auto try_control = [&](const String & event, const String & message) + { + if (client_gone) + return; + try + { + sendControlEvent(event, message); + } + catch (...) + { + note_client_gone(); + } + }; + + /// Server progress packets are incremental; accumulate the reads and track + /// the latest total estimate so each pushed event carries running totals. + UInt64 total_read_rows = 0; + UInt64 total_read_bytes = 0; + UInt64 total_rows_to_read = 0; + + /// Read any pending client frame (non-blocking): flow-control grants + /// (next/pause/resume) update the shared FlowControl; a Close/broken read means + /// the client is gone -> cancel the query and drain. + auto poll_client_frame = [&] + { + if (client_gone || !socket.poll(Poco::Timespan(0), Poco::Net::Socket::SELECT_READ)) + return; + WebSocketFrame frame; + try + { + frame = readWebSocketFrame(socket, monotonicNs() + MID_QUERY_FRAME_READ_TIMEOUT_NS); + } + catch (...) + { + frame.valid = false; + } + applyControlFrame(socket, frame, fc, &ws_write_mutex); + if (fc.client_gone) + note_client_gone(); + }; + + while (true) + { + if (!client_gone && fc.client_gone) + note_client_gone(); + if (!client_gone && out_buf.isBroken()) + note_client_gone(); + + /// Watch the client on every iteration, not only when the connection has no + /// data ready: both remote and local connections can continuously have + /// packets available while a client Close is pending. + poll_client_frame(); + + /// While no server data is pending, keep watching the client socket. + while (!connection.poll(50'000 /* microseconds */)) + { + if (!client_gone && out_buf.isBroken()) + note_client_gone(); + poll_client_frame(); + } + + if (!client_gone && fc.client_gone) + note_client_gone(); + + Packet packet = connection.receivePacket(); + switch (packet.type) + { + case Protocol::Server::Data: + { + if (client_gone) + break; /// Draining after cancel; discard. + if (!output) + { + auto & factory = FormatFactory::instance(); + const Block header = packet.block.cloneEmpty(); + output = use_parallel ? factory.getOutputFormatParallelIfPossible(format, out_buf, header, context) + : factory.getOutputFormat(format, out_buf, header, context); + } + if (packet.block.rows() > 0) + { + /// Materialize const/sparse/low-cardinality columns before writing: an + /// in-process LocalConnection hands blocks straight from the pipeline (e.g. + /// `SELECT 1` yields a ColumnConst), which row output formats mishandle. + /// Remote connections already materialize over the wire, so this is a cheap + /// no-op there. Mirrors ClientBase::onData. + output->write(materializeBlock(packet.block, !output->supportsSpecialSerializationKinds())); + output->flush(); /// Stream this block as its own frame(s). + if (out_buf.isBroken()) + note_client_gone(); /// Client left mid-stream. + } + break; + } + case Protocol::Server::Exception: + { + /// The result is incomplete, so discard buffered bytes rather than + /// finalizing the format (which would append a valid suffix). + output.reset(); + out_buf.cancel(); + const String message = packet.exception ? packet.exception->displayText() : "Unknown error from server"; + LOG_DEBUG(log, "Backend query exception: {}", message); + try_control("error", message); + return !client_gone; + } + case Protocol::Server::EndOfStream: + { + if (!client_gone && output) + { + output->finalize(); + out_buf.finalize(); + } + else + { + output.reset(); + out_buf.cancel(); + } + try_control(cancelled ? "cancelled" : "end", ""); + return !client_gone; + } + case Protocol::Server::Progress: + { + /// Mid-query push: forward running progress as a JSON text frame. + /// Text frames do not disturb the binary result stream. + if (!client_gone) + { + const auto values = packet.progress.getValues(); + total_read_rows += values.read_rows; + total_read_bytes += values.read_bytes; + if (values.total_rows_to_read) + total_rows_to_read = values.total_rows_to_read; + + const String json = R"({"event":"progress","read_rows":)" + std::to_string(total_read_rows) + + R"(,"read_bytes":)" + std::to_string(total_read_bytes) + + R"(,"total_rows_to_read":)" + std::to_string(total_rows_to_read) + "}"; + try + { + std::lock_guard lock(ws_write_mutex); + sendWebSocketText(socket, json); + } + catch (...) + { + note_client_gone(); + } + } + break; + } + case Protocol::Server::Log: + { + /// Server-side log lines for this query (client opts in with + /// `SETTINGS send_logs_level=...`). Push as a control frame. + if (!client_gone) + { + try + { + sendBlockEvent("log", packet.block); + } + catch (...) + { + note_client_gone(); + } + } + break; + } + case Protocol::Server::ProfileEvents: + { + /// Periodic profile-event counters emitted during execution. + if (!client_gone) + { + try + { + sendBlockEvent("profile_events", packet.block); + } + catch (...) + { + note_client_gone(); + } + } + break; + } + /// Not surfaced to the client yet (totals / extremes / profile info); + /// drain and continue. + case Protocol::Server::ProfileInfo: + case Protocol::Server::Totals: + case Protocol::Server::Extremes: + case Protocol::Server::TableColumns: + case Protocol::Server::PartUUIDs: + case Protocol::Server::TimezoneUpdate: + break; + default: + LOG_WARNING(log, "Unexpected packet {} from backend, ending session", packet.type); + output.reset(); + out_buf.cancel(); + try_control("error", "Unexpected packet from backend"); + return false; + } + } +} + +bool WebSocketSession::executeInsert(IServerConnection & connection, const String & query, const String & input_format) +{ + LoggerPtr log = getLogger("WebSocketSession"); + + /// Format the client's streamed data is in: the command's `format` if given, + /// else the session's `?format=`. The query text is forwarded verbatim (never + /// parsed); the backend receives native blocks via sendData regardless. + const String data_format = !input_format.empty() ? input_format : format; + + sendBackendQuery(connection, query, /* with_pending_data */ true); + + /// A remote server waits for external-tables data before replying with the + /// sample block; we have none, so send an empty set to unblock that native + /// handshake. An in-process LocalConnection has no such handshake (and does + /// not implement sendExternalTablesData), so skip it there. + if (connection.getConnectionType() != IServerConnection::Type::LOCAL) + { + ExternalTablesData external_tables_data; + connection.sendExternalTablesData(external_tables_data); + } + + /// The server replies with the sample block describing the target structure. + Block sample; + while (true) + { + Packet packet = connection.receivePacket(); + if (packet.type == Protocol::Server::Data) + { + sample = packet.block; + break; + } + if (packet.type == Protocol::Server::Exception) + { + const String message = packet.exception ? packet.exception->displayText() : "Unknown error from server"; + sendControlEvent("error", message); + return true; + } + if (packet.type == Protocol::Server::EndOfStream) + { + /// The server did not ask for data (e.g. the query carried its own source); + /// nothing to stream, we are done. + sendControlEvent("end", ""); + return true; + } + /// Ignore Log / Progress / TableColumns / TimezoneUpdate while waiting. + } + + /// The client streams the INSERT data as WebSocket binary frames. + auto data_in = std::make_unique(socket); + ReadBufferFromWebSocket * ws_in = data_in.get(); + + bool aborted = false; + try + { + auto source = context->getInputFormat(data_format, *data_in, sample, DEFAULT_BLOCK_SIZE); + Pipe pipe(source); + QueryPipeline pipeline(std::move(pipe)); + PullingPipelineExecutor executor(pipeline); + + Block block; + while (executor.pull(block)) + { + if (!block.empty()) + connection.sendData(block, /* name */ "", /* scalar */ false); + } + aborted = ws_in->wasAborted(); + } + catch (...) + { + /// Error parsing the client-supplied data: abort the insert. + const String message = getCurrentExceptionMessage(false); + LOG_DEBUG(log, "INSERT data error: {}", message); + connection.sendCancel(); + drainUntilEndOfStream(connection); + sendControlEvent("error", message); + return true; + } + + if (aborted) + { + /// Client disconnected mid-insert: cancel rather than commit partial data. + connection.sendCancel(); + drainUntilEndOfStream(connection); + return false; + } + + connection.sendData({}, "", false); /// Empty block signals end of data. + + while (true) + { + Packet packet = connection.receivePacket(); + switch (packet.type) + { + case Protocol::Server::EndOfStream: + sendControlEvent("end", ""); + return true; + case Protocol::Server::Exception: + { + const String message = packet.exception ? packet.exception->displayText() : "Unknown error from server"; + sendControlEvent("error", message); + return true; + } + default: + break; /// Ignore progress / log / profile events. + } + } +} + +void WebSocketSession::run(IServerConnection & connection) +{ + LoggerPtr log = getLogger("WebSocketSession"); + + /// The caller owns connection setup (and, for a remote transport, eager + /// authentication). `connection` is expected to be usable here. + LOG_DEBUG(log, "WebSocket session started"); + + while (true) + { + auto message = readClientMessage(); + if (!message) + break; + + if (message->empty()) + continue; + + try + { + /// Route by message kind, not by parsing SQL: a {"cmd":"insert",...} + /// control message is a streamed data INSERT; anything else is a plain + /// query (which covers SELECT / INSERT-SELECT / inline INSERT / DDL). + const InsertCommand ins = parseInsertCommand(*message); + bool keep_open = false; + if (ins.is_insert) + { + /// The client explicitly declared a streamed insert; honour it as-is. + keep_open = executeInsert(connection, ins.query, ins.format); + } + else if (parse_enabled) + { + /// Opt-in (?parse=1): parse the query to classify it, report the + /// decision to the client, and auto-route a streamed-data INSERT + /// without requiring the explicit control message. + const QueryClass qc = classifyQuery(*message); + sendQueryInfoFrame(socket, qc.verb, qc.streamed_insert ? "insert" : "query"); + keep_open = qc.streamed_insert ? executeInsert(connection, *message, qc.insert_format) + : executeSelect(connection, *message); + } + else + { + keep_open = executeSelect(connection, *message); + } + if (!keep_open) + break; + } + catch (...) + { + const String error_message = getCurrentExceptionMessage(false); + LOG_DEBUG(log, "Session error: {}", error_message); + try + { + sendControlEvent("error", error_message); + } + catch (...) + { + /// The socket may already be gone; nothing more we can do. + LOG_DEBUG(log, "Failed to deliver error event to client"); + } + break; + } + } + + LOG_DEBUG(log, "WebSocket session ended"); +} + +} diff --git a/src/Server/WebSocketSession.h b/src/Server/WebSocketSession.h new file mode 100644 index 000000000000..283c15cb593c --- /dev/null +++ b/src/Server/WebSocketSession.h @@ -0,0 +1,89 @@ +#pragma once + +#include +#include + +#include +#include + +#include + +namespace DB +{ + +class IServerConnection; + +/// Bridges one WebSocket session to a ClickHouse query connection, in both +/// directions, over the native `IServerConnection` abstraction. +/// +/// Transport-agnostic: the connection is injected into `run`, so the same bridge +/// serves the standalone edge proxy (a remote `Connection`) and an in-server +/// WebSocket endpoint (an in-process `LocalConnection`). For each query the client +/// sends as a text frame it runs `sendQuery` / `receivePacket` and streams the +/// result blocks back as binary frames (encoded with the chosen output format), +/// followed by a JSON control frame (`end` / `error` / `cancelled`). A Close frame +/// arriving mid-query triggers `IServerConnection::sendCancel`. +class WebSocketSession +{ +public: + WebSocketSession( + Poco::Net::StreamSocket & socket_, + ContextPtr context_, + String format_, + String logs_level_ = "", + bool flow_enabled_ = false, + Int64 flow_initial_credit_ = 0, + bool parse_enabled_ = false, + bool parallel_enabled_ = false, + String compression_method_ = ""); + + /// Drive the session loop over an already-connected `connection` until the + /// client closes or errors. The caller owns connection setup/teardown and + /// (for a remote transport) authentication. + void run(IServerConnection & connection); + +private: + /// Read one complete client message (reassembling fragments, answering pings). + /// Returns the message payload, or nullopt when the session should end + /// (Close frame, read error, or protocol violation). + std::optional readClientMessage(); + + /// Run a plain query and stream its result. Returns false if the session + /// should end afterwards (client closed mid-query). SQL is never parsed: + /// routing is driven by the client's message kind (a raw text frame is a + /// query -> executeSelect; a {"cmd":"insert",...} control message opts into + /// executeInsert). `with_pending_data=false` here, so SELECT / INSERT-SELECT / + /// inline INSERT / DDL all work and none waits for client data. + bool executeSelect(IServerConnection & connection, const String & query); + /// Stream a client-supplied data INSERT: send the query with `with_pending_data`, + /// parse the streamed frames with `input_format`, and `sendData` blocks. + bool executeInsert(IServerConnection & connection, const String & query, const String & input_format); + + void sendBackendQuery(IServerConnection & connection, const String & query, bool with_pending_data = false); + void drainUntilEndOfStream(IServerConnection & connection); + void sendControlEvent(const String & event, const String & message); + /// Serialize a Log / ProfileEvents block to a `{"event":...,"rows":[...]}` text frame. + void sendBlockEvent(const String & event, const Block & block); + + Poco::Net::StreamSocket & socket; + ContextPtr context; + String format; + String logs_level; /// If set, sent as `send_logs_level` so the backend pushes Log packets. + bool flow_enabled; /// Opt-in credit/window flow control for the SELECT push direction. + Int64 flow_initial_credit; /// Starting frame credit when flow control is enabled. + bool parse_enabled; /// Opt-in (?parse=1): parse SQL to auto-route inserts and report the query kind. + /// Opt-in (?parallel=1): format SELECT output on a thread pool for higher conversion + /// throughput, at the cost of coarser (batched) result frames. Ignored when flow control is on. + bool parallel_enabled; + /// Native-protocol codec the backend uses for result blocks (`network_compression_method`): + /// "lz4" / "zstd" for a remote transport, empty (or "none") to leave it unset (in-process + /// connections have no wire, so this is a no-op there). + String compression_method; + + /// Serializes complete WebSocket frame sends. With parallel output formatting the format's + /// collector thread writes result frames while the session thread pushes progress/log/control + /// frames; without this lock the two writers would interleave bytes and corrupt the stream. + std::mutex ws_write_mutex; +}; + +}