Add clickhouse-wsproxy: standalone WebSocket proxy (prototype)

.gitignore

@@ -20,6 +20,7 @@ ci/local.env
 .env
 /tests/integration/.env
 /ci/tmp
+/tmp
 /tests/venv
 /obj-x86_64-linux-gnu/
 

TODO.md

@@ -0,0 +1,112 @@
+# WebSocket query interface
+
+## Selected scope
+
+The upstream target for this branch is the in-server `ws_port` in
+`clickhouse-server`. It exposes the shared WebSocket query protocol through an
+in-process `LocalConnection` and keeps authentication, query execution, and format
+conversion inside the server.
+
+The standalone `clickhouse-wsproxy` binary remains in the branch as a compatible
+prototype and test peer. It exercises the same `WebSocketFrames` and
+`WebSocketSession` implementation through a remote `Connection`, but productionizing
+or deploying that binary is not part of the current upstream scope.
+
+Cloudflare deployment is analysis only. Nothing under that deployment design has
+been built; see `programs/wsproxy/CLOUDFLARE.md`.
+
+## Current implementation
+
+- `ws_port` is registered as a dedicated HTTP-upgrade listener in
+  `clickhouse-server`.
+- `WSHandler` authenticates the upgrade request, creates a server session and a
+  `LocalConnection`, then invokes `WebSocketSession::run`.
+- `WebSocketSession` supports query results, streamed inserts, progress, logs,
+  profile events, cancellation, output formats, optional SQL classification,
+  optional parallel formatting, and credit-based flow control.
+- `WebSocketFrames` implements the RFC 6455 frame layer shared by the server and
+  standalone proxy.
+- Browser upgrades with an `Origin` header are same-origin by default.
+  `ws_allowed_origins` configures a comma-separated allowlist for `ws_port`, and
+  `WSPROXY_ALLOWED_ORIGINS` provides the equivalent standalone-proxy setting.
+  Requests without `Origin` remain valid for non-browser clients.
+- Explicit Basic authentication fails closed when its header is malformed.
+- The initial `?flow=N` credit and later credit grants must be positive, fit in
+  `Int64`, and cannot overflow the accumulated credit.
+- Client messages have a cumulative 16 MiB limit across fragments. Mid-query
+  frames must complete within one second after the socket becomes readable.
+- Client write failures cancel the active query. The standalone proxy also bounds
+  stalled client writes with `WSPROXY_CLIENT_SEND_TIMEOUT_SEC`.
+- Protocol violations, invalid close frames, oversized messages, and invalid
+  flow-control grants use RFC-appropriate close handling.
+- The Node.js/Vitest suite runs the common protocol cases against both `ws_port`
+  and `clickhouse-wsproxy`. Adversarial cases cover fragmented message limits,
+  partial frames, interleaved control frames, invalid credit, origin policy, and
+  malformed credentials. Proxy-only cases cover the remote-backend boundary.
+
+`ws_port` currently speaks plaintext WebSocket (`ws://`). It does not provide a
+`ws_port_secure` listener. Deployments that expose the port outside a trusted
+network must terminate TLS at an ingress, load balancer, or other trusted proxy.
+
+## Protocol outline
+
+- A text frame containing SQL runs a normal query. Results are sent as binary
+  frames in the requested ClickHouse output format.
+- A text control message with `{"cmd":"insert", ...}` starts a streamed insert;
+  binary frames carry input data and an empty binary frame ends the input.
+- Text control frames report progress, logs, profile events, query classification,
+  and terminal `end`, `error`, or `cancelled` events.
+- Closing the WebSocket while a query is running cancels the query.
+- `?flow=N` enables frame-credit flow control. `?parallel=1` enables parallel
+  output formatting when flow control is disabled.
+
+The standalone proxy and `ws_port` intentionally use the same application
+protocol. Features that depend on a remote native-protocol hop, such as backend
+TLS and selecting its compression codec, apply only to `clickhouse-wsproxy`.
+
+## Work required before upstream review
+
+- Move the protocol coverage into repository-native ClickHouse CI. The current
+  Vitest suites are useful development coverage but are not yet CI gates.
+- Define general connection, session, memory, and concurrency limits for
+  `ws_port` and the standalone proxy.
+- Add graceful drain behavior so shutdown stops accepting new upgrades, cancels
+  or completes active work according to policy, and closes sessions predictably.
+- Add configuration documentation for `ws_port`, its plaintext-only transport,
+  origin policy, resource limits, and TLS-termination expectations.
+- Decide whether a native TLS listener is required for `ws_port`; until then,
+  document and validate the supported TLS-termination topology.
+- Replace the standalone proxy's development-only invalid-certificate mode with
+  explicit CA-based backend verification before any production deployment.
+- Decide whether the opt-in SQL classifier, parallel formatter, and application
+  flow-control extension belong in the first upstream version or a follow-up.
+
+## Deliberately out of scope
+
+- A production Cloudflare Workers/Containers deployment.
+- Production hardening of `clickhouse-wsproxy`, including `BaseDaemon`
+  integration and a configuration file.
+- A native TLS listener named `ws_port_secure`.
+- Backend connection pooling or replica load balancing. The standalone proxy
+  keeps one remote `Connection` per WebSocket session.
+
+## Development validation
+
+The test harness is documented in `programs/wsproxy/tests/README.md`. The primary
+command for the selected scope is:
+
+```bash
+cd programs/wsproxy/tests
+npm run test:server
+```
+
+The compatibility suite for the standalone prototype is:
+
+```bash
+cd programs/wsproxy/tests
+npm test
+```
+
+Historical benchmark notes remain under `programs/wsproxy/bench/`; they motivate
+the sidecar experiment but are not acceptance criteria for the in-server
+`ws_port`.

programs/CMakeLists.txt

@@ -25,6 +25,8 @@ option (ENABLE_CLICKHOUSE_KEEPER_CONVERTER "Util allows to convert ZooKeeper log
 
 option (ENABLE_CLICKHOUSE_KEEPER_CLIENT "ClickHouse Keeper Client" ${ENABLE_CLICKHOUSE_ALL})
 
+option (ENABLE_CLICKHOUSE_WSPROXY "Standalone WebSocket proxy (native protocol <-> WebSocket, edge format conversion)" ${ENABLE_CLICKHOUSE_ALL})
+
 if (NOT ENABLE_NURAFT)
     # RECONFIGURE_MESSAGE_LEVEL should not be used here,
     # since ENABLE_NURAFT is set to OFF for FreeBSD and Darwin.
@@ -126,6 +128,12 @@ if (ENABLE_CLICKHOUSE_KEEPER)
     add_subdirectory (keeper)
 endif()
 
+# Standalone binary with its own `main` and `add_executable` (see wsproxy/CMakeLists.txt),
+# so it is added here rather than via the multi-call `clickhouse_program_install` list below.
+if (ENABLE_CLICKHOUSE_WSPROXY)
+    add_subdirectory (wsproxy)
+endif()
+
 if (ENABLE_CLICKHOUSE_SELF_EXTRACTING AND NOT ENABLE_DUMMY_LAUNCHERS)
     add_subdirectory (self-extracting)
 endif ()

programs/server/Server.cpp

@@ -3653,6 +3653,29 @@ void Server::createServers(
             });
         }
 
+        if (server_type.shouldStart(ServerType::Type::WS))
+        {
+            /// WebSocket. An HTTP server whose handler upgrades the connection to a WebSocket and
+            /// bridges it to an in-process query connection (same wire protocol as clickhouse-wsproxy):
+            /// query as a text frame, results as binary frames in the ?format= output format, plus
+            /// mid-query progress/logs/profile-events and cancel-by-close.
+            port_name = "ws_port";
+            createServer(config, listen_host, port_name, listen_try, start_servers, servers, [&](UInt16 port) -> ProtocolServerAdapter
+            {
+                Poco::Net::ServerSocket socket;
+                auto address = socketBindListen(server_settings, socket, listen_host, port);
+                socket.setReceiveTimeout(settings[Setting::http_receive_timeout]);
+                socket.setSendTimeout(settings[Setting::http_send_timeout]);
+
+                return ProtocolServerAdapter(
+                    listen_host,
+                    port_name,
+                    "websocket: " + address.toString(),
+                    std::make_unique<HTTPServer>(
+                        httpContext(), createHandlerFactory(*this, config, async_metrics, "WSHandler-factory"), server_pool, socket, http_params, connection_filter, ProfileEvents::InterfaceHTTPReceiveBytes, ProfileEvents::InterfaceHTTPSendBytes));
+            });
+        }
+
         if (server_type.shouldStart(ServerType::Type::TCP))
         {
             /// TCP

programs/server/config.xml

@@ -182,6 +182,20 @@
     -->
     <tcp_port>9000</tcp_port>
 
+    <!-- Port for the WebSocket query interface. Clients open a WebSocket connection and send a
+         query as a text frame; results stream back as binary frames in the output format chosen
+         via the '?format=' URL parameter (default JSONEachRow), followed by a JSON control frame
+         ({"event":"end"} / "error" / "cancelled"). Mid-query progress, logs and profile events are
+         pushed as JSON frames, and closing the socket cancels the running query. This is the same
+         wire protocol as the standalone clickhouse-wsproxy sidecar. Disabled unless set. -->
+    <!-- <ws_port>9010</ws_port> -->
+
+    <!-- Browser WebSocket requests with an `Origin` header are same-origin by default. Set
+         `ws_allowed_origins` to a comma-separated list of full origins when a trusted browser
+         application is hosted elsewhere. Requests without `Origin` remain valid for non-browser
+         clients. -->
+    <!-- <ws_allowed_origins>https://app.example.com</ws_allowed_origins> -->
+
     <!-- Chunked capabilities for native protocol by server.
          Can be enabled separately for send and receive channels.
          Supported modes:

programs/wsproxy/CLOUDFLARE.md

@@ -0,0 +1,154 @@
+# Deploying `clickhouse-wsproxy` on Cloudflare (Containers + Workers)
+
+Feasibility notes and a deployment sketch for running the standalone proxy as a Cloudflare
+Container fronted by stateless Workers. **Status: analysis only — nothing here is built or included
+in the selected upstream scope.** The primary upstream target is the in-server `ws_port`; see
+`IN_SERVER.md`. The standalone `clickhouse-wsproxy` remains a compatible prototype, and its
+historical throughput and compression results are under `bench/`.
+
+## Verdict
+
+**The design appears feasible, but has not been validated by a deployment.** The proxy is already
+container-shaped: a single self-contained
+binary, configured entirely by `WSPROXY_*` env vars, holding **no local/disk state** (all session
+state is in-memory per connection), speaking WebSocket in and the native protocol out. Running it in
+a container needs essentially no code change; the work is the amd64 Linux build plus a thin
+Worker/Durable Object glue layer.
+
+It is arguably a *better* showcase of the value prop than a k8s sidecar, because the Worker gives you
+a programmable, DDoS-protected, auth-capable front door for free, and the compressed-native-to-region
+hop is exactly what the pitch is about.
+
+## Architecture: stateless Worker + stateful container
+
+The proxy's stateless/stateful split maps cleanly onto the platform:
+
+- **Worker = stateless front door / router.** Terminates the client WSS at the true edge, does
+  auth/routing, and forwards the upgrade to a container. Holds no session state.
+- **Container = the stateful proxy.** Runs the real binary, owns the per-session native `Connection`
+  and in-flight query, and does the format conversion. This *must* be a Container, not a Worker: the
+  value is the C++ `FormatFactory` + native protocol, which cannot realistically be reimplemented in
+  a Worker (JS/WASM) — that is the "clickhouse-cpp has no formats" problem again.
+- **Compressed hop to the region.** The container opens the outbound native + **ZSTD** connection to
+  ClickHouse Cloud (`:9440`). Converting at the container and keeping the ClickHouse-region leg
+  compressed is the whole point (see the compression findings in `bench/README.md`).
+
+```
+client  --WSS-->  Worker (true edge PoP)  --CF backbone-->  Container (regional)  --native+zstd (WAN)-->  ClickHouse Cloud
+                  stateless router                          stateful proxy = a Durable Object
+```
+
+### A Container *is* a Durable Object (verified)
+
+The `Container` class from the [`@cloudflare/containers`](https://github.com/cloudflare/containers)
+library **extends `DurableObject`**. Cloudflare's docs state it directly: *"a request passes through
+a Durable Object instance (the Container class extends a Durable Object class)."* Each container
+instance is backed by **exactly one** DO instance — the DO is the programmable sidecar that owns the
+container's lifecycle and routing; the container is the workload attached to it.
+
+Consequences:
+
+- Your container subclass gets a free per-instance place for coordination logic — warm-up, health,
+  draining, `alarm`-based idle handling, small storage — with no extra moving part.
+- The DO is single-threaded JS, but it is **not** in the per-frame data path: the client WebSocket
+  rides the container's exposed port via the stub's `fetch`; the DO only manages lifecycle. So
+  "DO is single-threaded" is **not** a throughput bottleneck for the proxy — as long as you forward
+  to the container port and do not proxy bytes through DO JS.
+- **A DO and its container may run in different locations** (placement is optimized for routing and
+  startup). This feeds the placement nuance below.
+
+## Running many proxy containers (scaling out)
+
+The proxy's sessions are **interchangeable** — each is just a fresh native `Connection`, a stateless
+pool — which is exactly the case the platform's helpers target:
+
+- `getRandom(env.WSPROXY, N)` — picks a random instance out of `N`. The stateless load balancer, and
+  the right fit here: each client WS upgrade → random container → new backend connection. The
+  WebSocket then naturally pins to that instance for its lifetime.
+- `getContainer(env.WSPROXY, name)` — addresses a *specific* named instance by ID (sticky/stateful
+  routing). Probably unneeded for the proxy, but available.
+- Multiple instances = multiple DO IDs; the library manages spin-up, pulling pre-fetched images at
+  other locations for fast cold starts.
+
+The Worker glue is small:
+
+```js
+import { getRandom } from "@cloudflare/containers";
+
+export default {
+  async fetch(request, env) {
+    // Optional: authenticate here, or let creds pass through to the container -> ClickHouse.
+    const instance = getRandom(env.WSPROXY, N); // N interchangeable proxy containers
+    return instance.fetch(request);             // forwards the WS upgrade to the container port
+  },
+};
+```
+
+**The catch — no autoscaling yet.** You pick and manage `N` yourself (a fixed fleet, or your own
+logic). Cloudflare says built-in autoscaling is planned but not available today. Capacity ≈
+`N × sessions-per-container`, so size `N` against a per-container session cap (see the thread-per-
+session note below). Until autoscaling lands, elasticity is on you — over-provision a fixed `N`, or
+build a small coordinator (itself a DO) that adjusts routing under load.
+
+## Concrete work items
+
+1. **x86_64 Linux build** *(the long pole).* Everything so far was built on macOS/arm64; Cloudflare
+   Containers are **amd64 Linux only**. ClickHouse builds amd64 Linux routinely (CI does), so this is
+   well-trodden — just not yet done for this binary.
+2. **Dockerfile.** A slim/distroless base + the ~306 MB binary + env; expose the listen port
+   (`WSPROXY_PORT`), set a `CMD`. Image ≈ 400-500 MB (under the platform's GB-scale image limits —
+   verify current limits).
+3. **Worker + Durable Object + `wrangler` config.** ~50-150 lines: accept the WS upgrade, `getRandom`
+   to a container, forward. Secrets (backend password) via Worker/container secrets → container env.
+4. **Secrets & egress.** Map `WSPROXY_BACKEND_*` into the container; settle ClickHouse-side source-IP
+   policy (see egress gotcha).
+
+No delivery estimate is asserted until the Linux image, Worker routing, current platform limits,
+and end-to-end WebSocket behavior have been validated.
+
+## Frictions / gotchas (Cloudflare-specific)
+
+- **Thread-per-session vs small containers *(biggest one)*.** The proxy is thread-per-session blocking
+  IO with default ~8 MB stacks → ~100 sessions ≈ ~800 MB just in stacks, and container instances are
+  memory-capped. Sessions-per-container is therefore bounded; lean on `getRandom` across instances
+  and/or shrink the thread stack size (a small proxy change worth doing before serious fan-in).
+- **CPU limits cap the conversion win.** Formatting is CPU-heavy and `?parallel` wants multiple cores,
+  but instance types meter vCPU. Right-size the instance; the parallel-formatting speedup is bounded
+  by the vCPU allotted.
+- **Egress IP allowlisting.** The container's outbound to ClickHouse Cloud uses Cloudflare's
+  shared/dynamic egress IPs. If the service restricts source IPs, you need CH-side allow-all or
+  Cloudflare egress ranges — settle this early (security posture).
+- **Cold starts.** Containers sleep on idle; the first connection pays container boot + proxy start +
+  backend handshake (seconds). Eager-connect adds to that.
+- **Placement affects the compression benefit.** Containers run in *regional* locations, not literally
+  every edge PoP, and may not co-locate with their DO. A container near the ClickHouse region → short
+  compressed hop (smaller win); near the user → long compressed hop (bigger win). Less placement
+  control than k8s.
+- **Keep the DO out of the per-frame data path** (forward to the container port; do not proxy frames
+  through DO JS).
+
+## Model shift: 1:1 sidecar → shared multi-tenant
+
+On k8s the proxy is a strict 1:1 sidecar (one proxy per app). On Cloudflare you would run a **shared
+multi-tenant** pool (many sessions per container, the Worker fanning out across instances). The proxy
+already supports this safely: it does **per-session credential pass-through** and keeps zero
+cross-session state, so multi-tenant is fine as long as each session carries its own credentials.
+
+## Alternative considered — pure Worker (no container)
+
+Workers can open outbound TCP (`connect` from `cloudflare:sockets`), so in principle the proxy could
+live entirely in a Worker. Rejected: it would require reimplementing the ClickHouse native protocol
+**and** the full format matrix in JS/WASM. Compiling ClickHouse itself to WASM is not feasible. The
+container is the right vehicle precisely because it runs the real C++ binary with full format
+coverage.
+
+## Sources
+
+- [Lifecycle of a Container — architecture](https://developers.cloudflare.com/containers/platform-details/architecture/)
+- [Scaling and Routing](https://developers.cloudflare.com/containers/platform-details/scaling-and-routing/)
+- [`cloudflare/containers` library README](https://github.com/cloudflare/containers/blob/main/README.md)
+- [Containers overview](https://developers.cloudflare.com/containers/)
+- [Containers coming to Workers (announcement)](https://blog.cloudflare.com/cloudflare-containers-coming-2025/)
+
+*(Cloudflare Containers are a recently-GA product; exact limits and APIs move — verify image-size,
+memory, vCPU, WS-to-container specifics, and egress ranges against the live docs before committing.)*