From d579efe48aaec67ce093dc9ff58b8dcc16d78a77 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 12 Aug 2026 15:09:03 -0400 Subject: [PATCH] docs: condense documentation prose to ~65% of original length The docs had accumulated redundancy, hedging, and verbose phrasing. This tightens prose across the site and README without changing what any page says: 9,153 words removed, 52 KB smaller. Every code fence is byte-identical to before, and every heading, table, list, link, and YAML frontmatter block is preserved. Identifier retention was checked mechanically per file -- inline code spans, link targets, and bare URLs present in the original are all present in the condensed version. Per-file prose ratios land between 62% and 77% for substantive pages. Four short stubs (404, sdk/pipes, sdk/reference, sdk/admin) compress less because they carry little redundancy to remove. --- README.md | 43 ++- docs/src/content/docs/404.md | 4 +- docs/src/content/docs/api.md | 363 ++++++++++----------- docs/src/content/docs/architecture.md | 100 +++--- docs/src/content/docs/claude-code.md | 188 ++++++----- docs/src/content/docs/deployment.md | 154 ++++----- docs/src/content/docs/development.md | 399 +++++++++++------------ docs/src/content/docs/durability.md | 74 ++--- docs/src/content/docs/getting-started.md | 62 ++-- docs/src/content/docs/ingest-pipeline.md | 231 +++---------- docs/src/content/docs/sdk/admin.md | 14 +- docs/src/content/docs/sdk/pipes.md | 10 +- docs/src/content/docs/sdk/queries.md | 55 ++-- docs/src/content/docs/sdk/reference.md | 24 +- docs/src/content/docs/sdk/streaming.md | 40 +-- docs/src/content/docs/why-wavehouse.md | 86 +++-- 16 files changed, 804 insertions(+), 1043 deletions(-) diff --git a/README.md b/README.md index 966bc683..02c21c1a 100644 --- a/README.md +++ b/README.md @@ -52,15 +52,15 @@ Full walkthrough → **[wavehouse.dev/getting-started](https://wavehouse.dev/get ## ✨ Why WaveHouse -ClickHouse is a phenomenal OLAP database, but pointing a frontend straight at it has sharp edges: one-row inserts trigger `Too many parts`, there's no backpressure or edge validation, no real-time push, and no row/column security. You end up building custom APIs, a Kafka queue, a batch consumer, a cache tier, and an auth service. **WaveHouse is that whole stack as one binary** — the only external dependency is ClickHouse. +Directly exposing ClickHouse to frontends causes `Too many parts` errors on single-row inserts and lacks backpressure, edge validation, real-time push, or row/column security. **WaveHouse replaces the need for custom APIs, Kafka queues, batch consumers, cache tiers, and auth services with one binary.** -If you're building user-facing analytics, WaveHouse is like **Supabase for ClickHouse** — or an **open-source Tinybird** that pushes data to the frontend in real time over SSE, not just pull-based REST. +It is an open-source alternative to Tinybird that pushes data via SSE. -- **Ingest** — async durable WAL (embedded NATS JetStream), `200 OK` instantly, background batch-flush; schema-validated against `system.columns`; optional ID-based dedup (idempotent ingest); dead-letter queue for failed inserts. -- **Query** — in-process Ristretto cache + `singleflight` coalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). -- **Real-time** — native SSE push, broadcast *before* the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients. -- **Security** — Hasura-style per-table, per-role column + row policies with JWT claim templating, stored in NATS KV. -- **Client** — `@wavehouse/sdk`: zero-dependency TypeScript client with query builder, live queries, streaming, and schema codegen. +- **Ingest**: Async durable WAL (NATS JetStream), instant `200 OK`, background batch-flush, schema validation via `system.columns`, idempotent ID-based dedup, and dead-letter queues. +- **Query**: Ristretto cache + `singleflight` coalescing, type-safe structured query AST, and parameterized SQL endpoints (named pipes). +- **Real-time**: Native SSE push broadcasting *before* ClickHouse flushes, with JetStream gap-fill for reconnecting clients. +- **Security**: Hasura-style per-table/role column and row policies using JWT claim templating, stored in NATS KV. +- **Client**: `@wavehouse/sdk` TypeScript client featuring a query builder, live queries, streaming, and schema codegen. ## 📊 How it compares @@ -74,11 +74,11 @@ If you're building user-facing analytics, WaveHouse is like **Supabase for Click | Row/column policies (JWT) | ✗ | custom | tokens only | ✓ Hasura-style | | Cost model | infra | infra + eng time | per-vCPU SaaS | infra only | -Full breakdown, failure modes, and the engineering rationale → **[wavehouse.dev/why-wavehouse](https://wavehouse.dev/why-wavehouse)**. +Details → **[wavehouse.dev/why-wavehouse](https://wavehouse.dev/why-wavehouse)**. ## 🛠️ Quick Start -Pick whichever fits — each ends with WaveHouse listening on `http://localhost:8080`. +All methods result in WaveHouse listening on `http://localhost:8080`. ### A. Docker Compose (recommended first run) @@ -87,7 +87,7 @@ git clone https://github.com/Wave-RF/WaveHouse.git && cd WaveHouse docker compose -f deployments/compose/standalone.yaml up -d ``` -The stack ships a permissive dev policy, so you can ingest without a token. Create a table in ClickHouse (Bring Your Own Schema), then ingest — see the [getting-started walkthrough](https://wavehouse.dev/getting-started) for the full ingest → query → stream tour. +Ships a permissive dev policy for tokenless ingest. See the [getting-started walkthrough](https://wavehouse.dev/getting-started). ### B. Prebuilt container image @@ -96,7 +96,7 @@ docker pull ghcr.io/wave-rf/wavehouse:latest # tagged release docker pull ghcr.io/wave-rf/wavehouse:dev # rolling main-branch build ``` -Both tags carry a signed [Sigstore](https://www.sigstore.dev/) build-provenance attestation — verify before you deploy: +Verify via [Sigstore](https://www.sigstore.dev/) provenance: ```bash gh attestation verify oci://ghcr.io/wave-rf/wavehouse:latest --repo Wave-RF/WaveHouse @@ -108,20 +108,17 @@ gh attestation verify oci://ghcr.io/wave-rf/wavehouse:latest --repo Wave-RF/Wave go install github.com/Wave-RF/WaveHouse/cmd/wavehouse@latest ``` -You'll still need ClickHouse reachable — point WaveHouse at it via `WH_CH_ADDR` (defaults to `localhost:9000`). -See [Configuration](https://wavehouse.dev/configuration). +Point to ClickHouse via `WH_CH_ADDR` (default `localhost:9000`). See [Configuration](https://wavehouse.dev/configuration). ## 🚦 Project status -WaveHouse is in **alpha** — built in the open, Apache-2.0-licensed, no vendor lock-in. See [SUPPORT.md](SUPPORT.md) for where to ask what, the alpha-stage response cadence (best-effort, 1–2 business days), and what's in vs. out of scope right now. +WaveHouse is in **alpha** (Apache-2.0). See [SUPPORT.md](SUPPORT.md) for scope and response cadence (1–2 business days). Track progress on the [**project board**](https://github.com/orgs/Wave-RF/projects/7). -Track what's shipped, in progress, and planned on the [**project board**](https://github.com/orgs/Wave-RF/projects/7). - -> **Alpha — expect change.** WaveHouse is pre-1.0: APIs, configuration, wire formats, and on-disk state can change between releases without a migration path, and some capabilities are still hardening. Pin a version and don't rely on stability guarantees until a tagged GA release. +> **Alpha — expect change.** APIs, configuration, wire formats, and on-disk state may change without migration paths. Pin versions until a GA release. ## 💻 Local Development -You'll need **Go 1.26+, GNU Make 4+, Docker (Compose v2), Node.js 22 LTS, and pnpm 11+**. See [development docs](https://wavehouse.dev/development) for the authoritative source of truth with the full list, version requirements, and gotchas. +Requires **Go 1.26+, GNU Make 4+, Docker (Compose v2), Node.js 22 LTS, and pnpm 11+**. See [development docs](https://wavehouse.dev/development). ```bash make tools # one-time bootstrap @@ -131,18 +128,18 @@ make dev # hot-reload on .go save ## 🤖 Working with Claude Code -> **AI-assisted, human-reviewed.** Much of WaveHouse — code and docs alike — is written with AI assistance ([Claude Code](https://claude.com/claude-code)). Every change, whether AI- or human-authored, goes through the same review gates, tests, and CI before it lands. We note it for transparency: treat the docs as the source of truth, and please [open an issue](https://github.com/Wave-RF/WaveHouse/issues) if anything reads as off or out of date. +WaveHouse is developed with AI assistance via [Claude Code](https://claude.com/claude-code). All changes undergo standard review, testing, and CI. Treat docs as the source of truth; [open an issue](https://github.com/Wave-RF/WaveHouse/issues) for inaccuracies. -The repo ships minimal team-wide [Claude Code](https://claude.com/claude-code) configuration — safety guardrails, a couple of slash commands / subagents, an auto-format hook, and [worktrunk](https://worktrunk.dev) project hooks for parallel agent workflows. Personal preferences (status line, model, allow lists) stay user-level. See [Claude Code & AI agents](docs/src/content/docs/claude-code.md) for setup + reference. `AGENTS.md` at the repo root is the canonical source of truth for project conventions. +The repo includes minimal team-wide configuration (guardrails, slash commands, auto-format hooks, and [worktrunk](https://worktrunk.dev) project hooks). See [Claude Code & AI agents](docs/src/content/docs/claude-code.md) and `AGENTS.md` for conventions. ## 🤝 Contributing -Issues, pull requests, and feedback welcome! See our [CONTRIBUTING.md](CONTRIBUTING.md) guidelines on how to structure your code and run the integration test suites. +Issues and PRs are welcome. Follow [CONTRIBUTING.md](CONTRIBUTING.md) for code structure and integration tests. ## 🛡️ Security -Found a vulnerability? **Don't open a public issue.** Email `security@wave-rf.com` per [SECURITY.md](SECURITY.md) — we acknowledge within 48 hours and aim for an initial assessment in 5 business days. +Email `security@wave-rf.com` per [SECURITY.md](SECURITY.md). We acknowledge within 48 hours and assess within 5 business days. Do not open public issues for vulnerabilities. ## 📜 License -WaveHouse is open source under the [Apache License 2.0](LICENSE). +Open source under the [Apache License 2.0](LICENSE). diff --git a/docs/src/content/docs/404.md b/docs/src/content/docs/404.md index 36e006ca..580259fd 100644 --- a/docs/src/content/docs/404.md +++ b/docs/src/content/docs/404.md @@ -38,12 +38,12 @@ head:

Signal lost

-

There's no page at this address — the link may be stale, or the page may have moved. One of these will get you back on the air:

+

This page doesn't exist—the link may be stale or moved. Use these to get back on the air:

-

Followed a link that should have worked? File an issue — broken links are bugs.

+

Link broken? File an issue.

diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index ee42e710..2e98c83f 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -9,27 +9,27 @@ Every HTTP endpoint WaveHouse exposes — ingest, query, streaming, schema intro ## Authentication -**There is no auth on/off switch** — the JWT middleware always runs. A request to `/v1/*` may include a JWT Bearer token: +**The JWT middleware always runs.** Requests to `/v1/*` may include a Bearer token: ```text Authorization: Bearer ``` -The JWT must use HMAC signing (HS256/HS384/HS512) or be validated via a JWKS endpoint (configured via `auth.jwks_url`). The accepted signing algorithm is pinned to the active verifier and checked *before* any key is consulted: an HMAC deployment accepts only `HS256`/`HS384`/`HS512`, and a JWKS deployment accepts only the asymmetric family (`RS256/384/512`, `ES256/384/512`, `PS256/384/512`, `EdDSA`). Tokens using `alg: none`, or an algorithm from the other family (e.g. an `HS256` token sent to a JWKS deployment), are rejected outright. +JWTs must use HMAC signing (HS256/HS384/HS512) or be validated via a JWKS endpoint (`auth.jwks_url`). The algorithm is pinned to the verifier and checked before key consultation: HMAC deployments accept only `HS256`/`HS384`/`HS512`; JWKS deployments accept only asymmetric families (`RS256/384/512`, `ES256/384/512`, `PS256/384/512`, `EdDSA`). Tokens using `alg: none` or the wrong family are rejected. -For SSE connections where custom headers are not possible, you can pass the token as a query parameter: +For SSE connections, tokens can be passed as a query parameter: ```text GET /v1/stream?token= ``` -The `Authorization` header takes precedence when both are provided: the `?token=` query parameter is only a fallback for clients that can't set headers (browser `EventSource`), so a token in the more log-leakable URL never overrides an explicit header credential. A `?token=` is stripped from the URL after extraction whichever credential wins, so it stays out of WaveHouse's own logs — but it has already crossed the wire in the request URI, so redact query strings at any proxy, CDN, or load balancer in front. +The `Authorization` header takes precedence; `?token=` is a fallback for clients like browser `EventSource`. To prevent log leaks, WaveHouse strips `?token=` from the URL after extraction, though you should still redact query strings at your proxy or CDN. -**Authentication is decoupled from authorization.** A request with **no token**, or an **invalid/expired/malformed** one, is *not* rejected outright — it falls back to an empty role that resolves to the policy `default_role`, and authorization is decided downstream. Because the bad-token reason is remembered, a request that is then denied for lacking permission fails loud (`401` "invalid/expired token") instead of a bare `403`. Elevated access requires a valid token whose role is granted (or equals the `admin_role`). A `403` body has two forms: a request that resolves to **no role at all** (no token and no `default_role` configured) returns `{"error":"forbidden: request has no role and no public default_role is configured"}`, while a request carrying a concrete-but-unauthorized role returns the bare `{"error":"forbidden"}` shown in the tables below. +**Authentication is decoupled from authorization.** Requests with no token, or an invalid/expired/malformed one, fall back to an empty role resolving to the `default_role` policy. If a request is later denied for lacking permission, it fails with `401` ("invalid/expired token") if the token was bad, rather than a bare `403`. Elevated access requires a valid token granted the required role (or the `admin_role`). A `403` body returns `{"error":"forbidden: request has no role and no public default_role is configured"}` if there is no role/default role, otherwise it returns `{"error":"forbidden"}`. -**Public (unauthenticated) access is driven by the policy.** Define a usable `default_role` and no-token requests are evaluated as that role (see [Roles & Access Control](#roles--access-control)); remove it and roleless requests are denied. Setting `default_role` equal to the `admin_role` is allowed — it makes every unauthenticated request admin (including `/v1/admin/*`), handy for local/dev — but it is logged loudly on every node that loads such a policy and must not be used in production. `/v1/admin/*` **and** the schema/DLQ endpoints are admin-only, and a pipe with **no `allowed_roles` authorizes nobody but the admin role** — but a pipe *can* be reached by the public when its `allowed_roles` lists the role the `default_role` resolves to (pipe access is plain allowlist membership, the same as any other role). +**Public access is policy-driven.** Define a `default_role` to allow unauthenticated requests (see [Roles & Access Control](#roles--access-control)); without it, roleless requests are denied. Setting `default_role` to `admin_role` grants all unauthenticated requests admin access (including `/v1/admin/*`); this is for local development only and must not be used in production. `/v1/admin/*` and schema/DLQ endpoints are admin-only. A pipe with no `allowed_roles` authorizes only the admin role, but public access is possible if `allowed_roles` includes the `default_role`. -**Operator key (non-JWT, break-glass).** A separate, role-free credential — `auth.operator_key` — authorizes a caller as a **full-access platform operator**: the entire data plane *and* the `/v1/admin/*` surface, without a JWT and independently of the token verifier. Present it in the standard `Authorization` header with the `Operator` scheme (forwarded verbatim by proxies, no collision with Bearer JWTs), or via the `X-Operator-Key` alias: +**Operator key (break-glass).** The `auth.operator_key` provides full-access platform operator privileges to the data plane and `/v1/admin/*` without a JWT. Use the `Operator` scheme or the `X-Operator-Key` alias: ```text Authorization: Operator @@ -37,58 +37,56 @@ Authorization: Operator X-Operator-Key: ``` -It is checked *before* the Bearer token (so it wins when both are present), compared in constant time, and — unlike a JWT bearing the `admin_role` — is honored **even when the policy is `nil`/deleted**, making it the only credential that can restore a wiped policy over HTTP. This deliberately bends "authentication is decoupled from authorization": a matching key both authenticates and authorizes in one step. It is disabled when empty (the default). See [Configuration — Authentication](/configuration#authentication) and [Access Control — Operator key](/access-control#operator-key). +Checked before Bearer tokens using constant-time comparison, this key is honored even if the policy is `nil`/deleted, allowing restoration of wiped policies via HTTP. It is disabled by default (empty). See [Configuration — Authentication](/configuration#authentication) and [Access Control — Operator key](/access-control#operator-key). ### Roles & Access Control -WaveHouse extracts the role from a configurable JWT claim path (`auth.role_claim`, default: `role`). Role handling: +WaveHouse extracts roles from a configurable JWT claim path (`auth.role_claim`, default: `role`). -- **`admin_role`** (policy field, `"admin"` by default, exact case-sensitive match) — Full access to all tables, raw SQL, and admin endpoints. There is no separate `service` role, though the non-JWT operator key (above) reaches the same surface without a token. -- **Other roles** — Access determined by the access control policy (see Admin endpoints below). +- **`admin_role`** (default `"admin"`, case-sensitive): Full access to all tables, raw SQL, and admin endpoints. +- **Other roles**: Access determined by the access control policy. -Policies support Hasura-style row-level and column-level permissions with JWT claim templating (e.g., `{{ jwt.app_metadata.tenant_id }}`). +Policies support Hasura-style row- and column-level permissions with JWT claim templating (e.g., `{{ jwt.app_metadata.tenant_id }}`). ## Response Format ### Error Responses -Error responses from WaveHouse carry a JSON body and the following headers: +WaveHouse error responses (4xx/5xx) include these headers: ```text Content-Type: application/json X-Content-Type-Options: nosniff ``` -The body is always a JSON object that includes an `error` field describing the failure: +The body is always a JSON object containing an `error` field: ```json {"error": "invalid json"} ``` -Some endpoints attach extra fields alongside `error` on their **failure** responses — e.g. a failing `/readyz` returns `{"status":"not ready","error":"…"}`. The guarantee is scoped to failures: whenever a response signals an error (any 4xx/5xx), an `error` field is present and parseable. Success responses carry each endpoint's own shape and need **not** include `error` — a healthy `/readyz` returns just `{"status":"ready"}`. +Some endpoints add extra fields; for example, a failing `/readyz` returns `{"status":"not ready","error":"…"}`. Success responses follow endpoint-specific shapes and may omit the `error` field (e.g., healthy `/readyz` returns `{"status":"ready"}`). -This contract holds for: +This contract applies to: -- Handler-emitted errors — validation (4xx), permission denials (403), not-found (404), backend errors (5xx). -- Router-level **404 Not Found** when the URL does not match any registered route. -- Router-level **405 Method Not Allowed** when the URL matches a route but the method is not registered. -- Server-level **500 Internal Server Error** when a handler panics — recovered, logged with stack, and reported to the client as JSON **when the handler has not yet committed any response headers or body bytes**. +- Handler errors: validation (4xx), permission denials (403), not-found (404), and backend errors (5xx). +- Router-level **404 Not Found** (unmatched URL). +- Router-level **405 Method Not Allowed** (unsupported method). +- Server-level **500 Internal Server Error** for recovered handler panics, provided no headers or body bytes were previously committed. -Historically some error paths defaulted to `text/plain` because they were emitted via `http.Error` or chi's default handlers; those paths now route through a shared `writeJSONError` helper so strict clients can branch on `Content-Type` consistently. - -The per-endpoint error tables below list the bodies you can expect for each status code; the `Content-Type` and `X-Content-Type-Options` headers above apply uniformly and are not repeated. +Paths that once defaulted to `text/plain` via `http.Error` or chi's defaults now route through a shared `writeJSONError` helper, so `Content-Type` is consistent. Per-endpoint tables below detail expected bodies per status code; the global headers apply uniformly. :::caution[Streaming / partial-write responses] -For SSE, streaming endpoints, or any handler that has already started writing the response, a later panic is recovered and logged server-side but no JSON 500 body is written — once headers are flushed, replacing them would corrupt the stream. Clients consuming streams should treat connection termination or truncated output as the failure signal in those cases. +For SSE, streaming endpoints, or handlers that have already started writing, later panics are logged server-side but no JSON 500 body is sent to avoid corrupting the stream. Clients should treat connection termination or truncated output as failure signals. ::: ## Endpoints ### `GET /livez` — Liveness Probe -> Canonical name (current Kubernetes convention — the kube-apiserver split that replaced the older conflated `/healthz`). Also served at **`/healthz`** (a permanent alias — the most widely-recognized name) and **`/health`** (a deprecated alias, scheduled for removal in v0.2.0). +> Canonical name (Kubernetes convention). Also served at **`/healthz`** (permanent alias) and **`/health`** (deprecated; removed in v0.2.0). -Returns `200 OK` once the gateway has discovered ClickHouse table schemas at least once. Returns `503 Service Unavailable` with a diagnostic body while the boot-time schema discovery retry loop is still running (ClickHouse unreachable, target database missing, etc.). No authentication required. +Returns `200 OK` after the gateway discovers ClickHouse table schemas once. Returns `503 Service Unavailable` with a diagnostic body while the boot-time schema discovery retry loop runs (e.g., ClickHouse unreachable, missing database). No authentication required. **Response (ready):** @@ -107,15 +105,13 @@ Returns `200 OK` once the gateway has discovered ClickHouse table schemas at lea Status code: `503 Service Unavailable` -The boot-degraded response lets an operator `curl /livez` to learn why the gateway isn't ready to serve traffic yet, instead of grepping a restart-loop log. The binary is bound on `:8080` and serves diagnostics, but is not yet accepting ingest/query traffic. Schema discovery retries with exponential backoff (2s → 60s); once a Refresh succeeds, `/livez` flips to `200` and stays there for the rest of the process lifetime — transient ClickHouse blips after that point are reflected in `/readyz`, not `/livez`. - ---- +The boot-degraded response allows operators to `curl /livez` for failure reasons instead of grepping logs. The binary binds on `:8080` for diagnostics before accepting traffic. Schema discovery retries with exponential backoff (2s → 60s); once successful, `/livez` stays `200`. Subsequent ClickHouse blips affect `/readyz`, not `/livez`. ### `GET /readyz` — Readiness Probe -> Canonical name (current Kubernetes convention). Also served at **`/ready`** — a deprecated alias kept for v0.1.x and scheduled for removal in v0.2.0. +Canonical name. **`/ready`** is a deprecated v0.1.x alias, removable in v0.2.0. -Returns `200 OK` if the process is fully booted (schema discovery complete) and ClickHouse is currently reachable. Returns `503 Service Unavailable` otherwise. No authentication required. +Returns `200 OK` if the process booted (schema discovery complete) and ClickHouse is reachable; otherwise `503 Service Unavailable`. No authentication required. **Response (ready):** @@ -133,7 +129,7 @@ Status code: `503 Service Unavailable` ### Liveness vs readiness — behavior matrix -`/livez` (liveness) and `/readyz` (readiness) answer different questions, so they diverge once the process has booted. `/livez` is **sticky**: after the first successful schema discovery it stays `200` for the rest of the process lifetime, even if ClickHouse later becomes unreachable — liveness asks "is the process alive and past boot," not "is its backend up right now." `/readyz` stays **conditional**: it pings ClickHouse on every call and drops back to `503` whenever ClickHouse is unreachable. +`/livez` and `/readyz` diverge after boot. `/livez` is **sticky**: after initial schema discovery, it remains `200` regardless of ClickHouse availability; it only confirms the process is alive and past boot. `/readyz` is **conditional**, pinging ClickHouse every call and returning `503` if unreachable. | State | `/livez` | `/readyz` | |----------------------------|:--------:|:---------:| @@ -142,21 +138,17 @@ Status code: `503 Service Unavailable` | Post-boot, ClickHouse dies | 200 ★ | 503 | | Post-boot, ClickHouse back | 200 | 200 | -★ Once boot completes, `/livez` no longer tracks ClickHouse state — a runtime ClickHouse outage surfaces in `/readyz` only. This is what keeps a Kubernetes `livenessProbe` from restart-looping the pod during a transient backend blip (see [Deployment → Boot-time degraded mode](/deployment#boot-time-degraded-mode)). - ---- +★ After boot, `/livez` ignores ClickHouse state; outages surface only in `/readyz`. This prevents Kubernetes `livenessProbe` restart-loops during transient backend blips (see [Deployment → Boot-time degraded mode](/deployment#boot-time-degraded-mode)). ### `GET /v1/health` — Liveness ping (public, content-free) -Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. - -This is what the SDK's `wh.sys.health()` calls, and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. +Returns **`200 OK`** when the gateway has booted, or **`503 Service Unavailable`** if boot-time schema discovery is failing. Both responses have empty bodies. No authentication required. ---- +The SDK's `wh.sys.health()` uses this endpoint for server selection in distributed setups. It mirrors `/livez` but exists as a `/v1` route because internal probes (`/livez`, `/readyz`, `/healthz`) may be filtered at the reverse proxy. This public API surface ensures reachability. It does **not** ping ClickHouse; readiness-based load balancing is handled by the proxy/LB via `/readyz`. ### `GET /version` — Build Info -Returns the build metadata embedded in the running binary — the `version`, `git_commit`, and `build_time` injected at compile time via `-ldflags`, plus the `go_version` read from the runtime. No authentication required: these are the same values logged at startup, so the endpoint discloses nothing the logs don't already. Useful for confirming exactly which build is deployed when troubleshooting. +Returns binary metadata: `version`, `git_commit`, and `build_time` (via `-ldflags`), plus the runtime `go_version`. No authentication required; these values are already logged at startup. Use this to confirm deployed builds during troubleshooting. **Response:** @@ -169,30 +161,28 @@ Returns the build metadata embedded in the running binary — the `version`, `gi } ``` -A binary built without the `-ldflags` injection (e.g. a bare `go build` rather than `make build`) reports the fallback values `"dev"` / `"unknown"` for `version` / `git_commit`. - ---- +Binaries built without `-ldflags` (e.g., `go build` instead of `make build`) report `"dev"` and `"unknown"` for `version` and `git_commit`. ### `POST /v1/ingest?table={table}` — Ingest Data -Accepts a single flat JSON object, a JSON array of objects, or a newline-delimited JSON (NDJSON) batch, validates each record against the ClickHouse schema for `{table}`, and publishes it to the message queue. Returns immediately — ClickHouse insertion happens asynchronously via the batch consumer. +Accepts a flat JSON object, JSON array of objects, or newline-delimited JSON (NDJSON) batch. Validates records against the ClickHouse schema for `{table}` and publishes them to a message queue. Returns immediately; insertion is asynchronous via the batch consumer. -**The format is auto-detected from the body — `Content-Type` is only a hint.** The first non-whitespace byte decides: `[` selects a JSON array, anything else a single JSON object. An explicit `Content-Type: application/x-ndjson` selects NDJSON line-framing *unless* the body starts with `[` (the array wins), so a batch works whether or not the header matches. +Format is auto-detected: `[` selects a JSON array; otherwise, it's a single JSON object. `Content-Type: application/x-ndjson` selects NDJSON unless the body starts with `[`. | Body | Typical `Content-Type` | Response | | ---- | ---------------------- | -------- | | one flat JSON object | `application/json` *(default)* | `{"ok":true}` (or `{"duplicate":true}`) | -| a JSON array of objects (any length, even 1) | `application/json` | per-record summary — see [Batch Ingest](#batch-ingest) | -| one JSON object per line (NDJSON) | `application/x-ndjson` | per-record summary — see [Batch Ingest](#batch-ingest) | +| a JSON array of objects | `application/json` | per-record summary — see [Batch Ingest](#batch-ingest) | +| NDJSON (one object per line) | `application/x-ndjson` | per-record summary — see [Batch Ingest](#batch-ingest) | -The inbound request body is capped at 16 MiB; a body over the cap is rejected with `413` (matching [`POST /v1/admin/query`](#post-v1adminquery--query-clickhouse)). For uploads larger than that, use the streaming NDJSON form below rather than one big body, and set your own outer limit at the [reverse proxy](/reverse-proxy#request-body-size-limits). +Request bodies are capped at 16 MiB; overflows return `413` (see [`POST /v1/admin/query`](#post-v1adminquery--query-clickhouse)). For larger uploads, use streaming NDJSON and configure the [reverse proxy](/reverse-proxy#request-body-size-limits). -The `{table}` URL query must match a table that exists in ClickHouse. WaveHouse discovers table schemas on startup and refreshes them periodically. +The `{table}` query must match an existing ClickHouse table. WaveHouse discovers schemas on startup and refreshes them periodically. :::note[Insert-only] -The ingest pipeline accepts only inserts. All other mutations — `DELETE`, `UPDATE`, `TRUNCATE`, `DROP`, `ALTER`, `REPLACE`, etc. — must be issued through [`POST /v1/admin/query`](#post-v1adminquery--query-clickhouse), which is restricted to the admin role (`admin_role`, the same gate as the rest of `/v1/admin/*`). +Only inserts are accepted here. All other mutations — `DELETE`, `UPDATE`, `TRUNCATE`, `DROP`, `ALTER`, `REPLACE`, etc. — must use [`POST /v1/admin/query`](#post-v1adminquery--query-clickhouse), restricted to the admin role (`admin_role`). -The policy engine authorizes mutations by inspecting the columns being written. That works for inserts but not for predicate-driven mutations like `DELETE … WHERE` — there's no way to prove the predicate matches only rows the caller is allowed to touch. Routing those statements through the admin-gated raw-SQL surface keeps the policy contract honest. +The policy engine authorizes inserts by inspecting columns. Predicate-driven mutations (e.g., `DELETE … WHERE`) are routed through the admin-gated raw-SQL surface because predicates cannot be proven to match only authorized rows. ::: **Request:** @@ -206,17 +196,24 @@ The policy engine authorizes mutations by inspecting the columns being written. } ``` -The body is a **flat JSON object** whose keys must match column names in the target ClickHouse table. Values must be type-compatible (see schema validation below). +The body must be a **flat JSON object** with keys matching ClickHouse column names and type-compatible values. **Schema Validation:** -- Unknown fields (not in the ClickHouse schema) are rejected. -- Type mismatches are rejected (e.g., sending a boolean for a `Float64` column). -- Missing required columns (non-nullable without a default) are rejected. -- Null values for non-nullable columns without a default are rejected. -- Type compatibility: `String` accepts JSON strings, numbers, and booleans (ClickHouse coerces the non-strings); `FixedString`/`UUID` accept the same at validation, but ClickHouse rejects a non-string value there, so it surfaces in the DLQ; `DateTime`/`Date`/`Enum` accept JSON strings or numbers; `IPv*` accepts JSON strings (a number passes validation but ClickHouse rejects it → DLQ); `Int*`/`Float*`/`Decimal` accept JSON numbers or strings — a string lets JavaScript callers avoid 64-bit precision loss, and its contents are ClickHouse's to judge (a non-numeric string is accepted here and surfaces in the DLQ, not as a `400`); `Bool` accepts JSON booleans and the numbers `0`/`1` (any other number, and *any* string — including `"true"` — passes validation but is rejected by ClickHouse → DLQ); `Array` accepts JSON arrays; `Map` accepts JSON objects; `Tuple` accepts JSON arrays or objects at validation, but ClickHouse takes an array only for an *unnamed* tuple and an object only for a *named* one — the other shape surfaces in the DLQ; any other ClickHouse type (`JSON`, `Variant`, `Dynamic`, geo, …) accepts any JSON value — WaveHouse defers to ClickHouse, so a bad value surfaces in the DLQ rather than as a `400`. -- `Nullable()` and `LowCardinality()` wrappers are handled transparently. -- Top-level `DateTime`/`DateTime64` values are rewritten to a canonical wire form on ingest — see [Timestamp canonicalization](#timestamp-canonicalization). +- Rejected: Unknown fields, type mismatches, missing required columns (non-nullable without default), or nulls in non-nullable columns without defaults. +- Type compatibility: + - `String`: JSON strings, numbers, booleans (coerced by ClickHouse). + - `FixedString`/`UUID`: Same as above at validation; non-strings are rejected by ClickHouse $\rightarrow$ DLQ. + - `DateTime`/`Date`/`Enum`: JSON strings or numbers. + - `IPv*`: JSON strings (numbers pass validation but fail in ClickHouse $\rightarrow$ DLQ). + - `Int*`/`Float*`/`Decimal`: JSON numbers or strings (strings prevent JS 64-bit precision loss; non-numeric strings $\rightarrow$ DLQ). + - `Bool`: JSON booleans, `0`, or `1` (others/strings pass validation but fail in ClickHouse $\rightarrow$ DLQ). + - `Array`: JSON arrays. + - `Map`: JSON objects. + - `Tuple`: JSON arrays (unnamed) or objects (named); opposite shapes $\rightarrow$ DLQ. + - Others (`JSON`, `Variant`, `Dynamic`, geo, …): any JSON value; failures surface in the DLQ. +- `Nullable()` and `LowCardinality()` are handled transparently. +- Top-level `DateTime`/`DateTime64` values use [Timestamp canonicalization](#timestamp-canonicalization). **Response (accepted):** @@ -224,7 +221,7 @@ The body is a **flat JSON object** whose keys must match column names in the tar {"ok": true} ``` -**Response (duplicate):** *(only when dedup is enabled)* +**Response (duplicate):** *(dedup enabled)* ```json {"duplicate": true} @@ -234,16 +231,16 @@ The body is a **flat JSON object** whose keys must match column names in the tar | Status | Body | Cause | | ------ | ---- | ----- | -| 400 | `{"error":"invalid json"}` | Malformed request body | -| 400 | `{"error":"unknown column ... for table ..."}` (also: `missing required column ...`, `type mismatch for column ...`, `null value for non-nullable column ...`) | Schema validation failure (unknown fields, type mismatches, missing required columns, null in a non-nullable column with no default). The body is the validator's message verbatim — there is no `validation failed:` prefix. | -| 400 | `{"error":"missing dedupe id field \"event_id\""}` | Only when dedupe is enabled with `dedupe.require_id: true` and the row lacks the configured `id_field`. With `require_id: false` (the default) the row is instead published un-deduped. Either way — reject or publish — the row is logged at `WARN` and counted by `wavehouse_ingest_dedupe_missing_id_total`. In a batch this is a per-record failure, not a whole-request error. | -| 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | A present-but-invalid/expired token was supplied and denied (the gate surfaces the token reason rather than silently falling back to `default_role`) | -| 403 | `{"error":"forbidden"}` (empty-role variant: `forbidden: request has no role and no public default_role is configured`) | The resolved role lacks `insert` on the table | -| 404 | `{"error":"unknown table: ..."}` | Table not found in ClickHouse schema | -| 413 | `{"error":"request body exceeded 16777216 bytes"}` | Request body over the 16 MiB cap | -| 500 | `{"error":"dedupe failed"}` | Deduplication backend error | -| 500 | `{"error":"publish failed"}` | Message queue error | -| 503 | `{"error":"service unavailable"}` | NATS JetStream stream full (backpressure). Response includes `Retry-After: 30` header. | +| 400 | `{"error":"invalid json"}` | Malformed body | +| 400 | `{"error":"..."}` | Schema failure (unknown column, missing required column, type mismatch, or null in non-nullable). Body is the validator's verbatim message. | +| 400 | `{"error":"missing dedupe id field \"event_id\""}` | Dedupe enabled with `dedupe.require_id: true` and row lacks `id_field`. If `false`, row is published un-deduped. Logged at `WARN`; counted by `wavehouse_ingest_dedupe_missing_id_total`. Per-record failure in batches. | +| 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | Invalid/expired token supplied. | +| 403 | `{"error":"forbidden"}` (or empty-role variant) | Role lacks `insert` permission on table. | +| 404 | `{"error":"unknown table: ..."}` | Table not found in schema. | +| 413 | `{"error":"request body exceeded 16777216 bytes"}` | Body over 16 MiB cap. | +| 500 | `{"error":"dedupe failed"}` | Deduplication backend error. | +| 500 | `{"error":"publish failed"}` | Message queue error. | +| 503 | `{"error":"service unavailable"}` | NATS JetStream stream full; includes `Retry-After: 30`. | **curl example:** @@ -255,45 +252,51 @@ curl -X POST "http://localhost:8080/v1/ingest?table=clicks" \ #### Timestamp canonicalization -**Send the canonical form — RFC 3339 UTC with the fraction already truncated to the column's precision and trailing zeros trimmed (spelled out below) — and the value is republished byte-for-byte.** A `DateTime`/`DateTime64` column value sent in any other accepted form — including RFC 3339 UTC with extra or trailing-zero fraction digits — is rewritten to that **one canonical wire form — RFC 3339 UTC** (`2026-06-21T04:00:00Z`, fraction truncated to the column's precision) before publishing, so the stored instant never changes but every consumer (the ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), the DLQ) sees the same spelling `/v1/query` renders. The accepted input forms: +**Send the canonical form—RFC 3339 UTC, fraction truncated to column precision and trailing zeros trimmed—and the value is republished byte-for-byte.** Other accepted forms are rewritten to this **one canonical wire form (RFC 3339 UTC)** (e.g., `2026-06-21T04:00:00Z`) before publishing. This ensures the stored instant remains unchanged while all consumers (ClickHouse insert, [SSE subscribers](#get-v1stream--server-sent-events-stream), DLQ, and `/v1/query`) see identical spelling. -- RFC 3339, any offset (`.`-fractions only — ClickHouse has no `,` separator). -- `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD`, zone-less — interpreted in the column's time zone, else the ClickHouse server's, exactly as ClickHouse itself would. -- A Unix-seconds string of exactly 9–10 digits (a `.fff` fraction is honored only for `DateTime64` columns, as ClickHouse does). -- A **non-negative integer** JSON number, unquoted — read the way ClickHouse reads bare numbers: Unix **seconds** for a `DateTime` column, but the column's raw **tick count** for `DateTime64` (a `DateTime64(3)` stores milliseconds, so `1750478400500` is the millisecond epoch `2025-06-21T04:00:00.5Z` — and `1750478400` is January 1970, not June 2025). +Accepted input forms: -**Fail-open**: a value in none of those forms is published verbatim — ClickHouse's more liberal parser decides insertability, and a value it too rejects surfaces via the DLQ, as before. `Date`/`Date32` columns pass through untouched. +- RFC 3339 with any offset (`.` fractions only). +- `YYYY-MM-DD[ T]HH:MM:SS[.fff]` or `YYYY-MM-DD` (zone-less): interpreted in the column's time zone, then the server's. +- Unix-seconds string of exactly 9–10 digits (`.fff` fraction honored only for `DateTime64`). +- **Non-negative integer** JSON number: read as Unix **seconds** for `DateTime`, or raw **tick count** for `DateTime64` (e.g., `1750478400500` is millisecond epoch `2025-06-21T04:00:00.5Z`). + +**Fail-open**: Values in other forms are published verbatim; ClickHouse's parser determines insertability. Rejected values surface via the DLQ. `Date`/`Date32` columns pass through untouched. :::note[Pass-through edge cases] -- Digit-strings of lengths other than 9–10 are ClickHouse's own forms — calendar shapes like `YYYYMMDD`, or its 13/16/19-digit ms/µs/ns epochs — and pass through untouched. -- A bare number with a fraction or exponent (`1750478400.5`) is passed through un-rewritten, and ClickHouse then fails the row for any timestamp column: it parses bare numbers as integers only — its lenient timestamp parsing, which accepts `"1750478400.5"` for a `DateTime64` column (on a plain `DateTime` the leftover fraction still fails the row), applies solely to quoted strings. -- An instant outside the column type's range also passes through — ClickHouse *saturates* out-of-range values spelling-dependently (and a `DateTime64(9)` column rejects the insert outright past the Int64-nanosecond ceiling, 2262-04-11 — a bound WaveHouse conservatively applies to every `DateTime64` of precision ≥ 7 when deciding what it may rewrite), so no rewrite there is safe. -- A time zone that doesn't resolve at runtime also causes pass-through: the binary embeds no tzdata, so named zones resolve from the runtime's zone database (the bundled distroless images ship one; a stripped-down custom runtime, or a server zone newer than the image's snapshot, may not resolve). An unresolvable *column* zone skips canonicalization for that column entirely; an unresolvable *server* default skips only zone-less values of columns without a declared zone — warned at schema refresh either way, and never guessed as UTC, which could move the stored instant. Remedy: install `tzdata` in a custom image, or point Go at a zone database via the `ZONEINFO` environment variable. -- Timestamps nested inside a composite column (`Array(DateTime)`, `Map(K, DateTime64)`, `Tuple(…, DateTime)`) pass through untouched; only top-level `DateTime`/`DateTime64` columns (including `Nullable`/`LowCardinality` wrappers) are canonicalized. -- The accepted grammar is differentially tested against a live ClickHouse: raw and canonicalized spellings must insert identically, or both fail. +- Digit-strings not 9–10 characters (e.g., `YYYYMMDD` or 13/16/19-digit epochs) pass through untouched. +- Bare numbers with fractions or exponents (`1750478400.5`) pass through; ClickHouse rejects these for timestamp columns as it parses bare numbers only as integers. +- Instants outside the column type's range pass through because ClickHouse saturates values spelling-dependently (e.g., `DateTime64(9)` rejects inserts past 2262-04-11). +- Unresolvable time zones cause pass-through: named zones resolve from the runtime database. If a column or server zone cannot resolve, canonicalization is skipped to avoid guessing UTC and shifting the instant. Remedy: install `tzdata` or set the `ZONEINFO` environment variable. +- Timestamps in composite columns (`Array`, `Map`, `Tuple`) pass through; only top-level `DateTime`/`DateTime64` (including `Nullable`/`LowCardinality`) are canonicalized. +- Grammar is differentially tested against live ClickHouse: raw and canonical spellings must insert identically or both fail. ::: :::caution[Upgrading WaveHouse against a pre-26.5 ClickHouse] -WaveHouse pins `date_time_input_format=best_effort` on its inserts — the ClickHouse server default since 26.5. On an older server whose default was `basic`, a plain `DateTime` column read an all-digit timestamp string of five or more digits as Unix seconds (shorter runs it rejected outright, where `best_effort` reads `"2026"` as a year); under `best_effort`, `"20260711"` stores 2026-07-11, not 1970-08-23, and some lengths (e.g. 12 digits) are rejected outright. (`DateTime64` columns diverge the same way on calendar-shaped runs — `"20260711"` is 1970-08-23 under `basic`, 2026-07-11 under `best_effort` — and additionally whenever an epoch run's unit doesn't match the column scale, e.g. a 16-digit microsecond epoch into a `DateTime64(3)`; an epoch run whose unit matches the column scale (a 13-digit millisecond epoch into a `DateTime64(3)`) reads identically too — only 9–10-digit Unix-seconds runs, with an optional fraction, agree at *every* scale.) The canonical form itself is what the pin rescues: under `basic` an RFC 3339 value's `Z` suffix is rejected outright (the row fails and lands in the DLQ), and the pin is what makes it insertable regardless of server version. Zone-less date-times and 9–10-digit Unix-seconds strings parse identically under both settings. +WaveHouse pins `date_time_input_format=best_effort` (the default since 26.5). On older servers using `basic`, all-digit strings $\ge$ 5 digits were read as Unix seconds; under `best_effort`, `"20260711"` is a date, not 1970-08-23. `DateTime64` columns diverge similarly on calendar shapes and mismatched epoch units (e.g., 16-digit $\mu$s into `DateTime64(3)`). The pin ensures RFC 3339 values with `Z` suffixes—which `basic` rejects—are insertable regardless of server version. ::: -**The canonical form, precisely.** This is the one strict timestamp spelling in WaveHouse — the same one `/v1/query` and `/v1/pipes/{name}` render for top-level timestamp columns and the SSE stream carries (the raw-SQL proxy `/v1/admin/query` instead renders server-side via `date_time_output_format=iso`, which keeps trailing fraction zeros), and the form the stream row-filter will require for timestamp comparisons once row-level enforcement lands ([#381](https://github.com/Wave-RF/WaveHouse/issues/381)): +**The canonical form, precisely.** This strict spelling is used by `/v1/query`, `/v1/pipes/{name}`, the SSE stream, and future row-filter comparisons ([#381](https://github.com/Wave-RF/WaveHouse/issues/381)): + +- `YYYY-MM-DDTHH:MM:SSZ` or `YYYY-MM-DDTHH:MM:SS.FZ`. Uppercase `T` and `Z`, always UTC, seconds always present. +- Fraction is **truncated** (not rounded) to column precision: `DateTime` has no fraction; `DateTime64(3)` has at most three digits. +- Trailing fractional zeros are trimmed and all-zero fractions dropped (Go's `time.RFC3339Nano`): `.120` $\to$ `.12Z`, `.000` $\to$ `Z`. +- Column time zones only affect *input* interpretation; output always ends in `Z`. -- `YYYY-MM-DDTHH:MM:SSZ`, or `YYYY-MM-DDTHH:MM:SS.FZ` when there is a sub-second part: uppercase `T` separator, uppercase `Z` suffix, always UTC — never a numeric offset — and seconds always present. -- The fraction is **truncated** (never rounded) to the column's precision: a `DateTime` column (whole seconds) never carries a fraction; a `DateTime64(3)` column carries at most three digits. -- Trailing fractional zeros are trimmed and an all-zero fraction is dropped (Go's `time.RFC3339Nano` rendering): `.120` becomes `.12Z`, `.000` becomes plain `Z` — byte-for-byte what `/v1/query` returns for the same stored value. -- A column's declared time zone changes only how zone-less *inputs* are interpreted, never the output: every canonical value ends in `Z`. +Examples for `DateTime64(3, 'America/New_York')`: -Examples for a `DateTime64(3, 'America/New_York')` column: `"2026-06-21 00:00:00.1239"` (zone-less, read in New York) → `"2026-06-21T04:00:00.123Z"`; `"1750478400.5"` (Unix-seconds string) → `"2025-06-21T04:00:00.5Z"`; the integer number `1750478400500` (ticks at the column's millisecond scale) → `"2025-06-21T04:00:00.5Z"`. +- `"2026-06-21 00:00:00.1239"` (zone-less) $\to$ `"2026-06-21T04:00:00.123Z"` +- `"1750478400.5"` (Unix-seconds string) $\to$ `"2025-06-21T04:00:00.5Z"` +- `1750478400500` (integer ticks) $\to$ `"2025-06-21T04:00:00.5Z"` #### Batch Ingest -A **JSON array** of objects (`[{…}, {…}]`) or an **NDJSON** body (`Content-Type: application/x-ndjson`, one JSON object per line) ingests a batch in a single request. Each record is validated, authorized, deduplicated, and published independently, so **one malformed or rejected record never blocks the rest of the batch**. (The SDK's `insert([...])` array helper uses the NDJSON form automatically; both forms return the same response.) +Ingest batches via a **JSON array** (`[{…}, {…}]`) or **NDJSON** body (`Content-Type: application/x-ndjson`, one object per line). Each record is validated, authorized, deduplicated, and published independently; one rejected record never blocks the batch. The SDK's `insert([...])` helper uses NDJSON automatically. -- **JSON array** — the most convenient form from most HTTP clients. A structural JSON syntax error fails the whole request (`400`), but a wrong-typed element (a non-object) is reported per-record like any other rejection. An explicit empty array (`[]`) is a valid, record-less batch (`200`, `total: 0`). -- **NDJSON** — the streaming-friendly form for very large uploads. Blank lines are skipped, and a single malformed *line* is reported and skipped (the newline reframes the next record). +- **JSON array**: Convenient for most clients. Structural syntax errors fail the whole request (`400`), but wrong-typed elements (non-objects) are reported per-record. An empty array (`[]`) is a valid batch (`200`, `total: 0`). +- **NDJSON**: Streaming-friendly for large uploads. Blank lines are skipped; malformed lines are reported and skipped. **Request (JSON array):** @@ -315,7 +318,7 @@ Content-Type: application/x-ndjson {"page": "/pricing", "score": 7} ``` -**Response (`200`):** a per-record summary. Each `results` entry mirrors the single-object response (`ok` / `duplicate` / `error`) plus its 1-based `index`. +**Response (`200`):** A per-record summary where each `results` entry mirrors the single-object response (`ok` / `duplicate` / `error`) plus its 1-based `index`. ```json { @@ -337,22 +340,22 @@ Content-Type: application/x-ndjson | `succeeded` | records validated and published | | `failed` | records rejected — see `results` | | `duplicates` | records skipped by dedup (when enabled) | -| `results` | per-record outcomes, each `{ index, ok\|duplicate\|error }` with `index` the 1-based record position. Truncated to the first 10,000 entries for very large batches (the counts stay authoritative). | +| `results` | per-record outcomes `{ index, ok\|duplicate\|error }`. Truncated to 10,000 entries for large batches; counts remain authoritative. | -A `200` is returned whenever the body was read and the records were processed — **even if every record failed**, so branch on `failed`/`results`, not the status code. Per-record problems (a malformed NDJSON line, a non-object array element, schema validation, column/check permission failures) are reported in `results` and the batch continues. Whole-request conditions abort with a non-`200` instead: +A `200` is returned if the body was read and processed—even if all records failed. Branch on `failed`/`results`, not status code. Per-record issues (malformed NDJSON lines, non-object array elements, schema/permission failures) are reported in `results`. Whole-request errors abort with non-`200`: | Status | Body | Cause | | ------ | ---- | ----- | -| 400 | `{"error":"empty body"}` / `{"error":"empty ndjson body"}` | The body has no records | -| 400 | `{"error":"invalid json: ..."}` | A structural JSON syntax error, or a truncated/unterminated JSON array (e.g. a cut-off upload — the whole request fails rather than reporting a partial success), or an oversized NDJSON line | -| 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | A present-but-invalid/expired token was supplied and denied (same auth gate as the single-object path; surfaces the token reason) | -| 403 | `{"error":"forbidden"}` (empty-role variant: `forbidden: request has no role and no public default_role is configured`) | The resolved role lacks `insert` on the table (checked once, before any record) | -| 413 | `{"error":"request body exceeded 16777216 bytes"}` | Request body over the 16 MiB cap | -| 500 | `{"error":"publish failed"}` / `{"error":"dedupe failed"}` | Message-queue or dedup-backend failure mid-batch | -| 503 | `{"error":"service unavailable"}` | NATS JetStream full (backpressure) mid-batch; includes `Retry-After: 30` | +| 400 | `{"error":"empty body"}` / `{"error":"empty ndjson body"}` | No records in body | +| 400 | `{"error":"invalid json: ..."}` | Structural JSON error, truncated array, or oversized NDJSON line | +| 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | Invalid/expired token | +| 403 | `{"error":"forbidden"}` (empty-role variant: `forbidden: request has no role and no public default_role is configured`) | Role lacks `insert` on table | +| 413 | `{"error":"request body exceeded 16777216 bytes"}` | Body over 16 MiB cap | +| 500 | `{"error":"publish failed"}` / `{"error":"dedupe failed"}` | Queue or dedup-backend failure mid-batch | +| 503 | `{"error":"service unavailable"}` | NATS JetStream full; includes `Retry-After: 30` | :::caution[At-least-once on retry] -A batch aborted partway (a `503`/`500`, or a JSON-array syntax error, after some leading records were already published) re-publishes those leading records when the whole batch is retried. Enable deduplication if duplicate suppression matters — this is the same at-least-once property the single-object path already has (the SDK retries both on `503`). +Batches aborted partway (`503`/`500`, or JSON syntax errors after some records published) re-publish leading records upon retry. Enable deduplication if duplicate suppression is required; the SDK retries both single and batch paths on `503`. ::: **curl example (JSON array):** @@ -371,27 +374,25 @@ curl -X POST "http://localhost:8080/v1/ingest?table=clicks" \ --data-binary $'{"page":"/home"}\n{"page":"/about"}\n' ``` ---- - ### `POST /v1/admin/query` — Query ClickHouse -Executes a SQL statement directly against ClickHouse. **WaveHouse proxies the SQL string verbatim to ClickHouse's HTTP interface** — any statement ClickHouse accepts works, including arbitrary DDL/DML/SYSTEM verbs and inline FORMAT directives. Multi-statement input (`SELECT 1; TRUNCATE t`) also works on recent ClickHouse versions where multi-query is enabled by default; older or restrictively-configured servers may reject the second statement with a clear error. Read queries return a JSON array of result rows; mutations/DDL return HTTP 200 with `[]` on success. DateTime columns are ISO-8601 formatted via the upstream `date_time_output_format=iso` setting — server-side rendering that keeps trailing fraction zeros, so a `DateTime64(3)` whole-second value returns `.000Z` here where `/v1/query` renders plain `Z`; other types are returned as ClickHouse renders them under `FORMAT JSON`. +Executes a SQL statement directly against ClickHouse. **WaveHouse proxies the SQL string verbatim to ClickHouse's HTTP interface**; any statement ClickHouse accepts works, including DDL/DML/SYSTEM verbs and inline FORMAT directives. Multi-statement input (`SELECT 1; TRUNCATE t`) works on recent ClickHouse versions where multi-query is enabled by default; older servers may reject the second statement. Read queries return a JSON array of rows; mutations/DDL return HTTP 200 with `[]`. DateTime columns use ISO-8601 via the upstream `date_time_output_format=iso` setting, preserving trailing fraction zeros (e.g., `DateTime64(3)` returns `.000Z` where `/v1/query` renders plain `Z`). Other types follow `FORMAT JSON`. :::note[Inline `FORMAT` overrides the JSON envelope] -ClickHouse's inline `FORMAT` clause (e.g. `SELECT 1 FORMAT CSV` or `… FORMAT Pretty`) takes precedence over the URL-level `default_format=JSON` setting. When the SQL contains an explicit `FORMAT`, the proxy forwards ClickHouse's raw response body (CSV, Pretty, TSV, …) and passes through the upstream `Content-Type` header — `text/csv`, `text/tab-separated-values`, etc. — so consumers see the right MIME type. The "extract the `data` array" behavior only applies when ClickHouse returned the `FORMAT JSON` envelope, which is the default. +ClickHouse's inline `FORMAT` clause (e.g. `SELECT 1 FORMAT CSV` or `… FORMAT Pretty`) takes precedence over the URL-level `default_format=JSON`. The proxy forwards ClickHouse's raw body (CSV, Pretty, TSV, …) and the upstream `Content-Type` — `text/csv`, `text/tab-separated-values`, etc. The "extract the `data` array" behavior only applies when ClickHouse returns the default `FORMAT JSON` envelope. ::: :::caution[64 MiB response cap] -The proxy buffers the upstream response in memory before forwarding (no row-streaming yet), so a `SELECT *` from a large table can pin RAM on the API server. To avoid an admin OOMing themselves, responses larger than 64 MiB return 502 with a `clickhouse response exceeded N bytes` error. Narrow the query with `LIMIT`, or use a streaming client outside WaveHouse that talks to ClickHouse directly (the standard escape hatch — the same admin credentials work). +The proxy buffers responses in memory; large `SELECT *` queries can pin RAM. Responses exceeding 64 MiB return 502 with a `clickhouse response exceeded N bytes` error. Use `LIMIT` or a streaming client talking to ClickHouse directly (using the same admin credentials). ::: -This endpoint **does not cache, does not singleflight, and emits `Cache-Control: no-store`** — every request goes straight to ClickHouse, mutation or read, and downstream HTTP caches are explicitly told not to store the response. Raw SQL is an admin escape hatch with infrequent, ad-hoc traffic, so the L1/singleflight machinery would only add complexity without a real hit-rate win. Use [`POST /v1/query?table={table}`](#post-v1querytabletable--structured-query) or [`GET/POST /v1/pipes/{name}`](#getpost-v1pipesname--execute-named-pipe) for the cached read paths (dashboards, high-QPS clients, etc.) — both share an in-process L1 (Ristretto) with singleflight coalescing. +This endpoint **does not cache, does not singleflight, and emits `Cache-Control: no-store`**. Every request goes straight to ClickHouse. For cached read paths (dashboards, high-QPS clients), use [`POST /v1/query?table={table}`](#post-v1querytabletable--structured-query) or [`GET/POST /v1/pipes/{name}`](#getpost-v1pipesname--execute-named-pipe), which share an in-process L1 (Ristretto) with singleflight coalescing. :::note[Admin only] -The route is mounted under `/v1/admin/*`, behind the `RequireAdmin` gate: only a caller whose JWT role equals the policy `admin_role` (`"admin"` by default) may use it. A request with no/invalid token resolves to the `default_role` (not the admin role unless `default_role` is deliberately set to it — a loudly-warned dev-only setting) and is rejected. Raw SQL has no per-statement scope check (a full SQL parser would be needed to authorize predicates), so the role gate is the entire authorization story, shared with the rest of `/v1/admin/*` (policy CRUD, pipes CRUD). The normal surfaces for non-admin callers are `POST /v1/ingest?table={table}` for writes, `POST /v1/query?table={table}` for structured reads, and `GET/POST /v1/pipes/{name}` for pre-defined queries — none of which expose raw SQL. +Mounted under `/v1/admin/*` behind the `RequireAdmin` gate: only callers with a JWT role matching the policy `admin_role` (`"admin"` by default) may use it. Requests with no/invalid tokens resolve to `default_role` and are rejected. Raw SQL has no per-statement scope check; the role gate is the sole authorization mechanism, shared with `/v1/admin/*` (policy/pipes CRUD). Non-admins should use `POST /v1/ingest?table={table}` for writes, `POST /v1/query?table={table}` for structured reads, or `GET/POST /v1/pipes/{name}` for pre-defined queries. ::: -`/v1/admin/query` is the only sanctioned surface for non-insert mutations (the ingest pipeline is insert-only). Granting raw-SQL access to a non-admin role via the policy engine is no longer supported: authenticate with the admin role (`admin_role`). +`/v1/admin/query` is the only sanctioned surface for non-insert mutations. Granting raw-SQL access to non-admin roles via the policy engine is unsupported; authenticate with the `admin_role`. **Request:** @@ -406,7 +407,7 @@ The route is mounted under `/v1/admin/*`, behind the `RequireAdmin` gate: only a | `sql` | string | Yes | SQL forwarded verbatim to ClickHouse's HTTP interface. | :::note[No parameter binding on this endpoint (yet)] -The earlier handler accepted a `params` array bound to `?` placeholders; the HTTP proxy doesn't. ClickHouse's native named-param syntax (`WHERE id = {id:UInt32}` with `param_id=42` on the URL query string) is *not* forwarded today either — the proxy only sets `default_format`, `date_time_output_format`, and `database` on the upstream URL, and the request body is `{"sql": "..."}` with no escape hatch for query-string params. The current contract is "send raw SQL, get rows back": inline literals into the SQL for now. For safe binding from user-supplied input, use the structured query endpoint (`POST /v1/query?table={table}`) — that's its job. +The proxy does not support `params` arrays or ClickHouse native named-param syntax (`WHERE id = {id:UInt32}` with `param_id=42` on the query string). The proxy only sets `default_format`, `date_time_output_format`, and `database` on the upstream URL. Use inline literals for now, or use the structured query endpoint (`POST /v1/query?table={table}`) for safe binding from user input. ::: **Response:** @@ -428,12 +429,12 @@ The earlier handler accepted a `params` array bound to `?` placeholders; the HTT | ------ | ---- | ----- | | 400 | `{"error":"invalid json"}` | Malformed request body | | 400 | `{"error":"missing sql"}` | Missing `sql` field | -| 400 | `{"error":""}` | ClickHouse rejected the statement with a 4xx (bad SQL, missing table, type error, …). The body carries ClickHouse's own error text verbatim, e.g. `Code: 60. DB::Exception: Table default.x doesn't exist.`. The proxy maps any ClickHouse 4xx to HTTP 400 — caller-fault, the request itself is what's wrong. | -| 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | The request carried a present-but-invalid/expired token and was denied for lacking permission (the gate surfaces the token reason) | -| 403 | `{"error":"forbidden"}` | Caller's role is not the policy `admin_role` (`"admin"` by default) | -| 502 | `{"error":""}` | ClickHouse returned a 5xx (internal error, overloaded, etc.). The proxy maps any ClickHouse 5xx to HTTP 502 — gateway-fault, the upstream service had a problem. Same body convention: ClickHouse's text is forwarded as-is. | -| 502 | `{"error":"clickhouse request failed: ..."}` | Transport-level failure reaching ClickHouse (connection refused, timeout, the upstream went away mid-request) | -| 502 | `{"error":"clickhouse response exceeded N bytes; ..."}` | Response body exceeded the 64 MiB memory-safety cap. Narrow the query, add a `LIMIT`, or use `FORMAT JSONEachRow` with a streaming client outside WaveHouse. | +| 400 | `{"error":""}` | ClickHouse rejected the statement (4xx). Body carries ClickHouse's error text verbatim, e.g. `Code: 60. DB::Exception: Table default.x doesn't exist.` | +| 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | Invalid/expired token | +| 403 | `{"error":"forbidden"}` | Caller's role is not the policy `admin_role` | +| 502 | `{"error":""}` | ClickHouse returned a 5xx. Body contains verbatim ClickHouse error text. | +| 502 | `{"error":"clickhouse request failed: ..."}` | Transport-level failure reaching ClickHouse | +| 502 | `{"error":"clickhouse response exceeded N bytes; ..."}` | Response body exceeded the 64 MiB cap. Use `LIMIT` or `FORMAT JSONEachRow` via a direct streaming client. | **curl example:** @@ -445,14 +446,12 @@ curl -X POST http://localhost:8080/v1/admin/query \ -d '{"sql": "SELECT * FROM clicks LIMIT 10"}' ``` ---- - ### `POST /v1/query?table={table}` — Structured Query -Executes a type-safe structured query against a table. The query AST is validated against the schema and converted to parameterized SQL. Permissions from the access control policy are enforced (column filtering, row-level security, aggregation restrictions). +Executes a type-safe structured query against a table. The AST is validated against the schema and converted to parameterized SQL, enforcing access control policies (column filtering, row-level security, aggregation restrictions). :::note[The column allowlist is a hard cap on every clause] -Every column the query references — in `columns`, an aggregation argument, `filters`, `group_by`, `order_by`, or `time_range` — must be permitted by the role's `allow_columns`/`deny_columns`, or the request is rejected with `403 column "x" not allowed`. A full-row read is requested with `"select_all": true` (expanded to the columns the role may read — never a raw `SELECT *`); **omitting `columns` returns nothing**, so a hidden column never leaks by being left out, grouped on, or filtered on. See [Access control → Column permissions](/access-control#column-permissions). +Every referenced column—in `columns`, aggregations, `filters`, `group_by`, `order_by`, or `time_range`—must be permitted by the role's `allow_columns`/`deny_columns` or the request returns `403 column "x" not allowed`. Use `"select_all": true` for a full-row read (expanded to permitted columns; never raw `SELECT *`). **Omitting `columns` returns nothing** to prevent hidden column leaks. See [Access control → Column permissions](/access-control#column-permissions). ::: **Request:** @@ -479,41 +478,39 @@ Every column the query references — in `columns`, an aggregation argument, `fi | Field | Type | Required | Description | | ----- | ---- | -------- | ----------- | -| `columns` | string \| string[] | No | Columns to SELECT — an array, or a single string for one column. A literal `"*"` is the column *named* `*`, **not** a wildcard. Omit (or send `[]` / `""`) to select nothing; use `select_all` for a full-row read. Mutually exclusive with `select_all`. | -| `select_all` | bool | No | Select every column the role may read (the all-columns wildcard, expanded server-side to the allow/deny set). Mutually exclusive with a non-empty `columns`, and with `aggregations`. | +| `columns` | string \| string[] | No | Columns to SELECT. A literal `"*"` is a column *named* `*`, not a wildcard. Omit (or send `[]`/`""`) to select nothing; use `select_all` for full-row reads. Mutually exclusive with `select_all`. | +| `select_all` | bool | No | Selects all columns the role may read. Mutually exclusive with non-empty `columns` and `aggregations`. | | `aggregations` | object[] | No | Aggregation functions (`fn`, `column`, `alias`). | | `filters` | object[] | No | WHERE conditions (`column`, `op`, `value`). Ops: eq, neq, gt, gte, lt, lte, in, like. | | `group_by` | string[] | No | GROUP BY columns. | | `order_by` | object[] | No | ORDER BY clauses (`column`, `dir`). | -| `limit` | int | No | Max rows. Omitted or above the configured `query.default_max_rows` (default 10,000) → silently capped at that value; a policy `max_rows` can lower it further (see [Access Control](/access-control#resource-limits)). | -| `time_range` | object | No | Time window (`column`, `since`, `until`). `since`/`until` accept RFC3339 or Go-duration relative values ("1h", "30m", "7d", "2w" — day and week suffixes expand to hours). Relative values mean that long *ago*. The window applies only when `column` and `since` are set — an `until` without `since` is ignored. | +| `limit` | int | No | Max rows. Omitted or above `query.default_max_rows` (default 10,000) → silently capped; policy `max_rows` can lower this further ([Access Control](/access-control#resource-limits)). | +| `time_range` | object | No | Time window (`column`, `since`, `until`). `since`/`until` accept RFC3339 or Go-durations ("1h", "30m", "7d", "2w"). Window applies only if `column` and `since` are set; `until` without `since` is ignored. | :::note[Identifier names] -Table, column, and alias names may contain any characters ClickHouse accepts — dots, spaces, unicode, reserved keywords — because every identifier is backtick-quoted automatically. The one exception is a name containing a literal `?`, which is rejected with `400` (a clickhouse-go positional-binder limitation tracked in [#279](https://github.com/Wave-RF/WaveHouse/issues/279)). +Identifiers are backtick-quoted, allowing dots, spaces, unicode, and keywords. Names containing literal `?` are rejected with `400` (clickhouse-go limitation [#279](https://github.com/Wave-RF/WaveHouse/issues/279)). ::: **Response:** -JSON array of result rows. Top-level `DateTime`/`DateTime64` values are returned in canonical RFC 3339 UTC (`2026-06-21T04:00:00.123Z`) — `Nullable` timestamp columns included (a SQL `NULL` renders as JSON `null`), while timestamps nested inside `Array`/`Map`/`Tuple` columns are rendered in the column's declared zone, else the ClickHouse server's, as the driver returns them — byte-identical to the [SSE stream](#get-v1stream--server-sent-events-stream) for values [canonicalized at ingest](#timestamp-canonicalization) (a fail-open pass-through that ClickHouse accepted still comes back canonical here, though it streamed in the producer's spelling). The response carries an `X-Cache: HIT` or `X-Cache: MISS` header — this endpoint shares the in-process L1 (Ristretto) + singleflight machinery (unlike `/v1/admin/query`, which always hits ClickHouse). +JSON array of result rows. Top-level `DateTime`/`DateTime64` values use RFC 3339 UTC (`2026-06-21T04:00:00.123Z`), including `Nullable` columns (SQL `NULL` as JSON `null`). Timestamps in `Array`/`Map`/`Tuple` use the column's or server's zone, identical to the [SSE stream](#get-v1stream--server-sent-events-stream) for values [canonicalized at ingest](#timestamp-canonicalization). Includes `X-Cache: HIT` or `X-Cache: MISS` headers via L1 (Ristretto) + singleflight machinery. -The inbound request body is capped at 1 MiB; a body over the cap is rejected with `413`. A query AST is bounded by nature (far under 1 MiB even with a large `in`-list), and the cap blocks a single-request memory-exhaustion vector on this public endpoint. Set a tighter or higher outer limit at your [reverse proxy](/reverse-proxy#request-body-size-limits) — but it can only narrow the effective limit, not raise it past this cap. +Request bodies are capped at 1 MiB (`413` if exceeded) to prevent memory exhaustion. Reverse proxy limits ([/reverse-proxy#request-body-size-limits](/reverse-proxy#request-body-size-limits)) can narrow but not raise this cap. **Error responses:** | Status | Body | Cause | | ------ | ---- | ----- | -| 400 | `{"error":"..."}` | Schema validation error (unknown column, bad aggregation, or an unparseable `time_range` `since`/`until` — neither a relative duration nor an RFC3339 timestamp) | +| 400 | `{"error":"..."}` | Schema validation error (unknown column, bad aggregation, or unparseable `time_range`) | | 403 | `{"error":"forbidden"}` | Role lacks select permission on table | | 403 | `{"error":"column \"x\" not allowed"}` | Column denied by policy | | 403 | `{"error":"aggregation \"x\" not allowed"}` | Aggregation fn denied by policy | | 404 | `{"error":"unknown table: x"}` | Table not found | -| 413 | `{"error":"request body exceeded 1048576 bytes"}` | Request body over the 1 MiB cap | - ---- +| 413 | `{"error":"request body exceeded 1048576 bytes"}` | Request body over 1 MiB cap | ### `GET/POST /v1/pipes/{name}` — Execute Named Pipe -Executes a pre-defined named query (pipe) with parameter binding. Parameters can be supplied via query string and/or JSON body. Results are cached in the shared L1 (Ristretto) with singleflight coalescing — same machinery as the structured query endpoint, and again, unlike `/v1/admin/query`. +Executes a pre-defined named query (pipe) with parameter binding via query string and/or JSON body. Results use shared L1 (Ristretto) caching with singleflight coalescing, unlike `/v1/admin/query`. **Query Parameters:** Any key matching a pipe parameter name. @@ -528,42 +525,40 @@ Executes a pre-defined named query (pipe) with parameter binding. Parameters can **Response:** -JSON array of result rows, with `X-Cache: HIT` or `X-Cache: MISS` indicating whether the row came from the in-process L1. +JSON array of result rows; `X-Cache: HIT` or `X-Cache: MISS` indicates if the row came from L1. -The POST parameter body is capped at 1 MiB; a body over the cap is rejected with `413` (the same control-plane cap as [`POST /v1/query`](#post-v1querytabletable--structured-query) — see [reverse proxy → body limits](/reverse-proxy#request-body-size-limits)). A malformed-but-within-cap body is ignored rather than rejected, since parameters may legitimately come from the query string alone. +POST bodies are capped at 1 MiB; exceeding this returns `413` (same as [`POST /v1/query`](#post-v1querytabletable--structured-query) — see [reverse proxy → body limits](/reverse-proxy#request-body-size-limits)). Malformed bodies within the cap are ignored, allowing parameters to come from the query string alone. **Error responses:** | Status | Body | Cause | | ------ | ---- | ----- | | 404 | `{"error":"pipe not found"}` | Pipe name not registered | -| 403 | `{"error":"forbidden"}` | Role not in pipe's `allowed_roles` (and not the admin role). Fails closed: a request with no role (no token, or a JWT missing `auth.role_claim`) is denied unless a `default_role` resolves it into the list; a pipe with no `allowed_roles` denies everyone but the admin role. | -| 400 | `{"error":"missing required parameter: x"}` | Required parameter not supplied | -| 400 | `{"error":"parameter \"x\": unsupported parameter type object"}` | A non-scalar value with no SQL literal form — a JSON object, whether supplied directly or nested as an array element. A JSON **array** is valid and renders as an `IN`-style `(…)` list. | -| 400 | `{"error":"parameter \"x\": array parameter must not be empty"}` | An empty array — it would render as the invalid `IN ()`. | -| 413 | `{"error":"request body exceeded 1048576 bytes"}` | POST body over the 1 MiB cap | - ---- +| 403 | `{"error":"forbidden"}` | Role not in `allowed_roles` (and not admin). Fails closed: requests without roles are denied unless a `default_role` resolves them; pipes with no `allowed_roles` deny all but admins. | +| 400 | `{"error":"missing required parameter: x"}` | Required parameter missing | +| 400 | `{"error":"parameter \"x\": unsupported parameter type object"}` | Non-scalar value without SQL literal form (JSON object). JSON arrays are valid and render as `IN` lists. | +| 400 | `{"error":"parameter \"x\": array parameter must not be empty"}` | Empty array (renders as invalid `IN ()`) | +| 413 | `{"error":"request body exceeded 1048576 bytes"}` | POST body over 1 MiB cap | ### `GET /v1/stream` — Server-Sent Events Stream -Opens a persistent SSE connection for real-time event streaming. Supports historical gap-fill from NATS JetStream using `DeliverByStartTime`. +Opens a persistent SSE connection for real-time event streaming. Supports historical gap-fill from NATS JetStream via `DeliverByStartTime`. **Query Parameters:** | Param | Type | Default | Description | | ----- | ---- | ------- | ----------- | -| `table` | string | (required) | Table name to subscribe to. Returns `400` only if missing/empty; other values aren't rejected — the name is encoded into a NATS-safe subject token (wildcards `*` / `>` are percent-encoded), so a nonexistent or odd name simply matches no events. | -| `since` | string | — | RFC 3339 or RFC 3339 Nano timestamp. If provided, replays historical events from NATS before switching to live streaming. | -| `token` | string | — | JWT token (alternative to `Authorization` header, useful for `EventSource`). Stripped from URL after extraction. | +| `table` | string | (required) | Table name to subscribe to. Returns `400` if missing/empty. Names are encoded into NATS-safe subject tokens (wildcards `*` / `>` are percent-encoded); nonexistent names match no events. | +| `since` | string | — | RFC 3339 or RFC 3339 Nano timestamp. Replays historical NATS events before live streaming. | +| `token` | string | — | JWT token (alternative to `Authorization` header). Stripped from URL after extraction. | **Headers:** | Header | Description | | ------ | ----------- | -| `Last-Event-ID` | RFC 3339 timestamp of the last received event. If present, overrides the `since` query parameter for automatic reconnection (standard `EventSource` behavior). | +| `Last-Event-ID` | RFC 3339 timestamp of the last received event. Overrides `since` for automatic reconnection (`EventSource` behavior). | -**Response:** SSE stream (`text/event-stream`). Each event includes an `id:` field set to the event's `received_timestamp`. The stream opens with a `: connected` comment and emits a minimal `:` keepalive comment periodically (every 30 seconds by default), which keeps a quiet connection from being closed by a proxy; both are standard SSE comments that `EventSource` ignores (raw consumers should skip `:`-prefixed lines). +**Response:** SSE stream (`text/event-stream`). Each event includes an `id:` field set to the `received_timestamp`. The stream starts with a `: connected` comment and emits a `:` keepalive comment every 30 seconds by default to prevent proxy closure; both are standard SSE comments ignored by `EventSource` (raw consumers should skip `:`-prefixed lines). ```text id: 2026-03-24T12:00:00.123Z @@ -573,16 +568,16 @@ id: 2026-03-24T12:00:01.456Z data: {"table_name":"clicks","received_timestamp":"2026-03-24T12:00:01.456Z","data":{"page":"/pricing"}} ``` -Each SSE connection is bound to a single `?table=`; to consume multiple tables, open one connection per table. +Each connection binds to one `?table=`; open separate connections for multiple tables. -Row values of top-level `DateTime`/`DateTime64` columns inside `data` arrive in the canonical RFC 3339 UTC form (ingest rewrites them before publishing — see [timestamp canonicalization](#timestamp-canonicalization)), so a live event and a `/v1/query` read of the same row agree on the instant in zone-explicit form — a zone-less spelling no longer parses as local time in a browser ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). The two renderings are byte-identical regardless of the declared time zone or a `Nullable` wrapper — a column declared with a non-UTC zone also streams as `Z`, and `/v1/query` normalizes it (nullable or not) to UTC before rendering. Canonicalization is fail-open at ingest, so a value outside the accepted input forms streams in whatever spelling the producer sent — and for exactly those events the byte-identity above does not hold: a spelling ClickHouse accepts anyway is stored and still queries back canonical, while one it too rejects lands in the DLQ and never becomes queryable at all. Events ingested before this behavior shipped likewise replay in their original spelling. +Top-level `DateTime`/`DateTime64` columns in `data` use canonical RFC 3339 UTC form (see [timestamp canonicalization](#timestamp-canonicalization)). Live events and `/v1/query` reads of the same row are byte-identical regardless of declared time zone or `Nullable` wrapper; non-UTC zones stream as `Z`. Canonicalization is fail-open at ingest: values outside accepted forms stream in original spelling, breaking byte-identity. Spells ClickHouse accepts are stored and query back canonical; rejected ones land in the DLQ. Events ingested before this behavior replay in original spelling ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). -**Note:** When access control policies are active, streamed events are filtered per the caller's role — denied columns are removed and tables without select permission are skipped. +**Note:** Streamed events are filtered by caller role—denied columns are removed and tables without select permission are skipped. -**CORS:** `/v1/stream` honors the `server.cors_allowed_origins` allowlist like every endpoint, so a browser `EventSource` from an allowed origin connects normally. `Last-Event-ID` is allow-listed in the CORS preflight so fetch-based clients can resume cross-origin. +**CORS:** Honors `server.cors_allowed_origins`. `Last-Event-ID` is allow-listed in CORS preflight for cross-origin resumption. :::caution[Behind a proxy: disable response buffering] -SSE needs one bit of proxy configuration: disable response buffering, or the proxy holds events until a buffer fills and clients receive nothing in real time. Idle timeouts are handled for you — the `:` keepalive comment above keeps a quiet stream alive under typical proxy/tunnel idle windows ([#226](https://github.com/Wave-RF/WaveHouse/issues/226)), so raising the idle/read timeout is now optional. Browser `EventSource` still auto-reconnects (resuming via `Last-Event-ID`) if a connection drops. See [Behind a reverse proxy → Server-Sent Events](/reverse-proxy#server-sent-events-sse) for nginx/Caddy/Cloudflare specifics. +Disable response buffering or proxies will hold events until the buffer fills. The `:` keepalive prevents idle timeouts ([#226](https://github.com/Wave-RF/WaveHouse/issues/226)), making higher idle/read timeouts optional. `EventSource` auto-reconnects via `Last-Event-ID`. See [Behind a reverse proxy → Server-Sent Events](/reverse-proxy#server-sent-events-sse). ::: **curl example:** @@ -595,14 +590,12 @@ curl -N "http://localhost:8080/v1/stream?table=clicks" curl -N "http://localhost:8080/v1/stream?table=clicks&since=2026-03-24T11:00:00Z" ``` ---- - ### `GET /v1/schema` — List All Table Schemas Returns all discovered ClickHouse table schemas. :::note[Admin only] -The schema and DLQ endpoints in this section require the `admin_role` (like [`/v1/admin/query`](#post-v1adminquery--query-clickhouse)); other callers get 401 (bad token) / 403. The quickstart's trial `public` role cannot call them. +Schema and DLQ endpoints require `admin_role` (e.g., [`/v1/admin/query`](#post-v1adminquery--query-clickhouse)); others receive 401 or 403 errors. The trial `public` role cannot call them. ::: **Response:** @@ -620,11 +613,9 @@ The schema and DLQ endpoints in this section require the `admin_role` (like [`/v ] ``` ---- - ### `GET /v1/schema?table={table}` — Get Table Schema -Returns the schema for a specific table. +Returns a specific table's schema. **Response:** @@ -642,21 +633,19 @@ Returns the schema for a specific table. | Status | Body | Cause | | ------ | ---- | ----- | -| 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | A present-but-invalid/expired token was supplied and denied (the gate surfaces the token reason) | -| 403 | `{"error":"forbidden"}` | Caller's role is not the policy `admin_role` (`"admin"` by default) | +| 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | Invalid or expired token supplied | +| 403 | `{"error":"forbidden"}` | Caller lacks `admin_role` (`"admin"` default) | | 404 | `{"error":"table not found"}` | Table not in discovered schemas | ---- - ### `POST /v1/schema/refresh` — Refresh Schemas -Triggers an immediate re-discovery of ClickHouse table schemas, then returns the refreshed schema list (same array shape as `GET /v1/schema`). Admin-only, like the rest of this section. +Triggers immediate ClickHouse table schema re-discovery and returns the refreshed list (same array shape as `GET /v1/schema`). Admin-only. **Error responses:** | Status | Body | Cause | | ------ | ---- | ----- | -| 401 / 403 | as above | Not the admin role | +| 401 / 403 | as above | Not admin role | | 500 | `{"error":"refresh failed"}` | ClickHouse discovery query failed | **Response:** @@ -672,25 +661,23 @@ Triggers an immediate re-discovery of ClickHouse table schemas, then returns the ] ``` ---- - ### `GET /v1/dlq/stats` — DLQ Statistics -Returns per-table message counts in the Dead Letter Queue. Admin-only, like the rest of this section. Before any failure has ever occurred, the endpoint returns `200` with `{"tables":{},"total":0}`. +Returns per-table message counts in the Dead Letter Queue. Admin-only. If no failures occurred, returns `200` with `{"tables":{},"total":0}`. **Error responses:** | Status | Body | Cause | | ------ | ---- | ----- | -| 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | A present-but-invalid/expired token was supplied and denied (the gate surfaces the token reason) | -| 403 | `{"error":"forbidden"}` | Caller's role is not the policy `admin_role` (`"admin"` by default) | +| 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | Invalid or expired token supplied | +| 403 | `{"error":"forbidden"}` | Role is not the policy `admin_role` (`"admin"` default) | | 500 | `{"error":"stream info failed"}` | NATS JetStream stream-info lookup failed | **Query Parameters:** | Param | Type | Default | Description | | ----- | ---- | ------- | ----------- | -| `table` | string | — | Filter stats to a specific table name (e.g., `?table=clicks` returns only the `clicks` count). | +| `table` | string | — | Filter stats to a specific table name (e.g., `?table=clicks`) | **Response:** @@ -704,13 +691,11 @@ Returns per-table message counts in the Dead Letter Queue. Admin-only, like the } ``` ---- - ### Admin Endpoints -Admin endpoints require the policy `admin_role` (`"admin"` by default, exact case-sensitive match). There is no separate `service` role. The JWT middleware always runs — a request with no/invalid token resolves to the `default_role` (not the admin role unless `default_role` is deliberately set to it — a loudly-warned dev-only setting) and is denied. +Admin endpoints require the `admin_role` policy (`"admin"` by default, case-sensitive). There is no separate `service` role. JWT middleware always runs; requests with no or invalid tokens resolve to the `default_role` and are denied unless that role has permissions. -The admin endpoints that accept a request body — `PUT /v1/admin/policy`, `POST /v1/admin/policy/validate`, and `PUT /v1/admin/pipes/{name}` — cap it at 1 MiB (the same control-plane backstop as the public read endpoints); an over-cap body is rejected with `413 {"error":"request body exceeded 1048576 bytes"}`. A policy document or pipe definition is bounded, so this never binds legitimate use. +Endpoints accepting request bodies—`PUT /v1/admin/policy`, `POST /v1/admin/policy/validate`, and `PUT /v1/admin/pipes/{name}`—cap input at 1 MiB. Over-cap bodies return `413 {"error":"request body exceeded 1048576 bytes"}`. #### `GET /v1/admin/policy` — Get Access Control Policy @@ -718,7 +703,7 @@ Returns the current access control policy. #### `PUT /v1/admin/policy` — Update Access Control Policy -Replaces the entire access control policy. Validated before saving. +Replaces and validates the entire access control policy. **Request:** @@ -752,11 +737,11 @@ Replaces the entire access control policy. Validated before saving. } ``` -The `default_role` field (optional) is the role assigned to any request that reaches the policy engine **without** a role — a valid token carrying no role claim, a request with no token at all, or one whose token was invalid/expired. **Setting it enables unauthenticated access:** roleless requests are evaluated as that role and receive exactly its permissions (or are denied if it grants none on the table/operation). If `default_role` is unset, a roleless request is denied. Setting it equal to the `admin_role` is allowed — every roleless request then becomes admin (including `/v1/admin/*`), which is handy for local/dev — but each node that loads such a policy logs a loud warning, and it must not be used in production. +The optional `default_role` is assigned to requests without a role (no token, invalid/expired token, or no role claim). **Setting this enables unauthenticated access.** If unset, roleless requests are denied. Setting it to `admin_role` grants admin access to all requests (including `/v1/admin/*`); this triggers a loud node warning and is for local/dev use only—not production. #### `POST /v1/admin/policy/validate` — Validate Policy (Dry Run) -Validates a policy without saving it. Returns `{"valid": true}` or an error. +Validates a policy without saving. Returns `{"valid": true}` or an error. #### `GET /v1/admin/pipes` — List Named Pipes @@ -780,7 +765,7 @@ Returns a specific named pipe definition. } ``` -**`allowed_roles`** restricts execution: the caller's role (a tokenless or roleless request is first resolved to the policy `default_role`) must appear in the list. The admin role (`admin_role`) always passes. Matching is exact — there is no `"*"` wildcard — and empty-string entries are ignored. An empty or omitted list authorizes **nobody but the admin role**, and a request whose role is absent or unlisted is denied (fails closed). +**`allowed_roles`** restricts execution: the caller's role (or `default_role` for roleless requests) must be in the list. The `admin_role` always passes. Matching is exact; no `"*"` wildcard exists, and empty strings are ignored. An empty or omitted list authorizes only the admin role; others are denied. #### `DELETE /v1/admin/pipes/{name}` — Delete Named Pipe @@ -788,7 +773,7 @@ Returns a specific named pipe definition. ### Internal Wire Format (NATS) -The message format used on NATS JetStream between ingest and the batch consumer: +Message format between ingest and batch consumer on NATS JetStream: ```json { @@ -805,12 +790,12 @@ The message format used on NATS JetStream between ingest and the batch consumer: | Field | Type | Description | | ----- | ---- | ----------- | | `table_name` | string | Target ClickHouse table (from URL). | -| `received_timestamp` | string | RFC 3339 nano timestamp when WaveHouse received the event. | -| `data` | object | The flat JSON body, with parseable `DateTime`/`DateTime64` column values rewritten to canonical RFC 3339 UTC (see [timestamp canonicalization](#timestamp-canonicalization)); other values as originally sent. | +| `received_timestamp` | string | RFC 3339 nano timestamp of WaveHouse receipt. | +| `data` | object | Flat JSON body; `DateTime`/`DateTime64` values rewritten to canonical RFC 3339 UTC ([timestamp canonicalization](#timestamp-canonicalization)); others as sent. | ### Client-Facing Format (SSE) -Same as the wire format — events are passed through directly: +Events pass through directly using the wire format: ```json { @@ -826,15 +811,17 @@ Same as the wire format — events are passed through directly: ## Dead Letter Queue (DLQ) -When a batch insert to ClickHouse fails (e.g., type errors, connection issues), the worker re-inserts the batch row by row: rows that succeed are acked, and only the rows that fail again are published to the DLQ NATS stream (`WAVEHOUSE_DLQ`) under subjects `dlq.{table}`. This prevents infinite retry loops — those messages are ACKed from the main stream and moved to the DLQ for inspection. The DLQ message body is the published `EventMessage` envelope (`{"table_name":…,"received_timestamp":…,"data":{…}}` — the failed row is under its `data` key, its `DateTime`/`DateTime64` values as published: canonicalized where WaveHouse could parse them, otherwise the producer's original spelling — see [timestamp canonicalization](#timestamp-canonicalization)); the failure reason, table, and time travel in the `X-DLQ-Table` / `X-DLQ-Error` / `X-DLQ-Timestamp` message headers. +If batch inserts to ClickHouse fail (e.g., type errors, connection issues), the worker re-inserts rows individually. Successful rows are acked; failures are published to the `WAVEHOUSE_DLQ` NATS stream under subjects `dlq.{table}` and ACKed from the main stream to prevent infinite loops. + +The DLQ message body is the `EventMessage` envelope (`{"table_name":…,"received_timestamp":…,"data":{…}}`). Failed rows are in the `data` key, with `DateTime`/`DateTime64` values canonicalized if WaveHouse parsed them, otherwise original (see [timestamp canonicalization](#timestamp-canonicalization)). Headers `X-DLQ-Table`, `X-DLQ-Error`, and `X-DLQ-Timestamp` contain failure details. -Use `GET /v1/dlq/stats` to monitor DLQ depth. +Monitor depth via `GET /v1/dlq/stats`. ## Generating a JWT for Testing -Needed whenever a caller must present a role — e.g. to reach an admin endpoint (role == `admin_role`) or any role beyond the policy `default_role`. The token must be signed with the configured `jwt_secret` (or a key the `jwks_url` serves) and must carry the role in its role claim (`auth.role_claim`, default `role`) — a token without the claim resolves to the policy `default_role`. +Required when callers need a specific role (e.g., `admin_role`) beyond the `default_role`. Tokens must be signed with the configured `jwt_secret` or a key from `jwks_url` and include the role in the claim specified by `auth.role_claim` (default: `role`). Tokens lacking this claim resolve to `default_role`. -`"change-me-in-production"` below is the placeholder shipped in the repo's `config.yaml` (what `make dev` / `./bin/wavehouse` load). The compose quickstart sets **no** secret — set `WH_AUTH_JWT_SECRET` on the `wavehouse` service and sign with that value (see [Development — Validating tokens](/development#validating-tokens)). +`"change-me-in-production"` is the placeholder in `config.yaml` used by `make dev` or `./bin/wavehouse`. The compose quickstart sets no secret; set `WH_AUTH_JWT_SECRET` on the `wavehouse` service and sign with that value (see [Development — Validating tokens](/development#validating-tokens)). ```bash # Using jwt-cli (https://github.com/mike-engel/jwt-cli): diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 06254e14..3a97bde7 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -9,7 +9,7 @@ This document describes the internal architecture of WaveHouse, a schema-aware C ## Overview -WaveHouse is a Go-based gateway that sits in front of ClickHouse, acting as the entry and exit point for data. It discovers your real ClickHouse table schemas, validates data at ingest time, batches inserts asynchronously, and provides real-time streaming and query caching. +WaveHouse is a Go gateway for ClickHouse. It discovers table schemas, validates ingest data, batches asynchronous inserts, and provides real-time streaming and query caching. ```mermaid flowchart TD @@ -43,7 +43,7 @@ flowchart TD ## Binaries -WaveHouse ships a single binary, `wavehouse`: an all-in-one process running the API, batch worker, embedded NATS JetStream, and optional embedded Pebble dedup. The only external dependency is ClickHouse. +`wavehouse` is a single binary running the API, batch worker, embedded NATS JetStream, and optional Pebble dedup. Dependency: ClickHouse. ## Internal Packages @@ -67,94 +67,94 @@ internal/ ### `api/` — HTTP Layer -The API layer uses [Chi](https://github.com/go-chi/chi) for routing with RequestID, a CORS middleware, and a custom JSON recoverer (`jsonRecoverer`) that emits a JSON `500` on panic instead of chi's plain-text `middleware.Recoverer`. +The API layer routes with [Chi](https://github.com/go-chi/chi): RequestID, CORS middleware, and a custom `jsonRecoverer` emitting a JSON `500` on panic instead of chi's plain-text `middleware.Recoverer`. -- **router.go** — Route definitions. Public: `/livez`, `/readyz`, and the content-free `/v1/health` SDK ping (plus the permanent `/healthz` alias and the deprecated `/health`, `/ready` aliases). Policy-gated: `/v1/ingest?table={table}`, `/v1/query?table={table}` (structured), `/v1/pipes/{name}` (named pipes), `/v1/stream`. Admin-only (`RequireAdmin` — role == `policy.admin_role`, or a request bearing the operator key's operator bit, which passes even under a nil policy): `/v1/schema/*`, `/v1/dlq/stats`, `/v1/admin/policy`, `/v1/admin/pipes/*`, `/v1/admin/query` (raw SQL — same gate as the rest of `/v1/admin/*`). -- **auth middleware** — the JWT/JWKS authentication middleware is its own package, [`auth/`](#auth--authentication); the router runs it on every `/v1/*` route. +- **router.go** — Route definitions. Public: `/livez`, `/readyz`, and the content-free `/v1/health` SDK ping (plus permanent alias `/healthz` and deprecated `/health`, `/ready`). Policy-gated: `/v1/ingest?table={table}`, `/v1/query?table={table}` (structured), `/v1/pipes/{name}` (named pipes), `/v1/stream`. Admin-only (`RequireAdmin` — role == `policy.admin_role` or operator key bit): `/v1/schema/*`, `/v1/dlq/stats`, `/v1/admin/policy`, `/v1/admin/pipes/*`, and `/v1/admin/query` (raw SQL). +- **auth middleware** — JWT/JWKS authentication via [`auth/`](#auth--authentication); applied to all `/v1/*` routes. - **policy.go** — CRUD handler for access control policies (`/v1/admin/policy`). - **pipes.go** — Named query pipe handlers: admin CRUD and execution with parameter binding. -- **structured_query.go** — Handler for `POST /v1/query?table={table}`: validates query AST, enforces permissions, builds and executes SQL. -- **ingest.go** — Accepts flat JSON body for `POST /v1/ingest?table={table}`, validates against discovered schema, optional dedup, publishes to NATS subject `ingest.{table}`. When dedup is on, a row missing the configured `id_field` can't be deduped: it is logged at `WARN` and counted by `wavehouse_ingest_dedupe_missing_id_total` (labeled by `table`), then published un-deduped — or rejected when `dedupe.require_id` is set ([#219](https://github.com/Wave-RF/WaveHouse/issues/219)). -- **query.go** — Proxies raw SQL for `POST /v1/admin/query` straight to ClickHouse's HTTP interface. **Not cached** — sets `Cache-Control: no-store` so every request hits ClickHouse; DateTime is rendered ISO-8601 via `date_time_output_format=iso` (the Go-side type conversion lives in the structured-query / pipes path, not here). -- **stream.go** — Real-time streaming via SSE. Callers select a table with the `?table=` query parameter. Each connection registers one `Subscriber` (the `stream/` package) with both the event `Hub` (under its `(topic, role)`) and the shared keepalive wheel, then drains both from a single byte-pump — so idle streams keep emitting `:` keepalive comments (surviving reverse-proxy idle timeouts) while live events arrive already projected and serialized. Per-event projection/serialization happens **once per role** in the `Hub`, not once per subscriber ([#294](https://github.com/Wave-RF/WaveHouse/issues/294)). Gap-fill replay from NATS JetStream (`DeliverByStartTime`) stays per-connection (low-volume, one-time on connect). -- **schema.go** — Schema discovery API: list all schemas, get one table, trigger refresh. -- **dlq.go** — DLQ stats endpoint and `EnsureDLQStream` helper for creating the `WAVEHOUSE_DLQ` NATS stream. -- **health.go** — Liveness (`/livez`), readiness (`/readyz`), and a content-free `Online` ping (`/v1/health`, the SDK's public liveness check); `/healthz` is a permanent alias of `/livez`, and `/health`/`/ready` are deprecated aliases. All three consult an optional `BootState` so they can return 503 while boot-time schema discovery is still failing in the retry loop (see `cmd/wavehouse/main.go`); once `BootState.Set(nil)` fires, `/livez` returns 200 and stays there. `/readyz` additionally pings ClickHouse each call; `/v1/health` deliberately does not. +- **structured_query.go** — Handler for `POST /v1/query?table={table}`: validates AST, enforces permissions, executes SQL. +- **ingest.go** — Accepts flat JSON for `POST /v1/ingest?table={table}`, validates schema, optionally dedups, and publishes to NATS subject `ingest.{table}`. If dedup is on but `id_field` is missing: logged at `WARN`, counted by `wavehouse_ingest_dedupe_missing_id_total` (labeled by `table`), then published un-deduped—or rejected if `dedupe.require_id` is set ([#219](https://github.com/Wave-RF/WaveHouse/issues/219)). +- **query.go** — Proxies raw SQL for `POST /v1/admin/query` to ClickHouse's HTTP interface. Sets `Cache-Control: no-store`; renders DateTime as ISO-8601 via `date_time_output_format=iso`. +- **stream.go** — SSE streaming via `?table=` parameter. Each connection registers one `Subscriber` (`stream/` package) with the event `Hub` (by topic, role) and keepalive wheel. Idle streams emit `:` comments to prevent proxy timeouts. Projection/serialization happens once per role in the `Hub` ([#294](https://github.com/Wave-RF/WaveHouse/issues/294)). NATS JetStream gap-fill (`DeliverByStartTime`) is per-connection. +- **schema.go** — Schema discovery: list schemas, get table, trigger refresh. +- **dlq.go** — DLQ stats and `EnsureDLQStream` helper for the `WAVEHOUSE_DLQ` NATS stream. +- **health.go** — Liveness (`/livez`), readiness (`/readyz`), and a content-free `Online` ping (`/v1/health`, the SDK's liveness check). `/healthz` aliases `/livez`; `/health`/`/ready` are deprecated. All consult `BootState` to return 503 during boot-time schema discovery (see `cmd/wavehouse/main.go`). Once `BootState.Set(nil)` fires, `/livez` returns 200. `/readyz` pings ClickHouse; `/v1/health` does not. ### `stream/` — SSE keepalive & fan-out -The SSE fan-out, factored out of `api/` so the delivery hot path ([#294](https://github.com/Wave-RF/WaveHouse/issues/294)) lives next to the keepalive primitives it shares. One abstraction per file. +The SSE fan-out is factored out of `api/` so the delivery hot path ([#294](https://github.com/Wave-RF/WaveHouse/issues/294)) resides with its shared keepalive primitives. One abstraction per file. -- **hub.go** — `Hub`, the event fan-out. Subscribers register under `(topic, role)`; `Broadcast` decodes each event once, applies each subscribed role's column policy once, builds one SSE frame per role, and fans it to every member of that role's `Bucket` — collapsing the prior per-subscriber `unmarshal → evaluate → filter → marshal` into one pass per distinct `(role, table)` output shape (the [#294](https://github.com/Wave-RF/WaveHouse/issues/294) lever; the measured ceiling was ~2 270 deliveries/s from re-projecting per subscriber). The `(topic, role)` key is sufficient because column visibility derives only from the role+table policy entry, never from JWT claims (claims feed only the row-level `WHERE`/`CHECK`, which the stream path does not apply). `ReplayFrame` shares the same projection for the handler's per-connection gap-fill. -- **subscriber.go** — `Subscriber`, the per-connection handle. It owns a single ready-to-write outbound queue of `Frame`s (each tagged with its `kind`, so the handler labels the write where it happens): producers — the keepalive wheel and the event `Hub` — fan frames in with `Send` (non-blocking; a full queue drops and the `Hub` counts it), and the handler drains `Frames()` to the client verbatim. The queue is sized for buffering live events (cap 64, up from the keepalive-only cap 1; #152 will make it a knob), and an `Evicted()` channel is the seam the slow-consumer follow-up closes to disconnect a wedged consumer. -- **bucket.go** — `Bucket`, the reusable fan-out primitive: a concurrency-safe set of subscribers. `Push` delivers a shared `Frame` to each fire-and-forget (the keepalive wheel's ring); `Snapshot` exposes the members so the event `Hub` can fan out while inspecting each `Send` result (to count drops). The `Hub` holds one `Bucket` per `(topic, role)` so a projected frame is built once and sent to every member instead of re-projected per subscriber. -- **heartbeat.go** — The keepalive wheel (`Heartbeater`). A single process-wide ticker fans a minimal `:` comment across the ring of `Bucket`s, waking ~1/N of live streams per tick so the writes don't synchronize. The effective per-connection keepalive period is `stream.keepalive_interval` (the wheel ticks every `keepalive_interval ÷ keepalive_buckets`, so one rotation spans the interval); the owning handler goroutine does the actual write, so the shared ticker never touches a `ResponseWriter` directly. -- **metrics.go** — `Metrics`, the SSE instrument set: `wavehouse_sse_active_streams` (open streams), `wavehouse_sse_stream_duration_seconds` (lifetime), `wavehouse_sse_frames_sent_total` / `wavehouse_sse_bytes_sent_total` (labeled by `kind`: `keepalive`, `event`, `replay`), and `wavehouse_sse_dropped_frames_total` (frames dropped to a full subscriber queue — the slow-consumer signal that was silent before #294). Nil-safe, so the handler holds one unconditionally and tests skip wiring it; one shared instance records both the handler's write sites and the `Hub`'s drop counts. Separate from `observability.RegisterSystemMetrics`, which covers only the NATS/Pebble system gauges. Streams are observed through these metrics rather than per-event traces (the router excludes `/v1/stream` from the HTTP tracer). +- **hub.go** — `Hub`, the event fan-out. Subscribers register under `(topic, role)`. `Broadcast` decodes each event once, applies the role's column policy once, builds one SSE frame per role, and fans it to every member of that role's `Bucket`. This collapses the prior per-subscriber `unmarshal → evaluate → filter → marshal` into one pass per distinct `(role, table)` output shape (the [#294](https://github.com/Wave-RF/WaveHouse/issues/294) lever; previous ceiling was ~2 270 deliveries/s). The `(topic, role)` key suffices because column visibility derives from the role+table policy, not JWT claims (claims only feed row-level `WHERE`/`CHECK`, which the stream path ignores). `ReplayFrame` uses this projection for per-connection gap-fill. +- **subscriber.go** — `Subscriber`, the per-connection handle. It owns an outbound queue of `Frame`s tagged by `kind`. Producers (keepalive wheel and event `Hub`) use `Send` (non-blocking; full queues drop frames and the `Hub` counts them), while the handler drains `Frames()` to the client. The queue is sized for live events (cap 64, up from keepalive cap 1; #152 will make this a knob). An `Evicted()` channel allows slow-consumer follow-up to disconnect wedged clients. +- **bucket.go** — `Bucket`, a concurrency-safe set of subscribers. `Push` delivers a shared `Frame` to each (used by the keepalive wheel); `Snapshot` exposes members so the `Hub` can fan out and count drops via `Send` results. The `Hub` holds one `Bucket` per `(topic, role)` to avoid re-projecting frames per subscriber. +- **heartbeat.go** — The keepalive wheel (`Heartbeater`). A process-wide ticker fans a `:` comment across the `Bucket` ring, waking ~1/N of streams per tick to prevent synchronized writes. The effective period is `stream.keepalive_interval` (wheel ticks every `keepalive_interval ÷ keepalive_buckets`). The handler goroutine performs the write; the ticker never touches a `ResponseWriter`. +- **metrics.go** — `Metrics`, the SSE instrument set: `wavehouse_sse_active_streams`, `wavehouse_sse_stream_duration_seconds`, `wavehouse_sse_frames_sent_total` / `wavehouse_sse_bytes_sent_total` (labeled by `kind`: `keepalive`, `event`, `replay`), and `wavehouse_sse_dropped_frames_total` (slow-consumer signal added in #294). Nil-safe, one shared instance records handler writes and `Hub` drops. Separate from `observability.RegisterSystemMetrics` (NATS/Pebble gauges). Streams use these metrics instead of per-event traces; the router excludes `/v1/stream` from the HTTP tracer. ### `auth/` — Authentication -- **auth.go** — `Middleware(cfg, store, logger)`: the auth middleware. Verifies JWT tokens with HMAC **or** JWKS (never both), with the accepted `alg` pinned to the active verifier and checked before any key is consulted (rejects `alg: none` and cross-family confusion). Extracts the caller's role from a configurable dot-path claim (`auth.role_claim`, default `role`). It always runs and never rejects — a missing/invalid/expired token yields an empty role (resolved to `default_role` downstream), with the token error stashed in context so a denying gate can fail loud (`401`, not a bare `403`). Before the Bearer token it checks a non-JWT operator key (`auth.operator_key`): a constant-time match on the presented credential — an `Authorization: Operator ` header, or the `X-Operator-Key` alias — stamps the live admin role plus an operator bit (`auth.WithOperator`) that `RequireAdmin` honors even under a nil policy — a full-access break-glass credential, audit-logged at Info with no client IP (`store`/`logger` back this path). A presented-but-wrong operator key is logged at `WARN` and counted by `wavehouse_auth_operator_key_failures_total` — a probing signal on the most privileged credential — then falls through like any unauthenticated request (the middleware never rejects). -- **context.go** — request-context accessors and their setters for the role, claims, and token error (`RoleFromContext`, `ClaimsFromContext`, `AuthErrorFromContext`, and the matching `With*` helpers). +- **auth.go** — `Middleware(cfg, store, logger)`: auth middleware. Verifies JWT tokens via HMAC **or** JWKS (never both), pinning the accepted `alg` to the active verifier and checking it before key consultation (rejects `alg: none` and cross-family confusion). Extracts caller roles from a configurable dot-path claim (`auth.role_claim`, default `role`). It never rejects; missing/invalid/expired tokens yield an empty role (resolved to `default_role` downstream), stashing the error in context so denying gates can return `401` instead of `403`. Before JWTs, it checks a non-JWT operator key (`auth.operator_key`) via constant-time match on `Authorization: Operator ` or `X-Operator-Key`. Matches grant the admin role and an operator bit (`auth.WithOperator`), which `RequireAdmin` honors even under nil policy—a break-glass credential audit-logged at Info without client IP (`store`/`logger` backed). Wrong keys log at `WARN`, increment `wavehouse_auth_operator_key_failures_total`, then fall through as unauthenticated. +- **context.go** — Request-context accessors and setters for role, claims, and token error (`RoleFromContext`, `ClaimsFromContext`, `AuthErrorFromContext`, and matching `With*` helpers). ### `cache/` — Query Cache - **cache.go** — `Cache` interface: `Get`, `Set`, `Close`. -- **local.go** — In-process cache using [Ristretto](https://github.com/dgraph-io/ristretto) with `sync.Map` TTL tracking. -- **tiered.go** — Wraps the local cache with [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) to prevent cache stampede on concurrent misses. The tiered interface accepts an optional second cache slot for future shared-cache backends, but ships with the slot empty. +- **local.go** — In-process cache via [Ristretto](https://github.com/dgraph-io/ristretto) and `sync.Map` TTL tracking. +- **tiered.go** — Wraps local cache with [singleflight](https://pkg.go.dev/golang.org/x/sync/singleflight) to prevent stampedes; includes an empty second slot for future shared backends. ### `config/` — Configuration -- **config.go** — Loads configuration from YAML file with environment variable overrides (using [cleanenv](https://github.com/ilyakaznacheev/cleanenv)). All settings use `WH_` prefixed env vars. See [Configuration Reference](/configuration). +- **config.go** — Loads YAML config with `WH_` environment variable overrides via [cleanenv](https://github.com/ilyakaznacheev/cleanenv). See [Configuration Reference](/configuration). ### `dedupe/` — Deduplication (Optional) - **dedupe.go** — `Deduplicator` interface: `CheckAndMark(ctx, eventID) (bool, error)`. -- **embedded.go** — Uses [Pebble](https://github.com/cockroachdb/pebble) (embedded key-value store). Key = event ID. +- **embedded.go** — Uses [Pebble](https://github.com/cockroachdb/pebble). Key = event ID. ### `discovery/` — Schema Discovery & Validation -- **discovery.go** — `SchemaRegistry` queries `system.columns` to discover ClickHouse table schemas. Each refresh also discovers the server's default time zone (`SELECT timezone()`) and bakes every `DateTime`/`DateTime64` column's canonicalization spec (precision + resolved zone) into the cached schema, so the per-record ingest path parses no type strings and loads no zones ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). Supports periodic auto-refresh, on-demand refresh, and `RetryRefresh` (boot-time exponential backoff loop used by `cmd/wavehouse` so a transiently unreachable ClickHouse doesn't crash-loop the binary). Thread-safe via `sync.RWMutex`. -- **timestamp.go** — `CanonicalizeTimestamps(schema, data)` rewrites every parseable value in a top-level `DateTime`/`DateTime64` column to the canonical RFC 3339 UTC wire form before the event is published ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)): zone-less values are interpreted in the column's declared zone, else the discovered server default — ClickHouse's own rule, so the spelling changes but never the instant. Fail-open: an unparseable value or unresolvable zone passes through verbatim for ClickHouse's own parser to judge; ingest never rejects a record over its timestamp spelling. -- **validation.go** — `Validate(schema, data)` checks incoming JSON against the discovered schema: unknown fields, type compatibility, missing required columns, null handling. +- **discovery.go** — `SchemaRegistry` queries `system.columns` to discover ClickHouse schemas. Refreshes also fetch the server's default time zone (`SELECT timezone()`) and bake each `DateTime`/`DateTime64` column's canonicalization spec (precision + resolved zone) into the cache, eliminating per-record type string parsing or zone loading ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). Supports periodic auto-refresh, on-demand refresh, and `RetryRefresh` (boot-time exponential backoff for `cmd/wavehouse` to prevent crash-loops during transient outages). Thread-safe via `sync.RWMutex`. +- **timestamp.go** — `CanonicalizeTimestamps(schema, data)` rewrites top-level `DateTime`/`DateTime64` values to RFC 3339 UTC wire form before publishing ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). Zone-less values use the column's declared zone or server default; unparseable values pass through verbatim for ClickHouse to judge. +- **validation.go** — `Validate(schema, data)` checks JSON against schemas for unknown fields, type compatibility, missing required columns, and null handling. - **discovery_test.go** — Unit tests for validation logic. ### `ingest/` — Ingest Pipeline, DLQ & Sweeping -- **worker.go** — `StartIngestWorker` launches an ingest pipeline: a JetStream consumer reads from the `WAVEHOUSE` stream via a durable `buffer-consumer` pull subscription, batches events per table, and performs bulk INSERTs to ClickHouse. The pipeline is **insert-only**. The wire format `EventMessage` carries `{table_name, received_timestamp, data}` and nothing else; the worker accepts any table name now (the table name in the NATS subject is `query.SafeEncodeNATS(rawUnsafeTableName)`), then bulk-INSERTs. The embedded NATS server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/admin/query` under the admin role (`policy.admin_role`) — see the Query Path section below; the `/v1/admin/*` `RequireAdmin` middleware enforces the check at the API layer, so a no/invalid-token request (resolved to `default_role`, not admin in a production config) never reaches the proxy. On a bulk-insert failure the batch is re-inserted row by row; rows that succeed are acked, and only the rows that fail again are routed to the DLQ (`sendToDLQ`), which republishes the as-published `EventMessage` envelope to `dlq.{table}` NATS subjects with the failure context in `X-DLQ-*` headers when DLQ is enabled — see [Ingest Pipeline](/ingest-pipeline) for the worker internals. -- **types.go** — `EventMessage` struct (TableName, ReceivedTimestamp, Data) and `BufferConsumerName` constant, shared across API handlers and the ingest pipeline. -- **sweeper.go** — `Sweeper` implements the Active Sweeper pattern. It runs every minute and purges NATS JetStream messages that are **both** ACKed by the buffer consumer (written to ClickHouse) **and** older than the configurable gap window. +- **worker.go** — `StartIngestWorker` launches an insert-only pipeline: a JetStream consumer reads from the `WAVEHOUSE` stream via a durable `buffer-consumer` pull subscription, batches events per table, and bulk-INSERTs to ClickHouse. The `EventMessage` wire format contains `{table_name, received_timestamp, data}`. The worker accepts any table name (NATS subject: `query.SafeEncodeNATS(rawUnsafeTableName)`). Since the embedded NATS server uses `DontListen: true` (`internal/mq/embedded.go`), only in-process Go code—currently the `/v1/ingest?table={table}` handler—can publish to `ingest.>` subjects. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) require `POST /v1/admin/query` under `policy.admin_role`; the `/v1/admin/*` `RequireAdmin` middleware blocks non-admin requests at the API layer. On bulk-insert failure, rows are re-inserted individually; failed rows route to the DLQ (`sendToDLQ`), republishing the `EventMessage` to `dlq.{table}` with `X-DLQ-*` headers if enabled. See [Ingest Pipeline](/ingest-pipeline). +- **types.go** — Contains `EventMessage` struct (TableName, ReceivedTimestamp, Data) and `BufferConsumerName` constant. +- **sweeper.go** — `Sweeper` implements the Active Sweeper pattern, purging NATS JetStream messages every minute if they are ACKed by the buffer consumer and older than the configured gap window. ### `mq/` — Message Queue -- **mq.go** — `Publisher` and `Subscriber` interfaces. `Message` struct with `DoubleAck(ctx)`, `Ack()`, and `Nak()`. -- **embedded.go** — In-process NATS server with JetStream. Creates stream `WAVEHOUSE` with subjects `ingest.>`. +- **mq.go** — `Publisher`/`Subscriber` interfaces; `Message` struct with `DoubleAck(ctx)`, `Ack()`, and `Nak()`. +- **embedded.go** — In-process NATS JetStream server. Creates stream `WAVEHOUSE` (subjects `ingest.>`). ### `observability/` — OpenTelemetry Pipeline -- **provider.go** — `InitProvider(ctx, serviceName, ProviderConfig)` wires the OTel pipeline. Each output is independently gated; the W3C TraceContext + Baggage propagator is always installed (cheap, harmless when traces are off). Returns `(shutdown, promHandler http.Handler, err)` — `promHandler` is non-nil only when `PrometheusEnabled` is true and reads from a *private* `prometheus.Registry` to avoid leaking the process/Go collectors that `prometheus.DefaultRegisterer` auto-registers. OTLP-metrics push (`MetricsEnabled`) and Prometheus exposition (`PrometheusEnabled`) are independent: either, both, or neither may be set, and any combination produces a single MeterProvider feeding the active readers. The Endpoint field is only dialed by the OTLP exporters (traces / metrics-OTLP / logs); Prometheus-only operation leaves it untouched. Provider init in `main.go` runs whenever `otel.enabled` OR `prometheus.enabled` is true, so Prometheus-only operation (Alloy/scrape, no collector) is a first-class mode. -- **logger.go** — `NewLogger(component, level, isJSON, otlpSampleRate)` produces a slog logger that fans out to stdout (always 100%) and the OTLP log exporter (DEBUG/INFO sampled at `otlpSampleRate`, WARN/ERROR always 100% as a non-configurable safety floor). `TraceHandler` injects `trace_id`/`span_id` from the active span when one exists. `otlpSamplerFn` is exposed (lowercase) for unit testing the per-level rate logic without driving through the slogmulti middleware. -- **metrics.go** — `RegisterSystemMetrics(natsServer, dedup)` registers observable gauges for embedded NATS connections, in-msgs, and Pebble dedupe storage stats. Wired in `cmd/wavehouse/main.go` after the providers are up. -- **tracer.go** — W3C TraceContext propagation over NATS message headers (`InjectNATS` / `ExtractNATS`) — bridges the API request span into the ingest worker so end-to-end traces survive the queue handoff. +- **provider.go** — `InitProvider(ctx, serviceName, ProviderConfig)` wires the OTel pipeline. W3C TraceContext + Baggage propagators are always installed. Returns `(shutdown, promHandler http.Handler, err)`. `promHandler` is non-nil if `PrometheusEnabled` is true; it uses a *private* `prometheus.Registry` to avoid leaking the process/Go collectors `prometheus.DefaultRegisterer` auto-registers. OTLP-metrics push (`MetricsEnabled`) and Prometheus exposition (`PrometheusEnabled`) are independent; any combination produces one MeterProvider. The Endpoint field is only used by OTLP exporters (traces/metrics-OTLP/logs). Provider init in `main.go` runs if `otel.enabled` OR `prometheus.enabled` is true, making Prometheus-only operation a first-class mode. +- **logger.go** — `NewLogger(component, level, isJSON, otlpSampleRate)` creates a slog logger fanning out to stdout (100%) and the OTLP log exporter (DEBUG/INFO sampled at `otlpSampleRate`, WARN/ERROR always 100%). `TraceHandler` injects `trace_id`/`span_id` from active spans. `otlpSamplerFn` is exposed for unit testing rate logic. +- **metrics.go** — `RegisterSystemMetrics(natsServer, dedup)` registers observable gauges for embedded NATS connections, in-msgs, and Pebble dedupe storage stats. Wired in `cmd/wavehouse/main.go`. +- **tracer.go** — W3C TraceContext propagation over NATS headers (`InjectNATS` / `ExtractNATS`) bridges API request spans into ingest workers for end-to-end traces. -The package's design invariants — stdout always 100%, WARN+ERROR always export at 100%, gRPC exporters dial lazily so unreachable collectors never block startup, private Prometheus registry — are documented in AGENTS.md "Key Design Decisions" #15 and must be preserved by anything touching this package. +Design invariants—100% stdout, 100% WARN+ERROR export, lazy gRPC dialing to prevent startup blocks, and private Prometheus registry—are documented in AGENTS.md "Key Design Decisions" #15. ### `policy/` — Access Control -- **policy.go** — Hasura-style policy types (`Policy`, `TablePolicy`, `RolePermissions`, `Filter`), `Evaluate()` function that resolves permissions against JWT claims (including `{{ jwt.claim.path }}` template resolution), the per-column decision `IsColumnAllowed()` plus its batch/projection forms `AllowedProjection()` and `RestrictsColumns()` (used to expand a `select_all` request into a role's allowed columns), `IsAggregationAllowed()`, `Validate()`. -- **store.go** — `Store` backed by NATS KV bucket `WAVEHOUSE_POLICY`. Supports file-based bootstrap (YAML/JSON), cluster-wide sync via KV Watch, local caching. +- **policy.go** — Hasura-style types (`Policy`, `TablePolicy`, `RolePermissions`, `Filter`). Includes `Evaluate()` for JWT claims (resolving `{{ jwt.claim.path }}`), column decisions via `IsColumnAllowed()`, `AllowedProjection()`, and `RestrictsColumns()` (expanding `select_all`), `IsAggregationAllowed()`, and `Validate()`. +- **store.go** — `Store` using NATS KV bucket `WAVEHOUSE_POLICY`. Supports YAML/JSON bootstrap, KV Watch sync, and local caching. ### `pipes/` — Named Query Pipes -- **pipes.go** — `NamedQuery` type with SQL template and parameter definitions, `Store` backed by NATS KV bucket `WAVEHOUSE_PIPES`. Supports `.sql` file directory bootstrap. `BindParams()` resolves `{{param}}` / `{{param:default}}` placeholders by inlining escaped literal values into the SQL (strings single-quote-escaped; arrays rendered as escaped `(…)` `IN`-lists). A non-scalar value with no SQL form (a JSON object, or an empty array) is rejected rather than emitted raw. +- **pipes.go** — `NamedQuery` type defines SQL templates and parameters; `Store` uses NATS KV bucket `WAVEHOUSE_PIPES`. Supports `.sql` directory bootstrap. `BindParams()` resolves `{{param}}`/`{{param:default}}` by inlining escaped literals (single-quoted strings, `(…)` `IN`-lists). Non-scalar values without SQL forms (JSON objects, empty arrays) are rejected. ### `query/` — Structured Query Engine - **ast.go** — `StructuredQuery` AST types: columns, aggregations, filters, group by, order by, limit, time range. -- **builder.go** — `Build()` converts AST to parameterized SQL. It is the single chokepoint that validates every referenced identifier against the schema **and** authorizes every column reference — projection, aggregation args, filters, group_by, order_by, time_range — against the role's column allowlist (the [#223](https://github.com/Wave-RF/WaveHouse/issues/223) hard cap). A full-row read is requested with `select_all`, which expands to the role's allowed columns rather than emitting a raw `SELECT *`; an omitted projection selects nothing, and `*` in `columns` is a literal column name. Every identifier is backtick-quoted via `internal/chsql` (`QuoteIdent`) so any ClickHouse-legal name is accepted — a name containing `?` is refused fail-closed ([#279](https://github.com/Wave-RF/WaveHouse/issues/279)). `InjectPermissionFilters()` adds row-level security. `ApplyMaxRows()` enforces limits. Timestamp bucketing for cache optimization. +- **builder.go** — `Build()` converts AST to parameterized SQL. It validates identifiers against the schema and authorizes all column references—projection, aggregation args, filters, group_by, order_by, time_range—against the role's allowlist ([#223](https://github.com/Wave-RF/WaveHouse/issues/223)). `select_all` expands to allowed columns instead of `SELECT *`; omitted projections select nothing; `*` is a literal name. Identifiers are backtick-quoted via `internal/chsql` (`QuoteIdent`); names with `?` fail closed ([#279](https://github.com/Wave-RF/WaveHouse/issues/279)). `InjectPermissionFilters()` adds row-level security. `ApplyMaxRows()` enforces limits. Timestamp bucketing optimizes cache. ### `chsql/` — ClickHouse SQL Helpers -- **chsql.go** — Dependency-free ClickHouse SQL helpers shared by `query/` and `policy/`, kept in their own package to break an import cycle. `QuoteIdent` is the single place every identifier — column, table, alias — becomes SQL text: always backtick-quoted and escaped, so any ClickHouse-legal name (dots, spaces, unicode, keywords) is safe. `BindUnsafe` reports whether a name contains a literal `?`, which would desync clickhouse-go's positional binder; such names are rejected fail-closed rather than silently mis-bound. +- **chsql.go** — Dependency-free helpers for `query/` and `policy/`, isolated to prevent import cycles. `QuoteIdent` backtick-quotes and escapes all identifiers (columns, tables, aliases), ensuring any ClickHouse-legal name is safe. `BindUnsafe` detects literal `?` characters; such names are rejected to avoid desyncing clickhouse-go's positional binder. ## Data Flows @@ -236,21 +236,7 @@ Client POST /v1/admin/query (browser, CDN, corp proxy) caches the result. ``` -The proxy-pattern wins are: zero classification logic on the WaveHouse -side (no isMutation heuristic to maintain), and any ClickHouse statement -type — including verbs added in future versions and inline FORMAT -overrides — works without WaveHouse code changes. Multi-statement input -(`SELECT 1; TRUNCATE t`) is supported when the upstream ClickHouse has -multi-query enabled, which is the default on recent versions; older or -restrictively-configured servers will return a clear error from -ClickHouse itself for the second statement. The proxy buffers the response in memory with a -64 MiB cap (502 with `clickhouse response exceeded N bytes` on overflow, -to keep a runaway `SELECT *` from pinning RAM on the API server), and -passes ClickHouse's `Content-Type` through when an inline `FORMAT` -directive overrides the default JSON envelope. The structured query -endpoint and pipes still go through `clickhouse-go`'s native driver -(Query/Exec) for performance and to keep the cached row-array shape -consistent. +This proxy pattern keeps classification logic out of WaveHouse: all ClickHouse statements — future verbs, inline `FORMAT` overrides — work without code changes. Multi-statement input (`SELECT 1; TRUNCATE t`) is supported if enabled on the server. The proxy buffers responses up to 64 MiB (502 with `clickhouse response exceeded N bytes` on overflow) and passes through ClickHouse's `Content-Type` for inline `FORMAT` directives. Structured queries and pipes use `clickhouse-go` native drivers for performance and consistent row-array shapes. ### Streaming Path diff --git a/docs/src/content/docs/claude-code.md b/docs/src/content/docs/claude-code.md index c418517f..98751231 100644 --- a/docs/src/content/docs/claude-code.md +++ b/docs/src/content/docs/claude-code.md @@ -5,124 +5,124 @@ sidebar: order: 12 --- -WaveHouse ships minimal team-wide [Claude Code](https://claude.com/claude-code) configuration. The repo commits only what's distinctly useful to every contributor; cosmetic and personal choices stay at the user level. +WaveHouse ships minimal team-wide [Claude Code](https://claude.com/claude-code) config: only what's useful to every contributor. Cosmetic and personal choices stay at the user level. -If you're new to Claude Code itself, the [official docs](https://code.claude.com/docs) cover the basics. This page is WaveHouse-specific. +New to Claude Code? The [official docs](https://code.claude.com/docs) cover basics; this page is WaveHouse-specific. ## Quick setup -1. **Install Claude Code**: follow the [official install guide](https://code.claude.com/docs/en/quickstart). On macOS: `brew install --cask claude-code`. +1. **Install Claude Code**: [official install guide](https://code.claude.com/docs/en/quickstart), or macOS `brew install --cask claude-code`. -2. **Authenticate**: log in to your Max subscription — Claude Code prompts you on first run. +2. **Authenticate**: log in to your Max subscription on first run. -3. **Bootstrap the repo**: `make tools`. This installs Go tools, pnpm deps, and **also configures git hooks** (`git config core.hooksPath .githooks`). Without this step, the team's pre-commit / pre-push gates won't fire. +3. **Bootstrap the repo**: `make tools` installs Go tools and pnpm deps and sets git hooks (`git config core.hooksPath .githooks`) for the pre-commit/pre-push gates. -4. **Optional — worktrunk**: install [worktrunk](https://worktrunk.dev) for parallel-agent worktree management. The team's project hooks live in `.config/wt.toml`. +4. **Optional — worktrunk**: [worktrunk](https://worktrunk.dev) manages parallel-agent worktrees via `.config/wt.toml`. ## How enforcement is layered | Layer | Lives in | Applies to | Purpose | | ----- | -------- | ---------- | ------- | -| **Git hooks** | `.githooks/` (installed by `make tools`) | Humans + Claude uniformly | Hard enforcement: `make verify` on commit; before push, `make ci` for a code change or `make verify` for a docs-only one (classifier-gated) | -| **Claude Code agent gate** | `.claude/hooks/agent-bash-gate.sh` (PreToolUse Bash) + `.claude/settings.json` deny rules | Agents only | Catches accidental violations of [Agent PR Discipline](#agent-pr-discipline): drafts only, no human reviewer adds, a marker (from a review or a logged skip) from every reviewer in `scripts/pre-push-reviewers.sh` required on any push to a non-main branch with commits ahead of `main`; PR title linted via `scripts/lint-pr-title.sh` on `gh pr create` / `gh pr edit --title` | -| **Claude Code ergonomic hooks** | `.claude/hooks/gofumpt-on-save.sh` (PostToolUse Edit/Write/MultiEdit), `.claude/hooks/review-marker.sh` (SubagentStop) | Claude only | gofumpt: auto-format on file edits (humans get this from their IDE). review-marker: on a reviewer's `VERDICT: ship_it`, writes that reviewer's `tmp/-passed-` marker | -| **Claude Code skills / agents / commands** | `.claude/skills/`, `.claude/agents/`, `.claude/commands/` | Claude only (when relevant) | Workflow guidance and on-demand helpers — not gates | +| **Git hooks** | `.githooks/` (installed by `make tools`) | Humans + Claude uniformly | Hard enforcement: `make verify` on commit; before push, `make ci` for code changes or `make verify` for docs-only (classifier-gated) | +| **Claude Code agent gate** | `.claude/hooks/agent-bash-gate.sh` (PreToolUse Bash) + `.claude/settings.json` deny rules | Agents only | Enforces [Agent PR Discipline](#agent-pr-discipline): drafts only, no human reviewer adds; a marker from every `scripts/pre-push-reviewers.sh` reviewer for pushes ahead of `main`; `scripts/lint-pr-title.sh` on `gh pr create` / `gh pr edit --title` | +| **Claude Code ergonomic hooks** | `.claude/hooks/gofumpt-on-save.sh` (PostToolUse Edit/Write/MultiEdit), `.claude/hooks/review-marker.sh` (SubagentStop) | Claude only | gofumpt: auto-format on edits. review-marker: writes `tmp/-passed-` on `VERDICT: ship_it` | +| **Claude Code skills / agents / commands** | `.claude/skills/`, `.claude/agents/`, `.claude/commands/` | Claude only (when relevant) | Workflow guidance and helpers — not gates | -Git hooks are the source of truth for "must pass before merge." `.claude/` layers agent-specific gates and ergonomic hooks on top; it doesn't substitute for the universal gates. +Git hooks are the source of truth for "must pass before merge"; `.claude/` adds agent-specific gates and ergonomics, never substitutes for them. ## Git hooks (`.githooks/`) -Two scripts, both committed to the repo: +Two committed scripts: | Hook | Behavior | | ---- | -------- | -| `pre-commit` | Runs `make verify` (tidy + fmt + vulncheck + lint, ~30s) — **blocks on failure**. Skipped if `make ci` or `make verify` already ran for the current tree state (cached via `scripts/ci-marker.sh`). | -| `pre-push` | Scales the bar to the change set (same classifier CI uses, `scripts/classify-paths.sh`): a **code** change requires the `make ci` marker (`tmp/ci-passed-tree-`); a **docs/prose-only** push requires only the `make verify` marker (`tmp/verify-passed-tree-`) — CI skips the Go/SDK suites for those too. **Blocks** if the required marker is absent. Fail-closed: an unclassifiable push falls back to requiring `make ci`. | +| `pre-commit` | Runs `make verify` (tidy, fmt, vulncheck, lint; ~30s). **Blocks on failure**. Skipped if `make ci`/`make verify` already ran for this tree state (cached via `scripts/ci-marker.sh`). | +| `pre-push` | Uses `scripts/classify-paths.sh`: **code** changes require the `make ci` marker (`tmp/ci-passed-tree-`); **docs/prose-only** pushes require the `make verify` marker (`tmp/verify-passed-tree-`). CI also skips Go/SDK suites for docs. **Blocks** if markers are absent; unclassifiable pushes default to requiring `make ci`. | -`--no-verify` is for intentional WIP / draft pushes. Agents should not use it — policy in AGENTS.md §"Agent PR Discipline", not regex-enforced. +Use `--no-verify` only for intentional WIP/drafts. Agents must not use it (see AGENTS.md §"Agent PR Discipline"). -Both markers are tree-keyed so commit-then-push works without a re-run when the tree is unchanged. `make ci` / `make verify` skip the marker write when `$CI` is set (CI runners don't push). Shared logic lives in `scripts/ci-marker.sh`. +Tree-keyed markers allow commit-then-push without re-running if the tree is unchanged. `make ci`/`verify` skip marker writes when `$CI` is set. Shared logic: `scripts/ci-marker.sh`. ## What's in `.claude/` and `.config/` | Path | Purpose | | ---- | ------- | -| `.claude/settings.json` | Team-wide: `deny` permissions (force-push / git reset --hard / filter-branch / update-ref -d, gh pr merge / ready / approve / request-changes, gh repo/release delete, gh secret delete, gh workflow disable, rm -rf / sudo rm), all three hooks wired | +| `.claude/settings.json` | Team-wide: `deny` permissions (force-push, git reset --hard, filter-branch, update-ref -d, gh pr merge/ready/approve/request-changes, gh repo/release delete, gh secret delete, gh workflow disable, rm -rf / sudo rm); all three hooks wired | | `.claude/hooks/gofumpt-on-save.sh` | PostToolUse Edit/Write/MultiEdit: auto-formats `.go` files | -| `.claude/hooks/agent-bash-gate.sh` | PreToolUse Bash: catches accidental Agent PR Discipline violations (drafts only, no human reviewer adds, a marker (from a review or a logged skip) from every reviewer in `scripts/pre-push-reviewers.sh` required on any push to a non-main branch with commits ahead of `main`; PR title linted via `scripts/lint-pr-title.sh` on `gh pr create` / `gh pr edit --title`) | -| `.claude/hooks/review-marker.sh` | SubagentStop: on a reviewer's `VERDICT: ship_it`, writes its `tmp/-passed-` marker (the reviewer set comes from `scripts/pre-push-reviewers.sh`). Filters by `agent_type` in-script (SubagentStop has no matcher). Reads `.last_assistant_message` (flat string) rather than PostToolUse:Agent's structured `tool_response` | -| `.claude/commands/cover.md` | `/cover [suite]` — suite dispatch + coverage threshold analysis | -| `.claude/commands/docs-review.md` | `/docs-review [path\|all]` — launches the `docs-reviewer` subagent. No-arg = the gating pre-push docs review; a path/`all` = advisory | -| `.claude/commands/prepush.md` | `/prepush [all]` — the pre-push gate: reads `scripts/pre-push-reviewers.sh`, runs the reviewers the change needs in parallel (fresh context), skips the rest on the record (`scripts/skip-pre-push-review.sh`), loops to `ship_it`. `all` forces the full set | -| `.claude/agents/pre-push-reviewer.md` | `pre-push-reviewer` subagent — canonical pre-push **code** review (one of the parallel pre-push reviewers in `scripts/pre-push-reviewers.sh`); also used for auditing others' PRs locally | -| `.claude/agents/docs-reviewer.md` | `docs-reviewer` subagent — docs-prose + code↔docs-sync review; **mandatory pre-push gate** (writes `tmp/docs-reviewer-passed-` on ship_it, in parallel with the other pre-push reviewers); advisory only for ad-hoc path/`all`. Scope via `scripts/docs-prose.sh` | -| `.claude/skills/pr-sync-with-main/SKILL.md` | "Fix this stale PR" workflow — merge origin/main, never rebase or force-push | -| `.claude/skills/pr-review-locally/SKILL.md` | "Review PR locally" workflow — `wt switch pr:` + the relevant reviewers (code, docs, …) in parallel, no PR comments | -| `.claude/skills/pm-triage/SKILL.md` | PM-review workflow — triage feedback / backlog / code TODOs and reconcile issue & PR status into well-scoped, tracked Task Board (project #7) issues; invoked as `/pm-triage` | -| `.claude/skills/integration-astro-view-transitions/` | Vendored PostHog-authored skill (installed by the PostHog wizard, v1.21.1) — reference patterns for the docs-site analytics: web snippet, ClientRouter view-transitions guard, user identify | -| `.claude/settings.local.json` | **Your personal overrides** — gitignored; put model choice, status line, allow lists, etc. here | +| `.claude/hooks/agent-bash-gate.sh` | PreToolUse Bash: blocks Agent PR Discipline violations (drafts only; markers from all `scripts/pre-push-reviewers.sh` reviewers for pushes ahead of `main`; titles linted by `scripts/lint-pr-title.sh` on `gh pr create`/`edit --title`) | +| `.claude/hooks/review-marker.sh` | SubagentStop: writes `tmp/-passed-` on `VERDICT: ship_it` (reviewer set from `scripts/pre-push-reviewers.sh`). Filters by `agent_type`; reads the `.last_assistant_message` string, not structured `tool_response` | +| `.claude/commands/cover.md` | `/cover [suite]` — suite dispatch and coverage threshold analysis | +| `.claude/commands/docs-review.md` | `/docs-review [path\|all]` — launches `docs-reviewer` subagent. No-arg = gating pre-push review; path/`all` = advisory | +| `.claude/commands/prepush.md` | `/prepush [all]` — pre-push gate: reads `scripts/pre-push-reviewers.sh`, runs required reviewers in parallel (fresh context), skips others via `scripts/skip-pre-push-review.sh`, loops to `ship_it`. `all` forces full set | +| `.claude/agents/pre-push-reviewer.md` | `pre-push-reviewer` subagent — canonical pre-push code review (via `scripts/pre-push-reviewers.sh`); also used for local PR auditing | +| `.claude/agents/docs-reviewer.md` | `docs-reviewer` subagent — docs-prose and code↔docs-sync review; mandatory pre-push gate (writes `tmp/docs-reviewer-passed-` on ship_it); advisory for ad-hoc paths. Scope via `scripts/docs-prose.sh` | +| `.claude/skills/pr-sync-with-main/SKILL.md` | "Fix this stale PR" workflow — merge origin/main; no rebase or force-push | +| `.claude/skills/pr-review-locally/SKILL.md` | "Review PR locally" workflow — `wt switch pr:` + parallel reviewers (code, docs, etc.); no PR comments | +| `.claude/skills/pm-triage/SKILL.md` | PM-review workflow (`/pm-triage`) — triage feedback/backlog/TODOs and reconcile issue/PR status into Task Board (project #7) issues | +| `.claude/skills/integration-astro-view-transitions/` | Vendored PostHog skill (v1.21.1) — docs-site analytics patterns: web snippet, ClientRouter view-transitions guard, user identify | +| `.claude/settings.local.json` | Personal overrides (gitignored): model choice, status line, allow lists, etc. | | `.config/wt.toml` | Worktrunk project hooks (post-start, pre-merge, pre-remove) | -Notably absent: no `.mcp.json`, no committed status line, no `permissions.allow` list. See [GitHub access](#github-access-gh-cli-vs-mcp) and [Permission posture](#permission-posture) below. +Absent: `.mcp.json`, a committed status line, a `permissions.allow` list — see [GitHub access](#github-access-gh-cli-vs-mcp) and [Permission posture](#permission-posture). ## Slash commands | Command | What it does | | ------- | ------------ | | `/cover [suite]` | Renders coverage for a suite (unit / integration / e2e / sdk / all / merge) and surfaces drops below threshold | -| `/docs-review [path\|all]` | Runs the `docs-reviewer` subagent — accuracy vs code, runnable examples, clarity, completeness, + code↔docs sync. No-arg gates the push (writes the docs marker on ship_it); a path/`all` is advisory. Complements misspell / markdownlint / links-validator | -| `/prepush [all]` | The mandatory pre-push self-review — judges which reviewers in `scripts/pre-push-reviewers.sh` the change needs, runs those in parallel (fresh context), skips the rest on the record (logged), and loops until each reaches `ship_it`. `all` forces the full set | +| `/docs-review [path\|all]` | Runs `docs-reviewer` subagent for accuracy, examples, clarity, completeness, and code↔docs sync. No-arg gates the push (writes docs marker on ship_it); path/`all` is advisory. Complements misspell / markdownlint / links-validator | +| `/prepush [all]` | Mandatory pre-push review: runs required `scripts/pre-push-reviewers.sh` in parallel until each reaches `ship_it`. `all` forces the full set | -To add a command: drop a `.md` file in `.claude/commands/`. Filename becomes the slash command. Frontmatter: `description` and `argument-hint`; body is the prompt with `$ARGUMENTS`. +To add commands, place a `.md` file in `.claude/commands/`. Filename is the command. Frontmatter requires `description` and `argument-hint`; body is the prompt using `$ARGUMENTS`. ## Subagents | Subagent | When to use | | -------- | ----------- | -| `pre-push-reviewer` | **Mandatory before pushing to a PR branch** (enforced by `.claude/hooks/agent-bash-gate.sh`), run in parallel with the other reviewers in `scripts/pre-push-reviewers.sh` — all must reach `ship_it`. Also used for auditing someone else's PR after `wt switch pr:`. Runs the canonical `.github/prompts/pr-review.md` workflow against the local branch in fresh context. Fetches PR comments + CI status + linked-issue acceptance criteria when on a PR branch. Returns `[MUST]`/`[SHOULD]`/`[MAY]` findings + a parseable `VERDICT: ship_it\|iterate\|block` line that drives the `tmp/pre-push-reviewer-passed-` marker. | -| `docs-reviewer` | **Mandatory before pushing to a PR branch** — runs in parallel with the other pre-push reviewers (all enforced by `.claude/hooks/agent-bash-gate.sh`). Reviews docs **prose** (accuracy-vs-code, runnable examples, clarity, completeness) **and code↔docs sync** (code that changed but whose docs didn't), using `.github/prompts/docs-review.md` over the `scripts/docs-prose.sh` denylist set (Starlight site + governance docs incl. the SDK readme). Default (branch) scope emits `VERDICT: ship_it\|iterate\|block` → writes `tmp/docs-reviewer-passed-`; a path/`all` is advisory (no marker). Posts no PR comments, never edits docs. Complements misspell / markdownlint / starlight-links-validator, never duplicates them. | +| `pre-push-reviewer` | **Mandatory before pushing to PR branches** (enforced by `.claude/hooks/agent-bash-gate.sh`), run in parallel via `scripts/pre-push-reviewers.sh`; all must reach `ship_it`. Also used for auditing others' PRs after `wt switch pr:`. Runs `.github/prompts/pr-review.md` in fresh context, fetching PR comments, CI status, and linked-issue criteria. Returns `[MUST]`/`[SHOULD]`/`[MAY]` findings plus a parseable `VERDICT: ship_it\|iterate\|block` driving the `tmp/pre-push-reviewer-passed-` marker. | +| `docs-reviewer` | **Mandatory before pushing to PR branches** (enforced by `.claude/hooks/agent-bash-gate.sh`), run in parallel via `scripts/pre-push-reviewers.sh`. Reviews prose accuracy, runnable examples, clarity, completeness, and code↔docs sync via `.github/prompts/docs-review.md` over the `scripts/docs-prose.sh` denylist (Starlight site + governance docs including SDK readme). Branch scope emits `VERDICT: ship_it\|iterate\|block` $\rightarrow$ `tmp/docs-reviewer-passed-`; path/`all` is advisory. Never comments on PRs or edits docs. Complements misspell, markdownlint, and starlight-links-validator. | Invoke via the `Agent` tool with `subagent_type: pre-push-reviewer`, or via `/agents`. -To add a subagent: drop a `.md` file in `.claude/agents/`. Frontmatter: `description` (used by main Claude to decide when to delegate), `tools`, `model`. Body is the system prompt. To make a reviewer a **mandatory pre-push gate** (the way `pre-push-reviewer` and `docs-reviewer` are), also add its name to `scripts/pre-push-reviewers.sh` — the single source of truth the hooks and `/prepush` read, so nothing else needs editing. See AGENTS.md §"Adding a pre-push reviewer". +To add one: a `.md` file in `.claude/agents/` with frontmatter (`description`, `tools`, `model`) and a system-prompt body. For a **mandatory pre-push gate**, add its name to `scripts/pre-push-reviewers.sh` (single source of truth for hooks and `/prepush`) — AGENTS.md §"Adding a pre-push reviewer". ## Skills -Skills load automatically into Claude's context when conversation patterns match their `description`. +Skills load into Claude's context when conversation patterns match their `description`. | Skill | Triggers on | | ----- | ----------- | -| `pr-sync-with-main` | When a PR shows "out-of-date with base branch", or a user asks to "fix the PR" / "sync with main". Documents the merge-not-rebase procedure and the WaveHouse-specific reason long-lived branches need it. | -| `pr-review-locally` | When a user asks to "review PR ", "audit PR ", "look at PR " — pulls the PR down via `wt switch pr:` (or `gh pr checkout`), runs the relevant reviewers (code, docs, …) in parallel in fresh context, surfaces their combined findings without commenting on the PR. | -| `pm-triage` | When a user asks to triage dogfooding feedback, re-prioritize the backlog (P0–P3), sweep code TODOs, or check that work is tracked — runs a PM-style review against the Task Board (project #7) and proposes well-scoped issues. Invoked as `/pm-triage`. | -| `integration-astro-view-transitions` | PostHog work on the Astro docs site (snippet patterns, view-transitions guard, identify). Vendored PostHog-authored content — wizard-installed, lightly edited only for markdownlint; the live setup (relay host, committed fallback) is documented in `PostHog.astro` itself and the CHANGELOG. | +| `pr-sync-with-main` | PR "out-of-date with base branch", or "fix the PR"/"sync with main". Documents the merge-not-rebase procedure and why WaveHouse has long-lived branches. | +| `pr-review-locally` | Requests to "review", "audit", or "look at PR ". Pulls it via `wt switch pr:` (or `gh pr checkout`), runs reviewers (code, docs, …) in parallel contexts, surfaces combined findings, comments on nothing. | +| `pm-triage` | Requests to triage dogfooding feedback, re-prioritize backlog (P0–P3), sweep TODOs, or check tracking. Runs PM review against Task Board (project #7), proposes scoped issues. `/pm-triage`. | +| `integration-astro-view-transitions` | PostHog Astro docs work (snippet patterns, view-transitions guard, identify). Vendored, wizard-installed, edited for markdownlint; live setup (relay host, committed fallback) lives in `PostHog.astro` and the CHANGELOG. | -To add a skill: create `.claude/skills//SKILL.md` with frontmatter `name` + `description` and the workflow body. Description quality matters — that's what Claude matches against to load the skill. +To add one: `.claude/skills//SKILL.md` with frontmatter `name` + `description` and the workflow body. Matching quality follows description quality. ## Agent PR Discipline -Agents (Claude Code etc.) have additional gating beyond what humans face — enforced by `.claude/hooks/agent-bash-gate.sh` (PreToolUse Bash) + deny rules in `.claude/settings.json`. Humans keep full git/gh affordances; agents have these extra constraints: +Agents get extra gating from `.claude/hooks/agent-bash-gate.sh` (PreToolUse Bash) and `.claude/settings.json` deny rules. Humans keep full git/gh affordances; agents don't: -- **Drafts only.** `gh pr create` must include `--draft`. Only humans transition draft → ready (`gh pr ready` is blocked), approve (`gh pr review --approve` is blocked), or request changes (`gh pr review --request-changes` is blocked). The gate also lints the PR title on `gh pr create` / `gh pr edit --title` via `scripts/lint-pr-title.sh` — the same rule as the required `CI` check's `PR title` job, so a malformed title is caught before the PR exists (fail-open when no quoted title is parseable). -- **No human reviewer assignment.** `gh pr edit --add-reviewer / --add-assignee` and `POST /requested_reviewers` are blocked. GitHub assigns reviewers natively — the `required_reviewers` ruleset rule requests the `@Wave-RF/wavehouse-admins` team and the team's code-review assignment picks the member; humans handle the rest. -- **Bot re-triggers via comments.** Agents CAN mention bots in PR comments to re-trigger reviews — `@coderabbitai review`, etc. This goes through `gh pr comment` (allowed), not the reviewer API. -- **Pre-push review required on PR branches.** Before `git push` to a branch with commits ahead of `main`, the agent runs the reviewers in `scripts/pre-push-reviewers.sh` (today: `pre-push-reviewer` for code, `docs-reviewer` for docs prose + code↔docs sync) that the change needs — in fresh context, in parallel — and **skips** any with nothing to do via `scripts/skip-pre-push-review.sh ""` (a logged skip that satisfies the marker, so a docs typo doesn't pay for a full code review). `/prepush` does all of this. `ship_it` requires zero findings at any severity — any `[MUST]` / `[SHOULD]` / `[MAY]` forces iterate; fix and re-invoke (always fresh context) until clean. The push gate requires a marker — from a `ship_it` or a logged skip — from every listed reviewer, and echoes the skips at push time. -- **Don't bypass.** `--no-verify` and hand-writing markers are policy violations, not regex-blocked. An agent that wants to bypass can edit the gate itself — trust the policy in AGENTS.md §"Agent PR Discipline". Markers come from `make ci`, the `review-marker.sh` hook, and `scripts/skip-pre-push-review.sh` (logged skips); nothing else. -- **PR reviews on others' PRs stay local.** Use the `pr-review-locally` skill for local-only audits — the reviewers' findings go to you, not the PR. +- **Drafts only.** `gh pr create` must include `--draft`. Only humans may flip draft → ready (`gh pr ready`), approve (`gh pr review --approve`), or request changes (`gh pr review --request-changes`). The gate lints titles with `scripts/lint-pr-title.sh` on `gh pr create` / `gh pr edit --title`, mirroring the CI `PR title` job (fail-open if no quoted title parses). +- **No human reviewer assignment.** `gh pr edit --add-reviewer / --add-assignee` and `POST /requested_reviewers` are blocked; the `required_reviewers` ruleset requests `@Wave-RF/wavehouse-admins`, and humans handle the rest. +- **Bot re-triggers.** Agents may mention bots (e.g., `@coderabbitai review`) via `gh pr comment`. +- **Pre-push reviews.** Before `git push` on a branch ahead of `main`, agents run every reviewer in `scripts/pre-push-reviewers.sh` (currently `pre-push-reviewer` for code, `docs-reviewer` for docs) in parallel, in fresh context; `/prepush` automates it. Log skips for irrelevant changes with `scripts/skip-pre-push-review.sh ""`. `ship_it` needs zero findings — any `[MUST]` / `[SHOULD]` / `[MAY]` means iterate until clean. The push gate wants a marker from every listed reviewer (`ship_it` or logged skip) and echoes skips at push time. +- **No bypass.** `--no-verify` or forged markers violate AGENTS.md §"Agent PR Discipline". Valid markers come only from `make ci`, the `review-marker.sh` hook, and `scripts/skip-pre-push-review.sh`. +- **Local reviews.** Use the `pr-review-locally` skill for audits; findings remain local and are not posted to the PR. -Full ruleset and rationale: AGENTS.md §"Agent PR Discipline". +Full ruleset: AGENTS.md §"Agent PR Discipline". ## Worktrunk integration -We use [worktrunk](https://worktrunk.dev) as the recommended worktree manager. It creates real `git worktree add` directories, so the committed `.claude/` config Just Works in every worktree. +We recommend [worktrunk](https://worktrunk.dev): it uses `git worktree add`, so committed `.claude/` configs work in every worktree. Project hooks in `.config/wt.toml`: | Hook | Command | Why | | ---- | ------- | --- | -| `post-start` | `make tools` | Bootstraps the new worktree: tools, modules, pnpm deps, **and git hooks** | +| `post-start` | `make tools` | Bootstraps tools, modules, pnpm deps, and git hooks | | `pre-merge` | `make verify` | Fast pre-merge gate (same as pre-commit) | -| `pre-remove` | `git status --short` | Surfaces uncommitted work before tearing down | +| `pre-remove` | `git status --short` | Surfaces uncommitted work before teardown | Common workflow: @@ -134,17 +134,17 @@ wt merge # squash + rebase + merge + clean up wt remove feat/new-thing # tear down (warns on uncommitted work) ``` -User-specific worktrunk config goes in `~/.config/worktrunk/config.toml`; the committed `.config/wt.toml` has team-wide hooks only. +User config: `~/.config/worktrunk/config.toml`; committed `.config/wt.toml` holds team-wide hooks only. ## GitHub access: gh CLI vs MCP -**We use `gh` CLI as the canonical GitHub access path**, not a GitHub MCP server. Reasons: +**Use `gh` CLI as the canonical GitHub access path**, not a GitHub MCP server, because: -- `gh` is already a hard dev requirement +- `gh` is a hard dev requirement - Works identically in Claude Code, terminal, and shell scripts -- No extra auth / approval / npx cold-start +- No extra auth, approval, or npx cold-start -The GitHub MCP server is useful for cross-repo code search and bulk graph queries, but neither is a daily WaveHouse pattern. If you want it, add at user level: +The GitHub MCP server suits cross-repo search and bulk graph queries — neither a daily WaveHouse pattern. To add it at user level: ```jsonc title="~/.claude.json" // ~/.claude.json — your user-level config @@ -163,40 +163,38 @@ Then `export GITHUB_TOKEN=$(gh auth token)` in your shell rc. ## Other useful MCP servers (user-level, optional) -None of these are committed at project level — pick what you actually use. +Not committed at project level; pick what you use. -- **[Grafana MCP](https://github.com/grafana/mcp-grafana)** — for querying Prometheus / Loki / Tempo / Pyroscope from Claude when debugging observability work. Useful if you touch `internal/observability/`. -- **ClickHouse MCP** (community) — direct schema introspection + query against your local `make dev` ClickHouse. Useful for `internal/discovery/` and ingest work, but `make deps-shell` (clickhouse-client REPL) is often enough. +- **[Grafana MCP](https://github.com/grafana/mcp-grafana)** — query Prometheus / Loki / Tempo / Pyroscope from Claude for observability work in `internal/observability/`. +- **ClickHouse MCP** (community) — schema introspection and queries against local `make dev` ClickHouse; useful for `internal/discovery/` and ingest, though `make deps-shell` usually suffices. -When [issue #121](https://github.com/Wave-RF/WaveHouse/issues/121) lands a SigNoz dev stack with `make dev-obs`, Grafana MCP pointed at that dev environment will become a natural choice for trace / log inspection. +Once [issue #121](https://github.com/Wave-RF/WaveHouse/issues/121) adds a SigNoz dev stack via `make dev-obs`, Grafana MCP becomes ideal for trace/log inspection. ## Permission posture -`.claude/settings.json` is structured around the reality that most contributors run Claude Code in `bypassPermissions` (yolo). In that mode: +`.claude/settings.json` assumes most contributors use `bypassPermissions` mode: -- ✅ `permissions.deny` **still fires** — the only programmatic guardrail -- ❌ `permissions.allow` / `permissions.ask` are moot (everything's already allowed) -- ✅ Hooks (Claude Code AND git) **still fire** — the behavioral layer +- ✅ `permissions.deny`: The only programmatic guardrail; still fires. +- ❌ `permissions.allow` / `permissions.ask`: Moot (everything is allowed). +- ✅ Hooks (Claude Code and git): Still fire as the behavioral layer. -So the file is `deny`-only by design. Personal allow / ask lists go in `.claude/settings.local.json` (gitignored). - -`defaultMode` is intentionally not set in committed config. Each user picks their preferred mode. +`deny`-only by design. Personal allow/ask lists go in `.claude/settings.local.json` (gitignored); `defaultMode` is omitted so users choose. The deny list blocks: | Blocked | Why | | ------- | --- | -| `git push --force`, `-f`, `--force-with-lease` | Force-pushing is destructive; also loses inline review-comment anchors | -| `git reset --hard origin:*` | Throws away local work | +| `git push --force`, `-f`, `--force-with-lease` | Destructive; loses inline review-comment anchors | +| `git reset --hard origin:*` | Discards local work | | `git filter-branch`, `git update-ref -d` | History-rewriting / ref destruction | | `gh pr merge`, `gh repo delete`, `gh release delete` | Irreversible / shared-state | -| `gh pr ready`, `gh pr review --approve` / `-a` / `--request-changes` / `-r` | [Agent PR Discipline](#agent-pr-discipline) — draft→ready and review verdicts are human actions | +| `gh pr ready`, `gh pr review --approve` / `-a` / `--request-changes` / `-r` | [Agent PR Discipline](#agent-pr-discipline) — draft→ready and reviews are human actions | | `gh workflow disable`, `gh secret delete` | Operational footguns | | `rm -rf /`, `rm -rf ~`, `rm -rf $HOME`, `sudo rm` | Filesystem destruction | ## Status line, output style, model -Not committed at project level. Personal preference — put in `.claude/settings.local.json`: +Personal preference; keep in `.claude/settings.local.json` (not committed): ```jsonc title=".claude/settings.local.json" { @@ -208,36 +206,36 @@ Not committed at project level. Personal preference — put in `.claude/settings ## Daily workflow -1. Write code (gofumpt-on-save formats Go files as you go). -2. `git commit` → pre-commit hook runs `make verify` (or skips if `make ci` already validated this tree). Fix anything that fails. -3. `git push` → the pre-push hook blocks until the tree is validated for what changed — `make ci` for a code change, `make verify` for a docs/prose-only one (run it, fix, retry). For agents, the agent-bash-gate hook also requires a marker for HEAD from every reviewer in `scripts/pre-push-reviewers.sh` on any push of a branch with commits ahead of `main` — including the first push, before the PR exists. Run `/prepush` → it judges which reviewers the change needs, runs those in parallel (on Ship it each marker auto-writes), and skips the rest on the record (`skip-pre-push-review.sh` writes their markers + logs why) → push succeeds. On Iterate/Block, fix, re-invoke (fresh context each time), repeat. -4. Open the PR with `gh pr create --draft` (agents required to use `--draft`; humans flip to ready when ready). -5. CI workflows fire on the new HEAD. Address review comments per AGENTS.md §Review Response. +1. Write code (gofumpt-on-save formats Go files). +2. `git commit`: pre-commit hook runs `make verify` (unless `make ci` already validated the tree). Fix failures. +3. `git push`: the pre-push hook blocks until `make ci` (code) or `make verify` (docs/prose) passes. For agents on a branch ahead of `main`, `agent-bash-gate` also requires markers from every reviewer in `scripts/pre-push-reviewers.sh`. `/prepush` picks and runs them in parallel (Ship it auto-writes markers; `skip-pre-push-review.sh` logs skips); on Iterate/Block, fix and re-invoke. +4. Open PR via `gh pr create --draft` (agents must use `--draft`; humans flip to ready later). +5. CI workflows fire on HEAD. Address comments per AGENTS.md §Review Response. -Helpers along the way: +Helpers: -- Stale PR ("out-of-date with base branch") → ask Claude to "fix the PR" (loads `pr-sync-with-main` skill — merges main, doesn't rebase or force-push). -- Reviewing someone else's PR → "review PR 120 locally" / "audit PR 120" (loads `pr-review-locally` skill — `wt switch pr:120` + the relevant reviewers in parallel, no PR comments). -- Re-trigger a bot reviewer → ask Claude to "ping coderabbit again" — Claude posts the appropriate `@` comment on the PR. -- Coverage check on a specific suite → `/cover unit` (or `integration`, `e2e`, `sdk`, `all`). +- Stale PR: "fix the PR" (`pr-sync-with-main` skill merges main; no rebase/force-push). +- Reviewing others: "review PR 120 locally" / "audit PR 120" (`pr-review-locally` skill runs `wt switch pr:120` + reviewers in parallel, no comments). +- Re-trigger bot reviewer: "ping coderabbit again" posts a `@` comment. +- Coverage check: `/cover unit`, `integration`, `e2e`, `sdk`, or `all`. -**Memory**: Claude Code maintains per-project memory in `~/.claude/projects//memory/`. `AGENTS.md` is the SHARED source of truth; memory is for personal observations / preferences that don't belong in committed config. +**Memory**: per-project memory lives in `~/.claude/projects//memory/`. `AGENTS.md` is the shared source of truth; memory holds personal notes not in committed config. ## Extending -The `.claude/` layout is intentionally small. Add things when they earn their keep: +Keep `.claude/` small. Add only as needed: -- New slash commands as recurring workflows emerge -- New subagents for specialized one-shot tasks -- New skills for context-loaded workflow patterns -- New Claude Code hooks for edit-time UX gaps (NOT for enforcement — gates go in `.githooks/` so they apply to humans too) -- Adjust `.githooks/pre-commit` / `pre-push` if the gates feel wrong +- Slash commands for recurring workflows +- Subagents for specialized one-shot tasks +- Skills for context-loaded patterns +- Claude Code hooks for edit-time UX (not enforcement; use `.githooks/` for human-applicable gates) +- Adjust `.githooks/pre-commit` or `pre-push` if gates are incorrect -If you build something useful, commit it and update this doc. +Commit useful additions and update this doc. ## Reference -- [AGENTS.md](https://github.com/Wave-RF/WaveHouse/blob/main/AGENTS.md) — project conventions, architecture, code style, doc-sync rules, SDK-sync rules, branch maintenance. Source of truth for all AI agent work. -- [Claude Code docs](https://code.claude.com/docs) — official Claude Code reference. -- [worktrunk.dev](https://worktrunk.dev) — worktree manager. -- `.github/prompts/pr-review.md` — the canonical review prompt that `pre-push-reviewer` runs locally. +- [AGENTS.md](https://github.com/Wave-RF/WaveHouse/blob/main/AGENTS.md) — Source of truth for AI agent work: conventions, architecture, style, doc/SDK-sync rules, branch maintenance. +- [Claude Code docs](https://code.claude.com/docs) — Official reference. +- [worktrunk.dev](https://worktrunk.dev) — Worktree manager. +- `.github/prompts/pr-review.md` — Canonical `pre-push-reviewer` local prompt. diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 87aa98e6..b5dc77c0 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -9,7 +9,7 @@ How to run WaveHouse in production — single binary, Docker images, releases, h ## Single binary -WaveHouse runs as one process with embedded NATS and optional Pebble dedup. The only external dependency is ClickHouse. +WaveHouse runs as one process with embedded NATS and optional Pebble dedup; ClickHouse is the only external dependency. ### Quick Start with Docker Compose @@ -38,10 +38,7 @@ curl -X POST "http://localhost:8080/v1/ingest?table=clicks" \ -d '{"page": "/home", "button": "signup", "score": 42.5}' ``` -This starts: - -- **ClickHouse** on ports 8123 (HTTP) and 9000 (native) -- **WaveHouse** on port 8080 +This starts ClickHouse (ports 8123 HTTP, 9000 native) and WaveHouse (port 8080). ### Binary @@ -53,7 +50,7 @@ make build ./bin/wavehouse ``` -Or override any config with environment variables: +Override config via environment variables: ```bash WH_CH_ADDR=clickhouse.example.com:9000 \ @@ -69,9 +66,7 @@ WH_SCHEMA_REFRESH_INTERVAL=30 \ docker build -f deployments/Dockerfile -t wavehouse:latest . ``` -This builds the runtime image `wavehouse:latest`. (The published `ghcr.io` images are built by GoReleaser from `deployments/Dockerfile.goreleaser`, not this command — see Registry below.) - -All images use multi-stage builds (Go Alpine builder → distroless runtime) for minimal attack surface. +Builds runtime image `wavehouse:latest`. Published `ghcr.io` images use GoReleaser and `deployments/Dockerfile.goreleaser`. All images use multi-stage builds (Go Alpine builder → distroless runtime) for minimal attack surface. ### Registry @@ -81,7 +76,7 @@ Production images are published to GitHub Container Registry via GoReleaser: ghcr.io/wave-rf/wavehouse: ``` -Published images carry a signed [Sigstore](https://www.sigstore.dev/) build-provenance attestation (stored in the registry). Verify one before deploying: +Images have signed [Sigstore](https://www.sigstore.dev/) build-provenance attestations. Verify before deploying: ```bash gh attestation verify oci://ghcr.io/wave-rf/wavehouse: --repo Wave-RF/WaveHouse @@ -89,7 +84,7 @@ gh attestation verify oci://ghcr.io/wave-rf/wavehouse: --repo Wave-RF/WaveH ## Releases -Releases are built with [GoReleaser](https://goreleaser.com/). The configuration is in `.goreleaser.yaml`. The release archives attached to each GitHub Release carry a signed [Sigstore](https://www.sigstore.dev/) build-provenance attestation — verify a downloaded archive with `gh attestation verify --repo Wave-RF/WaveHouse`. (This covers the prebuilt archives, not `go install`, which compiles from source.) +Releases use [GoReleaser](https://goreleaser.com/) via `.goreleaser.yaml`. GitHub Release archives include signed [Sigstore](https://www.sigstore.dev/) build-provenance attestations; verify with `gh attestation verify --repo Wave-RF/WaveHouse`. This applies to prebuilt archives, not source-compiled `go install`. ### Supported Platforms @@ -102,7 +97,7 @@ Releases are built with [GoReleaser](https://goreleaser.com/). The configuration ### Creating a Release -Tag and push to trigger the release workflow: +Tag and push to trigger the workflow: ```bash git tag v0.1.0 @@ -111,9 +106,9 @@ git push origin v0.1.0 ## Environment Variables -All configuration can be set via environment variables. This is the recommended approach for container deployments. See [Configuration Reference](/configuration) for the full list. +Configuration via environment variables is recommended for container deployments. See [Configuration Reference](/configuration) for the full list. -Key variables for production: +Key production variables: ```bash # Required @@ -170,18 +165,18 @@ WH_DLQ_ENABLED=true # Dead Letter Queue for failed inserts ## Persistent Storage (REQUIRED for containers) -WaveHouse keeps all embedded state under a single configurable root, `WH_DATA_DIR` (yaml: `data_dir`). Subdirectories are convention, not config: +WaveHouse stores embedded state under a configurable root, `WH_DATA_DIR` (yaml: `data_dir`). Subdirectories are convention-based: -- `/nats` — embedded NATS JetStream. Holds in-flight events between an ingest POST and the ingest worker → ClickHouse flush, plus the `mq.gap_window_minutes` window of history that powers SSE gap-fill across restarts. -- `/pebble` — Pebble dedup KV. Only used when `WH_DEDUPE_ENABLED=true`. +- `/nats`: Embedded NATS JetStream. Holds in-flight events between ingest POST and ClickHouse flush, plus the `mq.gap_window_minutes` history for SSE gap-fill across restarts. +- `/pebble`: Pebble dedup KV. Used only when `WH_DEDUPE_ENABLED=true`. -In a Docker / Podman / Kubernetes deployment, **`data_dir` must resolve to a host-backed volume**. The reference compose file `deployments/compose/standalone.yaml` sets `WH_DATA_DIR=/app/data` and binds a `wavehouse-data:/app/data` volume — copy that pattern. The bundled Dockerfiles pre-create `/app/data` and `/app/pipes` owned by the nonroot user (UID 65532); the binary creates the `nats/` and `pebble/` subdirectories under `/app/data` itself on first run. +In Docker, Podman, or Kubernetes, **`data_dir` must resolve to a host-backed volume**. Follow the pattern in `deployments/compose/standalone.yaml`, which sets `WH_DATA_DIR=/app/data` and binds a `wavehouse-data:/app/data` volume. Dockerfiles pre-create `/app/data` and `/app/pipes` owned by the nonroot user (UID 65532); the binary creates `nats/` and `pebble/` subdirectories on first run. -If `data_dir` resolves into the container's writable overlay layer instead, **JetStream state is wiped on every restart**: in-flight events are lost, gap-fill stops bridging restarts, and disk usage accumulates inside `/var/lib/docker` instead of the volume the operator chose. +If `data_dir` resides in the container's writable overlay layer, **JetStream state is wiped on every restart**: in-flight events are lost, gap-fill fails, and disk usage accumulates in `/var/lib/docker`. -Beyond persistence, the *speed* of that volume matters: JetStream `fsync`s every event to `/nats` before the ingest endpoint returns `200`, so the volume's `fsync` latency is your ingest latency floor. Managed cloud block storage handles this without thinking; commodity or virtualized substrates (ZFS without a SLOG, qcow2-on-`ext4`, spinning disks) can stall ingest with multi-second `fsync` tails. See [Durability & Storage](/durability) to measure yours before going live. +Volume speed is critical: JetStream `fsync`s every event to `/nats` before the ingest endpoint returns `200`. Volume `fsync` latency defines your ingest latency floor. Managed cloud block storage is typically sufficient; however, commodity or virtualized substrates (e.g., ZFS without SLOG, qcow2-on-`ext4`, spinning disks) may cause multi-second `fsync` stalls. See [Durability & Storage](/durability) to measure performance. -WaveHouse runs a simple existence check on startup and logs a `WARN` if `/nats` (or `/pebble` when dedupe is on) is missing or empty: +WaveHouse logs a `WARN` if `/nats` (or `/pebble` when dedupe is on) is missing or empty at startup: ```text wrap=false WARN data directory does not exist — starting with no prior state. @@ -189,20 +184,20 @@ WARN data directory does not exist — starting with no prior state. persisting; verify your mount. ``` -On a first-ever run this is expected. On every subsequent run it should be silent — so when this warning *does* fire after a redeploy, that's the most direct signal that the persistent volume isn't actually persisting. +This is expected on the first run but signals a persistence failure if it occurs after a redeploy. ### Distroless Permission Traps (named volume vs bind mount) -WaveHouse images run as the distroless `nonroot` user (UID 65532). Bind mounts and named volumes interact with this differently, and the distroless image has no shell to `chown` things at runtime — so getting the host side wrong produces a hard-to-read permission error from NATS or Pebble at startup. +Images run as the distroless `nonroot` user (UID 65532). Because distroless images lack a shell to `chown` at runtime, incorrect host permissions cause NATS or Pebble startup errors. -**Named volumes** (the recommended pattern): +**Named volumes** (recommended): ```yaml volumes: - wavehouse-data:/app/data ``` -On first attach to an empty named volume, Docker performs a "copy-up": the contents and ownership of `/app/data` *from the image* are copied into the volume. The bundled `Dockerfile` and `Dockerfile.goreleaser` both pre-create `/app/data` and `/app/pipes` with `chown -R 65532:65532`, so the volume inherits the right ownership automatically. **No host-side `chown` needed.** Subsequent restarts reuse whatever's in the volume. +On first attach, Docker performs a "copy-up" of the image's `/app/data` contents and ownership. Since `Dockerfile` and `Dockerfile.goreleaser` pre-create these with `chown -R 65532:65532`, the volume inherits correct ownership automatically. **No host-side `chown` is needed.** **Bind mounts** (host directory): @@ -211,7 +206,7 @@ volumes: - /srv/wavehouse:/app/data ``` -Bind mounts do **not** copy-up — Docker exposes the host directory as-is, and the image's pre-created dir is masked entirely. If `/srv/wavehouse` is owned by `root:root` on the host (the default for a freshly `mkdir`'d directory), the binary fails at startup with a permission error from NATS: +Bind mounts do not copy-up; they expose the host directory as-is. If `/srv/wavehouse` is owned by `root:root`, the binary fails with a permission error: ```text wrap=false ERROR mq init failed error="..." path=/app/data/nats @@ -219,27 +214,27 @@ ERROR mq init failed error="..." path=/app/data/nats directory must be owned by UID 65532..." ``` -The fix is one host-side command before first start: +UID 65532 is the canonical distroless `nonroot` user; the number works whether or not your host has a matching name in `/etc/passwd`. The error log carries this hint, so copy the suggested `chown` and re-run. Fix before starting: ```bash sudo mkdir -p /srv/wavehouse sudo chown -R 65532:65532 /srv/wavehouse ``` -UID 65532 is the canonical distroless `nonroot` user; the same number works regardless of whether your host has a matching name in `/etc/passwd`. The error log includes this remediation hint, so if you see "permission denied" at startup, copy the suggested `chown` command and re-run. +UID 65532 is the canonical distroless `nonroot` user. If you see "permission denied," use the suggested `chown` command from the logs. -**Pipes bind mount** follows the same rule — but mount it **read-only** since pipes is a seed, not state: +**Pipes bind mounts** follow the same rules but should be **read-only**, as pipes are seeds, not state: ```yaml volumes: - ./my-pipes:/app/pipes:ro # :ro is intentional, see below ``` -Read-only mounts don't need write permission for the container user, so `chown` isn't strictly required — but matching ownership keeps everything consistent. +Read-only mounts do not strictly require `chown` for the container user, though matching ownership maintains consistency. ## Pipes Bootstrap (optional, read-only) -Named query pipes live in NATS KV (`WAVEHOUSE_PIPES`). On first run, you can seed them from `.sql` files by setting `WH_PIPES_DIR` and bind-mounting the directory **read-only**: +Named query pipes reside in NATS KV (`WAVEHOUSE_PIPES`). Seed them from `.sql` files by setting `WH_PIPES_DIR` and bind-mounting the directory **read-only**: ```yaml services: @@ -251,56 +246,54 @@ services: - ./my-pipes:/app/pipes:ro # ← read-only seed ``` -The directory is a *seed*, not authoritative storage: after bootstrap, the API + KV are the source of truth. Runtime pipe edits go through `PUT /v1/admin/pipes/{name}`, not by editing the files. The `:ro` mount makes that contract explicit and prevents accidental writes from confusing future readers. Empty default (`WH_PIPES_DIR=""`) skips bootstrap entirely — most users will create pipes via the API. +This directory is a seed; thereafter, the API and KV are authoritative. Edit runtime pipes via `PUT /v1/admin/pipes/{name}`, not files. The `:ro` mount prevents accidental writes. Setting `WH_PIPES_DIR=""` skips bootstrap. ## Health Checks -API servers in standalone mode expose liveness and readiness endpoints under the Kubernetes-convention names `/livez` and `/readyz`: - -- `GET /livez` — Liveness probe. Returns 200 once the gateway has discovered ClickHouse table schemas at least once. Returns 503 with a diagnostic body while the boot-time schema discovery retry loop is still running (e.g. ClickHouse unreachable, target database missing). After successful boot, `/livez` stays 200 — transient ClickHouse blips at runtime are reflected in `/readyz`, not `/livez`. -- `GET /readyz` — Readiness probe. Returns 200 if the gateway is fully booted and ClickHouse is currently reachable, 503 otherwise. +Standalone API servers expose Kubernetes-convention endpoints: -`/healthz` remains registered as a **permanent alias** of `/livez` (it's the most widely-recognized name); `/health` and `/ready` are **deprecated aliases** for the v0.1.x line and will be removed in v0.2.0. Point new deployments at the `/livez` / `/readyz` names. +- `GET /livez` — Liveness probe. Returns 200 after the gateway discovers ClickHouse table schemas once. Returns 503 with diagnostics during boot-time schema discovery retries (e.g., ClickHouse unreachable). After boot, it remains 200; runtime ClickHouse blips affect `/readyz` only. +- `GET /readyz` — Readiness probe. Returns 200 if the gateway is booted and ClickHouse is reachable; otherwise 503. -Configure your load balancer or orchestrator to use these endpoints. +`/healthz` is a **permanent alias** of `/livez`. `/health` and `/ready` are **deprecated aliases** for v0.1.x and will be removed in v0.2.0. Use `/livez` and `/readyz` for new deployments. -**Exposure.** Probes share the API server's port (`:8080`) — kubelet probes the container internally, so there's no separate-port convention for them (metrics are the signal that optionally gets its own `prometheus.port`). If you forward `:8080` to the public internet the probe paths become reachable. The **recommended** posture is to keep `/livez`/`/readyz`/`/healthz` to internal callers and expose only **`/v1/health`** publicly (the SDK's content-free liveness ping, which never touches ClickHouse). `/readyz` issues a ClickHouse `Ping` on every call, so a public `/readyz` lets an unauthenticated flood become per-request backend pings, and the bare probes leak boot/readiness state — keeping them internal is a [reverse-proxy/ingress concern](/reverse-proxy#health-probes), and your orchestrator reaches them the internal way (kubelet on the container, LB on the backend) regardless. +**Exposure.** Probes use the API server port (`:8080`). Metrics optionally use `prometheus.port`. If `:8080` is public, probe paths are reachable. **Recommended:** keep `/livez`, `/readyz`, and `/healthz` internal; expose only **`/v1/health`** publicly (a content-free ping that doesn't touch ClickHouse). `/readyz` issues a ClickHouse `Ping` per call, so a public `/readyz` turns an unauthenticated flood into per-request backend pings, and bare probes leak boot state. Internal routing is a [reverse-proxy/ingress concern](/reverse-proxy#health-probes); orchestrators reach probes internally via kubelet or LB. ### Boot-time degraded mode -If ClickHouse is unreachable when WaveHouse starts (connection refused, missing database, DNS failure, etc.), the gateway no longer exits — it binds `:8080` and serves `/livez` 503 with the latest schema-discovery error as the diagnostic. Schema discovery retries in the background with exponential backoff (2s → 60s cap). Once a Refresh succeeds, `/livez` flips to 200 and normal serving begins automatically. +If ClickHouse is unreachable at start, the gateway binds `:8080` and serves `/livez` 503 with the latest schema-discovery error instead of exiting. Discovery retries in the background with exponential backoff (2s $\to$ 60s cap). Once a Refresh succeeds, `/livez` flips to 200 and serving begins. -This means: +Consequently: -- The binary itself no longer exits and crash-loops every ~10s under a supervisor. Process state is preserved across CH outages. -- An operator can `curl /livez` and read the exact failure mode instead of grepping a restart-loop log. -- `/v1/ingest?table={table}` and other schema-aware endpoints will reject requests with a 4xx until discovery succeeds, since the schema registry is empty. +- The binary does not crash-loop under a supervisor; process state is preserved across outages. +- Operators can `curl /livez` for failure modes instead of grepping logs. +- Schema-aware endpoints (e.g., `/v1/ingest?table={table}`) return 4xx until discovery succeeds. -**Important — orchestrator restart semantics.** `/livez` returning 503 during the retry window is what most LB / `depends_on` setups want (route around the unready instance, hold dependents), but a Kubernetes `livenessProbe` pointed at `/livez` will still mark the pod unhealthy and restart it after `failureThreshold × periodSeconds` elapses (default ~30s) — effectively re-creating the restart loop at a slower cadence. Use a `startupProbe` to gate liveness/readiness until the first successful schema discovery (see the K8s example below). Docker `HEALTHCHECK` marks the container `(unhealthy)` but does not restart it by default, so docker-compose deployments don't need a separate startupProbe-equivalent — the `HEALTHCHECK`'s `--start-period=15s` plus `service_healthy` dependency wait covers the same idea at a smaller scale. +**Orchestrator restart semantics.** A Kubernetes `livenessProbe` on `/livez` will restart the pod after `failureThreshold × periodSeconds` (default ~30s), recreating a restart loop. Use a `startupProbe` to gate liveness/readiness until first schema discovery. Docker `HEALTHCHECK` marks containers `(unhealthy)` without restarting them, so `docker-compose` deployments only need `--start-period=15s` and `service_healthy` dependencies. ### Docker `HEALTHCHECK` -Both bundled Dockerfiles (`deployments/Dockerfile` and `deployments/Dockerfile.goreleaser`) ship a built-in `HEALTHCHECK` that probes `/livez` every 10 seconds. Because the runtime image is distroless (no shell, no `curl`/`wget`), the check uses the binary's own `health` subcommand: +Bundled Dockerfiles (`deployments/Dockerfile` and `deployments/Dockerfile.goreleaser`) include a `HEALTHCHECK` probing `/livez` every 10 seconds. Since the runtime image is distroless (no shell, no `curl`/`wget`), it uses the binary's `health` subcommand: ```dockerfile HEALTHCHECK --interval=10s --timeout=3s --start-period=15s --retries=3 \ CMD ["/app/wavehouse", "health"] ``` -The `health` subcommand is a thin client that does an HTTP `GET http://127.0.0.1:$WH_SERVER_PORT/livez` and exits 0 (200 OK) or 1 (anything else). It honors `WH_SERVER_PORT` so it tracks whatever port the server is actually listening on. +The `health` subcommand performs an HTTP `GET http://127.0.0.1:$WH_SERVER_PORT/livez`, exiting 0 (200 OK) or 1. It honors `WH_SERVER_PORT`. -You can run it manually for debugging: +Debug manually: ```bash docker exec my-wavehouse /app/wavehouse health echo $? # 0 = healthy, 1 = unhealthy ``` -`docker ps` will show `(healthy)` / `(unhealthy)` in the STATUS column once the start-period elapses. +`docker ps` shows status in the STATUS column after the start-period. ### Compose `depends_on: service_healthy` -The Dockerfile `HEALTHCHECK` lets dependent services wait for WaveHouse to be ready before starting: +The Dockerfile `HEALTHCHECK` allows dependent services to wait for WaveHouse: ```yaml services: @@ -315,11 +308,11 @@ services: condition: service_healthy ``` -If you need different intervals (e.g. faster probes for E2E tests), override per-service via the compose `healthcheck:` block — that replaces the image's HEALTHCHECK for that container. +Override via a compose `healthcheck:` block for different intervals (e.g., E2E tests). ### Kubernetes / orchestrator note -K8s `livenessProbe` and `readinessProbe` use kubelet HTTP probes from outside the container — they don't go through the Dockerfile `HEALTHCHECK` at all. Configure them directly against `/livez` and `/readyz` in the PodSpec, and add a `startupProbe` so the boot-time schema-discovery retry window doesn't trip liveness and restart the pod: +K8s probes use kubelet HTTP calls, bypassing the Dockerfile `HEALTHCHECK`. Configure them in the PodSpec with a `startupProbe` to prevent boot-time restarts: ```yaml startupProbe: @@ -333,15 +326,15 @@ readinessProbe: httpGet: { path: /readyz, port: 8080 } ``` -Until `startupProbe` succeeds, kubelet doesn't run `livenessProbe` or `readinessProbe` against the pod — so a slow or temporarily-unreachable ClickHouse can't restart-loop the pod via the liveness path. Size `failureThreshold` to your expected worst-case CH boot time; the default 30 × 10s = 5min is generous and works for compose-on-NAS-style deployments where CH and WaveHouse can race during a host reboot. +`livenessProbe` and `readinessProbe` only run after `startupProbe` succeeds. Set `failureThreshold` to your worst-case CH boot time; 5min (30 $\times$ 10s) is generally sufficient. ## Behind a reverse proxy -WaveHouse serves plain HTTP on `:8080` and does **not** terminate TLS, manage certificates, or rate-limit — put a reverse proxy, CDN, or tunnel (nginx, Caddy, Cloudflare Tunnel) in front for any internet-facing deployment. A few behaviors only matter behind a proxy: TLS termination, the request-body size limits, Server-Sent Events buffering (WaveHouse now sends keepalive comments so quiet streams survive proxy idle timeouts, [#226](https://github.com/Wave-RF/WaveHouse/issues/226)), header/auth forwarding, and which health paths to expose. See **[Behind a reverse proxy](/reverse-proxy)** for the full guide and example nginx/Caddy/Cloudflare configs. +WaveHouse serves plain HTTP on `:8080` without TLS termination, certificate management, or rate-limiting; use a reverse proxy, CDN, or tunnel (nginx, Caddy, Cloudflare Tunnel) for internet deployments. Proxy considerations include TLS termination, request-body size limits, header/auth forwarding, health paths, and SSE buffering (keepalive comments now prevent idle timeouts, [#226](https://github.com/Wave-RF/WaveHouse/issues/226)). See **[Behind a reverse proxy](/reverse-proxy)** for guides and configs. ## ClickHouse Schema -WaveHouse uses a **Bring Your Own Schema** model. You create your tables in ClickHouse with whatever columns and engines you need. WaveHouse discovers the schemas automatically via `system.columns` and validates ingest data against them. +WaveHouse uses **Bring Your Own Schema**. Create tables in ClickHouse with any columns or engines; WaveHouse automatically discovers schemas via `system.columns` and validates ingest data. Example table: @@ -355,21 +348,21 @@ CREATE TABLE IF NOT EXISTS clicks ( ORDER BY (page); ``` -WaveHouse discovers this schema on startup and refreshes it every `schema.refresh_interval` seconds (default: 60). You can also trigger an immediate refresh via `POST /v1/schema/refresh` (admin-only). +Schemas refresh every `schema.refresh_interval` seconds (default: 60) or via admin-only `POST /v1/schema/refresh`. ## Dead Letter Queue (DLQ) -When `dlq.enabled` is `true` (default), a failed batch insert is retried row by row and the rows that fail again are published to the `WAVEHOUSE_DLQ` NATS stream under subjects `dlq.{table}`. This prevents infinite retry loops. Monitor DLQ depth via `GET /v1/dlq/stats`. +If `dlq.enabled` is `true` (default), failed batch inserts retry row by row; persistent failures publish to `WAVEHOUSE_DLQ` NATS stream under `dlq.{table}` to prevent infinite loops. Monitor via `GET /v1/dlq/stats`. ## Observability -Set `otel.enabled: true` (or `WH_OTEL_ENABLED=true`) to export traces, metrics, and logs, then point the OpenTelemetry SDK at your collector or gateway with the standard `OTEL_EXPORTER_OTLP_ENDPOINT` env var (always include a scheme — `https://` selects TLS, `http://` selects plaintext; with the endpoint unset the SDK defaults to **TLS** at `localhost:4317`, so a plaintext local collector needs `http://localhost:4317` set explicitly). `OTEL_EXPORTER_OTLP_HEADERS` carries cloud auth and `OTEL_EXPORTER_OTLP_CERTIFICATE` trusts a private CA, so telemetry can go to a local collector or straight to a TLS-protected cloud gateway with no sidecar. Each signal can be toggled independently — see [Configuration → OTel](/configuration#otel) for the full table of knobs. +Set `otel.enabled: true` (or `WH_OTEL_ENABLED=true`) to export traces, metrics, and logs. Use `OTEL_EXPORTER_OTLP_ENDPOINT` for the collector/gateway; include a scheme (`https://` for TLS, `http://` for plaintext). If unset, the SDK defaults to **TLS** at `localhost:4317`. Use `OTEL_EXPORTER_OTLP_HEADERS` for cloud auth and `OTEL_EXPORTER_OTLP_CERTIFICATE` for private CAs. See [Configuration → OTel](/configuration#otel) for all toggles. -WaveHouse **pushes** to an OTel collector; scraping-style pipelines (Promtail/Grafana Alloy → Loki, Vector, Fluent Bit) read stdout directly and own their own sample rates. The `otel.{traces,logs}.sample_rate` knobs apply only to the OTLP push path. Stdout always emits 100%. The logger fans out to both stdout and OTLP, so stdout output never disappears regardless of collector state. gRPC exporters are lazy, so an unreachable collector does not block startup — transient export errors are surfaced via the OTel SDK's error handler instead. +WaveHouse **pushes** to OTel collectors. Scraping pipelines (Promtail/Grafana Alloy → Loki, Vector, Fluent Bit) read stdout directly; `otel.{traces,logs}.sample_rate` only affects the OTLP push path. Stdout always emits 100% and remains active regardless of collector state. gRPC exporters are lazy; unreachable collectors won't block startup, and errors surface via the SDK handler. ### Pattern: Local collector (SigNoz, OTel Collector, Alloy) -A local collector almost always speaks **plaintext** gRPC, but the SDK's unset default endpoint is **TLS** at `localhost:4317` — so enabling OTel alone is not enough. Point it at the collector with an explicit `http://` scheme: `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317` (or set `OTEL_EXPORTER_OTLP_INSECURE=true`). All three signals (traces, metrics, logs) push through the same connection. This is the simplest setup. +Local collectors usually use **plaintext** gRPC. Since the SDK default is TLS, you must explicitly set `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317` (or `OTEL_EXPORTER_OTLP_INSECURE=true`). All signals push through one connection. ```yaml otel: @@ -378,9 +371,9 @@ otel: ### Pattern: Direct-to-cloud OTLP (Honeycomb, Grafana Cloud) -Set `OTEL_EXPORTER_OTLP_ENDPOINT` to an `https://` URL to select TLS (system root CAs), and `OTEL_EXPORTER_OTLP_HEADERS` for the per-RPC auth every cloud OTLP gateway expects — no sidecar required to terminate TLS or inject auth. For a private or self-signed gateway, point `OTEL_EXPORTER_OTLP_CERTIFICATE` at the CA certificate; for mutual TLS, add `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` and `OTEL_EXPORTER_OTLP_CLIENT_KEY`. These apply to the **trace and metric** signals only — the pinned gRPC logs exporter ignores the env TLS-cert vars (upstream bug [open-telemetry/opentelemetry-go#6661](https://github.com/open-telemetry/opentelemetry-go/issues/6661)), so against a private-CA gateway the logs signal falls back to system roots and won't connect; route logs through a local collector (which terminates TLS itself) until the fix lands upstream. +Use an `https://` URL for TLS and `OTEL_EXPORTER_OTLP_HEADERS` for auth. For private gateways, use `OTEL_EXPORTER_OTLP_CERTIFICATE`; for mutual TLS, add `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE` and `OTEL_EXPORTER_OTLP_CLIENT_KEY`. Note: the gRPC logs exporter ignores these TLS-cert vars (upstream bug [open-telemetry/opentelemetry-go#6661](https://github.com/open-telemetry/opentelemetry-go/issues/6661)); route logs through a local collector if using a private CA. -**Honeycomb** (single endpoint, per-RPC auth): +**Honeycomb**: ```bash export WH_OTEL_ENABLED=true @@ -388,7 +381,7 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io:443 export OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=YOUR_API_KEY ``` -**Grafana Cloud OTLP gateway** (Basic auth): +**Grafana Cloud OTLP gateway**: ```bash export WH_OTEL_ENABLED=true @@ -399,7 +392,7 @@ export OTEL_EXPORTER_OTLP_HEADERS="authorization=Basic $(printf '%s' "$INSTANCE_ ### Pattern: Datadog (via local DDOT Collector) -Datadog has no public direct-to-cloud OTLP endpoint — telemetry must transit a local OTLP receiver that re-exports over Datadog's own protocol. The supported receiver is the [DDOT Collector](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/) embedded in the Datadog Agent, which exposes a standard OTLP receiver on `4317`. Point WaveHouse at the local receiver as plaintext — the API-key auth lives on the Agent, so no `OTEL_EXPORTER_OTLP_HEADERS` is needed: +Datadog requires a local OTLP receiver (e.g., [DDOT Collector](https://docs.datadoghq.com/opentelemetry/setup/ddot_collector/) in the Datadog Agent). Point WaveHouse at the local plaintext receiver; auth is handled by the Agent: ```bash export WH_OTEL_ENABLED=true @@ -408,25 +401,19 @@ export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317 # plaintext gRPC; DD_ ### Pattern: Grafana Cloud / Mimir / Loki / Tempo via Grafana Alloy -The Grafana stack typically wants Prometheus-style scraping for metrics, stdout scraping for logs, and OTLP push for traces. Wire it like this: +- **Logs**: Alloy scrapes stdout (Docker socket/file tail/k8s API). No WaveHouse config needed. +- **Traces**: Set `OTEL_EXPORTER_OTLP_ENDPOINT` to Alloy's `otelcol.receiver.otlp` listener (`http://alloy:4317`); Alloy forwards to Tempo. +- **Metrics**: Set `prometheus.enabled: true`. Alloy's `prometheus.scrape` reads `http://wavehouse:8080/metrics`. The `prometheus` block is independent of `otel.*`: leave `otel.enabled: false` for scrape-only, or combine both if traces still go via OTLP. -- **Logs**: Alloy scrapes stdout via the Docker socket / file tail / k8s logs API. No WaveHouse config needed — stdout always emits 100%. -- **Traces**: Set `OTEL_EXPORTER_OTLP_ENDPOINT` to Alloy's `otelcol.receiver.otlp` listener (`http://alloy:4317`). Alloy forwards to Tempo. -- **Metrics**: Set `prometheus.enabled: true`. Alloy's `prometheus.scrape` reads `http://wavehouse:8080/metrics` (or whatever port you configured). The `prometheus` block is independent of `otel.*` — you can leave `otel.enabled: false` if Alloy is only scraping (no OTLP push at all), or combine the two if traces still go via OTLP. - -For the metrics path specifically: WaveHouse uses the OTel SDK's Prometheus exporter under the hood, which translates OTel metric names to Prometheus conventions automatically (dots and dashes become underscores; counters get a `_total` suffix). Existing OTel instruments don't need renaming. +WaveHouse uses the OTel SDK Prometheus exporter, automatically translating OTel metric names to Prometheus conventions (e.g., dots to underscores; counters get `_total`). ### Separating the `/metrics` listener -By default, `prometheus.port` is `0`, which mounts `/metrics` on the main API server port (typically `8080`). This is the friendliest setup for compose / quick-start use. - -For production posture where metrics should not be exposed on the public API listener, set `port` to a separate non-zero value (e.g. `9091`). WaveHouse spins up a dedicated HTTP listener bound to that port serving only `/metrics`. Firewall the port to internal networks only; the main API listener stays where it was. Both listeners participate in graceful shutdown. +By default, `prometheus.port: 0` mounts `/metrics` on the main API port (usually `8080`). For production, set `port` to a non-zero value (e.g., `9091`) to create a dedicated HTTP listener for metrics. Firewall this port to internal networks. ### Local Observability Stack -We intentionally do not maintain a heavy, multi-node observability cluster (like SigNoz or an ELK stack) for local development. Instead, we use lightweight, ephemeral, single-container tools that boot instantly and clean themselves up. - -The underlying Docker run scripts live in `scripts/otel/` and are invoked via Make: +We use lightweight, ephemeral tools via scripts in `scripts/otel/`: ```bash make obs-aspire # Simplest, in-memory, no login @@ -435,19 +422,20 @@ make obs-grafana # Full Grafana LGTM stack, auto-login enabled make obs-front ``` -All options automatically listen on standard OTLP ports (`4317` gRPC / `4318` HTTP) as **plaintext** receivers. If you are running WaveHouse directly on your host (e.g. `make dev`), set `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317` to reach them — the SDK's unset default dials `localhost:4317` over **TLS**, which a plaintext receiver rejects. +These use **plaintext** receivers on ports `4317` (gRPC) and `4318` (HTTP). -If you are running a containerized WaveHouse (e.g., via `deployments/compose/standalone.yaml`), you must override its environment to reach the host-bound collector: `OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4317`. +- Host-run WaveHouse (`make dev`): Set `OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317`. +- Containerized WaveHouse: Set `OTEL_EXPORTER_OTLP_ENDPOINT=http://host.docker.internal:4317`. ### Dashboards -Because we use ephemeral, single-container observability tools for local development, we no longer maintain strict, version-controlled JSON dashboards in this repository. +We do not maintain version-controlled JSON dashboards due to the ephemeral nature of local tools. -- If you use `make obs-aspire`, the UI is pre-built and requires zero configuration. -- If you use `make obs-grafana`, it is pre-configured to automatically provision the internal data sources and bypass the login screen. You can use Grafana's "Explore" tab to quickly jump between logs and traces. -- If you use `make obs-front`, it allows custom and comparison dashboards like grafana, but is simpler and easier to configure like aspire. +- `make obs-aspire`: Pre-built UI, zero config. +- `make obs-grafana`: Auto-provisioned data sources and bypassed login; use "Explore" for logs/traces. +- `make obs-front`: Supports custom dashboards with simpler configuration than Grafana. -For production deployments, you should construct dashboards specific to your telemetry vendor (Datadog, Honeycomb, New Relic, etc.) based on the standard OpenTelemetry metrics and traces WaveHouse emits. +For production, build vendor-specific dashboards based on standard OTel emissions. ## Resetting Data in Development diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 1899b8c1..7fdf2a24 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -9,26 +9,26 @@ Everything you need to build, test, lint, and contribute to WaveHouse — from f ## Prerequisites -You need these on your `PATH` before any `make` recipe will work end-to-end: +Ensure these are on your `PATH` before running `make`: | Tool | Required version | Why | Install | | ---- | ---------------- | --- | ------- | -| **Go** | 1.26+ (matches `go.mod`) | Compiles `cmd/wavehouse`; also runs the pinned `tool` deps (`gotestsum`, `gofumpt`, `goimports`, `govulncheck`, `deadcode`, `gsa`, `goda`) via `go tool` | [go.dev/dl](https://go.dev/dl/) | -| **GNU Make** | **4.0+** | The Makefile uses `--output-sync=target` (Make 4 only) and bash-pinned recipes. macOS ships with BSD Make 3.81, which **will not work** | macOS: `brew install make` then use `gmake` or put `$(brew --prefix make)/libexec/gnubin` on your PATH. Linux: usually already installed | -| **bash** | 4+ recommended | Recipes are pinned to `bash`; the helper scripts under `scripts/` use `set -euo pipefail` and bash arrays | macOS default is bash 3.2 (works for current recipes, but `brew install bash` is safer); Linux distros ship 4+ | -| **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/`; the E2E and integration suites boot ClickHouse via testcontainers (no compose file) | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups | -| **Node.js** | 22 LTS — pinned via `.nvmrc` at the repo root | Runtime for pnpm and the Vitest suites. Pinned to match CI (`setup-node` uses 22) and to avoid Node-major surprises; older Vitest versions in this repo were known to crash on Node 26 with a V8 heap-allocation abort | [nodejs.org](https://nodejs.org/) or `nvm use` / `fnm use` / `volta` (all read `.nvmrc`) | -| **pnpm** | 11.1+ (pinned via `packageManager` in the root `package.json`) | Package manager for the TypeScript SDK, E2E test harness, and docs site (managed as a single pnpm workspace from the repo root); `make build-ts`, `make test-ts`, `make test-e2e`, `make build-docs`, `make dev-docs`, `make preview-docs` all shell out to `pnpm` | `corepack enable && corepack prepare pnpm@11.1.3 --activate` (recommended), or `npm i -g pnpm` | -| **git** + **curl** | any recent | `git` for source + version metadata in builds; `curl` is used by the Makefile to fetch the pinned `golangci-lint` binary into `.bin/` | usually preinstalled | +| **Go** | 1.26+ (matches `go.mod`) | Compiles `cmd/wavehouse` and runs pinned `tool` deps (`gotestsum`, `gofumpt`, `goimports`, `govulncheck`, `deadcode`, `gsa`, `goda`) via `go tool` | [go.dev/dl](https://go.dev/dl/) | +| **GNU Make** | **4.0+** | Required for `--output-sync=target` and bash-pinned recipes. macOS BSD Make 3.81 is incompatible | macOS: `brew install make` (use `gmake` or add `$(brew --prefix make)/libexec/gnubin` to PATH). Linux: usually preinstalled | +| **bash** | 4+ recommended | Recipes and `scripts/` use `set -euo pipefail` and bash arrays | macOS default is 3.2 (`brew install bash` recommended); Linux ships 4+ | +| **Docker** *(or Podman)* | Engine 20.10+ with Compose **v2** plugin | Used for `deployments/compose/` and testcontainers (ClickHouse) in E2E/integration suites | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io). Honors `DOCKER_HOST` for rootless Podman | +| **Node.js** | 22 LTS (via `.nvmrc`) | Runtime for pnpm and Vitest; matches CI to avoid V8 heap-allocation aborts seen in Node 26 | [nodejs.org](https://nodejs.org/) or `nvm`/`fnm`/`volta` | +| **pnpm** | 11.1+ (pinned via `packageManager` in `package.json`) | Manages TypeScript SDK, E2E harness, and docs workspace; used by `make build-ts`, `test-ts`, `test-e2e`, and `*-docs` targets | `corepack enable && corepack prepare pnpm@11.1.3 --activate` or `npm i -g pnpm` | +| **git** + **curl** | any recent | `git` for source/versioning; `curl` fetches `golangci-lint` into `.bin/` | usually preinstalled | ### Auto-installed by `make tools` -Run `make tools` once after cloning to populate everything that doesn't have to be on your PATH: +Run `make tools` after cloning to install: -- **`golangci-lint` v2.11.4** → installed to `.bin/_/` (version-pinned in the Makefile; bumping the version triggers a reinstall). Not in `go.mod` because its dependency tree conflicts with the main module. -- **`air` v1.65.1** → installed to `.bin/_/` via `go install`; used by `make dev` for hot-reload. Same exclusion principle as `golangci-lint` — air's transitive deps (Hugo, Sass libs) would bloat `go.sum`. -- **Go `tool` deps** (`gotestsum`, `gofumpt`, `goimports`, `govulncheck`, `go-test-coverage`, `gocover-cobertura`, `deadcode`, `gsa`, `goda`) — pinned in `go.mod` via native `tool` directives (Go 1.24+), invoked with `go tool `. `make tools` runs `go mod download` so they're cached; they compile lazily on first invocation. -- **pnpm deps** for `clients/ts/`, `tests/e2e/sdk/`, and `docs/` (via `pnpm install --frozen-lockfile`). `make tools` runs only the pnpm install; the Playwright Chromium binary (~130 MB) is fetched on-demand by `make build-docs` / `make dev-docs` via the internal `install-playwright-docs` target, so Go-only contributors don't pay the download cost. When you do hit `build-docs` / `dev-docs`, Chromium is required by two parts of the docs *build*: `rehype-mermaid` (SVG diagram rendering) and the `diagram-png` integration (`docs/src/integrations/diagram-png.mjs`), which rasterizes each diagram to light/dark PNGs (a solid surface-card variant plus a transparent-background variant for slide decks) at `astro:build:done` for the Copy/Download buttons in the zoom lightbox. Both reuse the same Playwright Chromium, as does the manual `docs/scripts/screenshot.mjs` QA helper. `starlight-links-validator` runs under `build-docs` / CI only — the `dev-docs` watch loop skips it so a mid-edit dangling link doesn't fail every rebuild (CI still enforces link validity before merge; run `DOCS_WATCH_STRICT=1 make dev-docs` to keep the validator on locally). The `--with-deps` flag (which apt-installs Chromium's system libraries: `libnspr4`, `libnss3`, etc.) is only added when `$CI` is set, so contributor laptops don't get an unexpected `sudo` prompt. On Linux dev machines without those libs already present, run `pnpm exec playwright install-deps chromium` once manually. The docs site is a pnpm workspace package (`wavehouse-docs`); the root Makefile drives it directly via `pnpm --filter` (no sub-Makefile) — the `*-docs` targets show up in `make help`. It is also a real `@wavehouse/sdk` consumer (the landing page's live demo imports the workspace package), so `check-docs` / `build-docs` / `dev-docs` build the SDK first via `build-ts`; if you drive Astro directly through pnpm (e.g. `pnpm --filter wavehouse-docs run start`), run `make build-ts` once first so the dep resolves. +- **`golangci-lint` v2.11.4** $\rightarrow$ installed to `.bin/_/`. Not in `go.mod` due to dependency conflicts. +- **`air` v1.65.1** $\rightarrow$ installed to `.bin/_/` for hot-reload via `make dev`. Excluded from `go.sum` to avoid bloating with Hugo/Sass deps. +- **Go `tool` deps** (`gotestsum`, `gofumpt`, `goimports`, `govulncheck`, `go-test-coverage`, `gocover-cobertura`, `deadcode`, `gsa`, `goda`) $\rightarrow$ pinned in `go.mod` (Go 1.24+); cached via `go mod download`. +- **pnpm deps** for `clients/ts/`, `tests/e2e/sdk/`, and `docs/` $\rightarrow$ installed via `pnpm install --frozen-lockfile`. Playwright Chromium (~130 MB) is fetched on-demand by `make build-docs` or `dev-docs` for `rehype-mermaid` (SVG rendering) and the `diagram-png` integration (`docs/src/integrations/diagram-png.mjs`), which rasterizes diagrams to light/dark PNGs at `astro:build:done`; the manual `docs/scripts/screenshot.mjs` QA helper reuses the same Chromium. `--with-deps` (apt-installing `libnspr4`, `libnss3`, etc.) is added only when `$CI` is set, so laptops get no surprise `sudo` prompt; on Linux without those libs, run `pnpm exec playwright install-deps chromium` once. The docs site (`wavehouse-docs`) is a pnpm workspace package driven by the root Makefile via `pnpm --filter`. Since it consumes `@wavehouse/sdk`, `make build-ts` must run before Astro builds. `starlight-links-validator` runs in CI and `build-docs`; skip it in `dev-docs` unless `DOCS_WATCH_STRICT=1` is set. ### Verify your setup @@ -40,18 +40,18 @@ node --version # v22.x (matches .nvmrc and CI) pnpm --version # 11.1+ ``` -If any of those are wrong/missing, the Makefile recipes will fail with confusing errors (e.g. `--output-sync` is unrecognized on Make 3.81; `pnpm: command not found` on `make test-ts`). +Wrong or missing versions produce confusing recipe errors (`--output-sync` unrecognized on Make 3.81; `pnpm: command not found` on `make test-ts`). ### Optional but recommended | Tool | Why | Install | | ---- | --- | ------- | -| **[Claude Code](https://claude.com/claude-code)** | The repo ships team-wide configuration in `.claude/` — slash commands, subagents, hooks, status line. See [Claude Code & AI agents](/claude-code) for setup. | `brew install --cask claude-code` (macOS) or follow [official install](https://code.claude.com/docs/en/quickstart) | -| **[worktrunk](https://worktrunk.dev)** | Wraps `git worktree` for parallel-agent workflows. Project hooks live in `.config/wt.toml` (auto-runs `make tools` on new worktrees, `make verify` on pre-merge). | `brew install worktrunk && wt config shell install` | +| **[Claude Code](https://claude.com/claude-code)** | Uses repo config in `.claude/` (commands, subagents, hooks). See [Claude Code & AI agents](/claude-code) | `brew install --cask claude-code` or [official install](https://code.claude.com/docs/en/quickstart) | +| **[worktrunk](https://worktrunk.dev)** | Wraps `git worktree`. `.config/wt.toml` auto-runs `make tools` on new worktrees and `make verify` pre-merge | `brew install worktrunk && wt config shell install` | ## Quick Start -This is the fastest way to get a fully functional local environment: +Fastest way to get a functional local environment: ```bash # 1. Clone and bootstrap (Go modules + golangci-lint + pnpm deps) @@ -78,23 +78,22 @@ docker compose -f deployments/compose/dependencies.yaml exec clickhouse \ make dev ``` -WaveHouse is now running at `http://localhost:8080` in standalone mode with: +WaveHouse runs at `http://localhost:8080` in standalone mode with: -- **Embedded NATS** (JetStream) — no external MQ needed -- **L1 cache only** (Ristretto) — no external cache needed -- **Fail-closed** by default — `config.yaml` seeds no policy, so every request is denied until you seed one (see [Test the API](#test-the-api)) -- **Dedup disabled** by default — no Pebble needed -- **Schema discovery** — automatically finds your ClickHouse tables +- **Embedded NATS** (JetStream) and **L1 cache** (Ristretto): no external MQ or cache needed. +- **Fail-closed**: `config.yaml` seeds no policy, so requests are denied until you seed one (see [Test the API](#test-the-api)). +- **Dedup disabled**: Pebble not required by default. +- **Schema discovery**: automatically finds ClickHouse tables. ### Test the API -`make dev` is **fail-closed** — `config.yaml` seeds no policy, so every request is denied. Point it at the shipped dev policy (the `public` trial role: read/write `clicks`/`events`, no token) and (re)start it: +`make dev` is fail-closed. Use the shipped dev policy (`public` trial role: read/write `clicks`/`events`, no token) to enable requests: ```bash WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml make dev ``` -Then the tokenless data-plane calls work (create a `clicks` table first — see the [Getting Started](/getting-started) walkthrough): +Tokenless data-plane calls (ensure `clicks` table exists): ```bash # Ingest an event @@ -119,7 +118,7 @@ curl http://localhost:8080/livez # → {"status":"ok"} curl http://localhost:8080/readyz # → {"status":"ready"} ``` -The admin surface — `/v1/schema`, `/v1/admin/query` (raw SQL), `/v1/dlq/stats` — needs the **admin** role, which the `public` trial role doesn't have. Mint an admin JWT (see [Validating tokens](#validating-tokens) below) and pass it: +Admin endpoints (`/v1/schema`, `/v1/admin/query`, `/v1/dlq/stats`) require the **admin** role. Mint an admin JWT (see [Validating tokens](#validating-tokens)) and pass it: ```bash curl -s http://localhost:8080/v1/schema -H "Authorization: Bearer $TOKEN" | jq @@ -130,83 +129,68 @@ curl -s http://localhost:8080/v1/dlq/stats -H "Authorization: Bearer $TOKEN" ### How `make dev` works -`make dev` is a one-stop convenience target for backend and frontend -development. The recipe is essentially: +`make dev` is a convenience target for backend and frontend development: ```make dev: deps-up $(AIR) air -c .air.toml ``` -`deps-up` runs `docker compose ... up -d --wait clickhouse`, which blocks until the ClickHouse container's `/ping` healthcheck flips to healthy. `$(AIR)` lazily installs air to `.bin/_/` if missing. Then air takes over: it watches `cmd/` and `internal/` (the `.go` and `.yaml` files within them), rebuilds `tmp/wavehouse` on change, and restarts the binary. Config is **not** hot-reloaded: `make dev` runs the binary with `WH_CONFIG=.config.local.yaml` — a gitignored personal copy seeded **once** from `config.yaml` on first run (it won't re-copy if it already exists). So to change dev config, edit `.config.local.yaml` (not `config.yaml`) and restart `make dev`; air watches neither root file. +`deps-up` starts ClickHouse and blocks until the `/ping` healthcheck is healthy. `$(AIR)` installs air to `.bin/_/`. Air watches `cmd/` and `internal/`, rebuilds `tmp/wavehouse` on change, and restarts the binary. -`air` is pinned to a specific version and installed via `go install` rather than a `go.mod` tool directive — its transitive deps (Hugo, godartsass, Sass libs) would bloat `go.sum` for everyone. Same exclusion principle as `golangci-lint`. +Config is **not** hot-reloaded: `make dev` uses `WH_CONFIG=.config.local.yaml` (a gitignored copy seeded once from `config.yaml`). To apply config changes, edit `.config.local.yaml` and restart `make dev`. Air is installed via `go install` to avoid bloating `go.sum` with transitive dependencies. -**While `make dev` is running you get:** +**Features of `make dev`:** -- WaveHouse on `http://localhost:8080` with `cors_allowed_origins: ["*"]`, so a browser-based app on any localhost port can hit the API directly. -- A placeholder JWT secret (`change-me-in-production`) ships in `config.yaml`, but **no policy** is seeded — so the stack is fail-closed until you seed one (see [Test the API](#test-the-api)). Override the secret via `WH_AUTH_JWT_SECRET`. -- ClickHouse on `http://localhost:8123` (HTTP) and `localhost:9000` (native protocol), Compose project name `wavehouse-dev` so containers/volumes are namespaced. -- Hot reload: editing any `.go` file under `cmd/` or `internal/` triggers a debounced rebuild + restart. Config isn't hot-reloaded — `make dev` loads `.config.local.yaml` (a gitignored copy seeded once from `config.yaml`), so edit `.config.local.yaml` and restart to apply config changes. Air's stdout/stderr stream live so you see compile errors and server logs in the same terminal. +- WaveHouse on `http://localhost:8080` with `cors_allowed_origins: ["*"]`, so any localhost-port browser app can hit the API. +- Placeholder JWT secret `change-me-in-production` in `config.yaml`; override via `WH_AUTH_JWT_SECRET`. +- ClickHouse on `http://localhost:8123` (HTTP) and `localhost:9000` (native), namespaced under `wavehouse-dev`. +- Debounced rebuilds for `.go` files in `cmd/` or `internal/`. ### Dev convenience targets -These are the small targets behind `make dev` — useful directly when you want -to run WaveHouse outside of air (e.g. `make build && ./bin/wavehouse`), or -when you need to poke at ClickHouse: - | Target | What it does | | ------ | ------------ | | `make deps-up` | Start ClickHouse and block until healthy. Idempotent. | -| `make deps-down` | Stop ClickHouse. Data volume is preserved. | -| `make deps-logs` | `docker compose logs -f clickhouse` (Ctrl+C detaches; container keeps running). | -| `make deps-shell` | Drop into a `clickhouse-client` REPL on the running container. | -| `make deps-wipe` | Stop ClickHouse **and destroy its data volume**. Use when you want a clean schema. | -| `make clean-all` | Nuclear option — every `make` artifact + dev/E2E containers + volumes + `data/`. | +| `make deps-down` | Stop ClickHouse; preserves data volume. | +| `make deps-logs` | Stream ClickHouse logs (`docker compose logs -f clickhouse`). | +| `make deps-shell` | Enter `clickhouse-client` REPL on the container. | +| `make deps-wipe` | Stop ClickHouse and destroy its data volume for a clean schema. | +| `make clean-all` | Remove all make artifacts, containers, volumes, and `data/`. | -**Stopping `make dev`**: `Ctrl+C` stops air, which propagates SIGINT to WaveHouse for a graceful shutdown (NATS JetStream flush, etc.). ClickHouse stays up — re-running `make dev` is fast because the volume is preserved. Use `make deps-down` or `make deps-wipe` to stop ClickHouse explicitly. +**Stopping**: `Ctrl+C` stops air and gracefully shuts down WaveHouse (e.g., NATS JetStream flush). ClickHouse remains running; use `make deps-down` or `make deps-wipe` to stop it. ### Running with observability -WaveHouse natively exports standard OpenTelemetry (OTLP) data to `127.0.0.1:4317`. Rather than coupling a heavy observability database stack to the dev server, we provide three lightweight, single-container dashboard options. - -You run these in a separate terminal tab alongside `make dev` or your test suites (`make test-e2e`). - -They block the terminal and stream logs; simply press `Ctrl+C` to instantly tear them down and clean up the container. +WaveHouse exports OTLP data to `127.0.0.1:4317`. Run these in a separate terminal alongside `make dev` or `make test-e2e`: | Target | What it does | UI URL | | ------ | ------------ | ------ | -| `make obs-aspire` | Boots the Aspire dashboard. Extremely fast, in-memory only, and requires no login. Ideal for quick trace and log debugging. | `http://localhost:18888` | -| `make obs-grafana` | Boots Grafana LGTM (Loki, Grafana, Tempo, Prometheus). Pre-configured to bypass login. Best for advanced UI charting and trace-to-log correlation. | `http://localhost:3000` | -| `make obs-front` | Boots OTel-Front for a basic, alternative trace viewer. | `http://localhost:8000` | +| `make obs-aspire` | Aspire dashboard. Fast, in-memory, no login. Ideal for quick debugging. | `http://localhost:18888` | +| `make obs-grafana` | Grafana LGTM (Loki, Grafana, Tempo, Prometheus). Best for charting and correlation. | `http://localhost:3000` | +| `make obs-front` | OTel-Front basic trace viewer. | `http://localhost:8000` | -**Typical Workflow:** - -1. Open Tab 1: run `make obs-aspire` (UI opens automatically) -2. Open Tab 2: run `make dev` (or `make test-e2e`) -3. View traces, metrics, and logs flowing into the UI instantly. No accounts or auth tokens required. +**Workflow:** Run `make obs-aspire` in Tab 1, then `make dev` in Tab 2 to view traces and metrics instantly. ### Using the SDK against `make dev` -There's no bundled playground — point the published `@wavehouse/sdk` client at your local server (`baseURL: "http://localhost:8080"`), with the dev policy seeded so requests are authorized: +Point the `@wavehouse/sdk` client at `baseURL: "http://localhost:8080"` with a seeded dev policy: ```bash WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml make dev ``` -See the [SDK guide](/sdk) for the client API and examples. - -Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. +See the [SDK guide](/sdk) for examples. Frontend apps (Vite, Next.js) can use `createClient` directly; permissive CORS allows cross-origin requests. ### Validating tokens -There is no auth on/off switch — the JWT middleware always runs, but authorization is the policy's job (a `nil`/unseeded policy denies every token-based caller, admins included — only the operator key below still reaches the admin surface). To exercise token auth in dev, seed a policy *and* set a known secret — the dev policy's `admin_role` defaults to `admin`, so a JWT with `role: admin` unlocks the admin surface: +There is no auth on/off switch: the JWT middleware always runs, but authorization is the policy's job — a `nil`/unseeded policy denies every token-based caller, admins included, and only the operator key still reaches the admin surface. To test token auth, seed a policy and set a known secret: ```bash WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml WH_AUTH_JWT_SECRET=my-secret make dev ``` -The **operator key** is a non-JWT alternative: set one and send it in an `Authorization: Operator ` header (or the `X-Operator-Key` alias). Before any policy is seeded it reaches the **admin surface** — enough to seed or restore a policy over HTTP (the break-glass path). Once a policy is loaded, the key's role resolves to `admin`, so it then has full data-plane access too (pipes, queries, streaming, ingest) — handy for trialing without minting a JWT: +The **operator key** is a non-JWT alternative via `Authorization: Operator ` or `X-Operator-Key`. It accesses the admin surface even without a seeded policy (break-glass path). Once a policy is loaded, it resolves to `admin` for full data-plane access. ```bash WH_AUTH_OPERATOR_KEY=dev-operator-key make dev @@ -216,7 +200,7 @@ curl -H "Authorization: Operator dev-operator-key" http://localhost:8080/v1/admi curl -H "X-Operator-Key: dev-operator-key" http://localhost:8080/v1/admin/policy ``` -Then mint a token (role == the policy `admin_role`) and call an admin endpoint: +Mint a token (role must match the policy `admin_role`): ```bash # Using jwt-cli (https://github.com/mike-engel/jwt-cli) @@ -236,7 +220,7 @@ Set `WH_DEDUPE_ENABLED=true` and `WH_DEDUPE_ID_FIELD=event_id`: WH_DEDUPE_ENABLED=true WH_DEDUPE_ID_FIELD=event_id make dev ``` -Then include the dedup field in your ingest body: +Include the dedup field in ingest: ```bash curl -s -X POST "http://localhost:8080/v1/ingest?table=clicks" \ @@ -275,22 +259,24 @@ make build go build -o bin/wavehouse ./cmd/wavehouse ``` +Run `make build` for all binaries or `go build -o bin/wavehouse ./cmd/wavehouse` individually. + ## Running Modes at a Glance -| What you want | Command | +| Goal | Command | | ------------- | ------- | -| Hot-reload standalone dev server | `make dev` | -| Standalone binary (default config) | `make build && ./bin/wavehouse` | +| Hot-reload dev server | `make dev` | +| Standalone binary (default) | `make build && ./bin/wavehouse` | | Standalone via Docker Compose | `docker compose -f deployments/compose/standalone.yaml up -d` | -| Infrastructure deps only (ClickHouse) | `docker compose -f deployments/compose/dependencies.yaml up -d clickhouse` | +| ClickHouse deps only | `docker compose -f deployments/compose/dependencies.yaml up -d clickhouse` | ## Testing ### How It Works -All test commands use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. +All test commands use [gotestsum](https://github.com/gotestyourself/gotestsum) for colored output and summaries. Tool versions are pinned in `go.mod` via `tool` directives; the Makefile uses `go run`, removing the need for global installations. -All tests run with Go's **race detector** (`-race`) enabled by default. WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. +Tests run with Go's **race detector** (`-race`) by default to catch concurrency issues in NATS consumers, singleflight caching, and SSE hubs. ### Quick Reference @@ -324,44 +310,41 @@ make ci make cov ``` -Each test target writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. - -**Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). - -**Extra flags**: All test targets accept `ARGS="..."` for additional `go test` flags (e.g., `-run`, `-count`, `-timeout`). +Each target writes `covdata` to `tmp/coverage//data/`, renders reports, and gates against thresholds in `.testcoverage.yml`. `make cov` merges run suites and gates against the total. -**Note on timing**: gotestsum's `DONE ... in X.XXXs` reports pure test execution time. The total wall time includes Go compiling all packages — the first run compiles everything (~15s), subsequent runs use the build cache (~1s). +**Verbose output**: Use `V=1` for full output instead of compact `testdox` format. +**Extra flags**: Pass `ARGS="..."` for additional `go test` flags (e.g., `-run`, `-count`). +**Timing**: gotestsum's `DONE ... in X.XXXs` reports pure test execution time; total wall time includes compilation (~15s first run, ~1s cached). ### Test Structure | Category | Location | Docker? | Command | | -------- | -------- | ------- | ------- | | Unit tests | `internal/*/_test.go` | No | `make test` | -| SDK unit tests | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | +| SDK unit tests | `clients/ts/src/**/*.test.ts` | No | `make test-ts` | | Integration tests (Go) | `tests/integration/*_test.go` | Yes | `make test-integration` | | E2E tests (SDK) | `tests/e2e/sdk/*.test.ts` | Yes | `make test-e2e` | -- **Unit tests** live beside the code they test (e.g., `internal/discovery/discovery_test.go`). They use mocks or embedded NATS (in-process, no Docker needed). -- **Integration tests** use the `//go:build integration` build tag. The `setupTestEnv` helper starts a ClickHouse testcontainer, embedded NATS, ingest worker, and a full API router via `httptest.Server`. DLQ tests use `assert.Eventually` with a 30-second timeout for the 5-second ingest worker batch window. - -Shared test utilities live in `internal/testutil/` (e.g., `testutil.NopLogger()` for silencing embedded NATS output). +- **Unit tests**: beside the code they test (e.g. `internal/discovery/discovery_test.go`); use mocks or embedded NATS. +- **Integration tests**: Use `//go:build integration`. `setupTestEnv` starts a ClickHouse testcontainer, embedded NATS, ingest worker, and API router via `httptest.Server`. DLQ tests use `assert.Eventually` (30s timeout) for the 5s batch window. +- **Utilities**: `internal/testutil/` (Go), e.g. `testutil.NopLogger()` to silence embedded NATS. ### Adding New Tests -- **Unit test for `internal/foo/`** → create `internal/foo/foo_test.go` (same package). -- **Integration test needing Docker** → add a subtest under `tests/integration/` (e.g. a new file with `//go:build integration`). -- **E2E test via SDK** → add a `tests/e2e/sdk/*.test.ts` file. These tests exercise the full pipeline (ingest → ClickHouse → query) through the TypeScript SDK. Run with `make test-e2e`. -- **Test helpers** → add to `internal/testutil/` (Go) or `tests/e2e/sdk/helpers.ts` (E2E). +- **Unit test (`internal/foo/`)** $\rightarrow$ create `internal/foo/foo_test.go`. +- **Integration test (Docker)** $\rightarrow$ add subtest under `tests/integration/` with `//go:build integration`. +- **E2E SDK test** $\rightarrow$ add `tests/e2e/sdk/*.test.ts` to exercise the full pipeline via the TS SDK. Run with `make test-e2e`. +- **Helpers** $\rightarrow$ add to `internal/testutil/` (Go) or `tests/e2e/sdk/helpers.ts` (E2E). ### E2E Tests via SDK -The primary E2E integration test suite lives in `tests/e2e/sdk/`. It uses the TypeScript SDK as the test harness — every ingest→query test simultaneously validates the full Go backend pipeline and confirms SDK compatibility. +Located in `tests/e2e/sdk/`, these use the TS SDK as a harness to validate the Go backend and SDK compatibility. **Architecture**: -- `scripts/orchestrator` — the E2E entrypoint behind `make test-e2e`: it starts a clean ClickHouse **testcontainer** per run, launches the `wavehouse-cov` binary on a random free port, runs the SDK suite against it, then SIGINTs the binary to flush coverage. No Compose file is involved. CI runs the exact same path. -- `tests/e2e/sdk/setup.ts` — `globalSetup`. Probes the `CLICKHOUSE_URL` / `WAVEHOUSE_URL` the orchestrator injects, creates the per-suite tables, refreshes the schema, and bootstraps the baseline policy. It starts nothing itself and fails fast if either URL isn't up. It also prints the active Node/undici version, warning when the local Node major differs from `.nvmrc` — a runtime-specific transport bug is otherwise indistinguishable from a code failure (see [#440](https://github.com/Wave-RF/WaveHouse/issues/440)). -- `tests/e2e/sdk/helpers.ts` — JWT factories, typed client constructors, async wait helpers, direct ClickHouse query helper. +- `scripts/orchestrator`: Entrypoint for `make test-e2e`. Starts a ClickHouse testcontainer, launches `wavehouse-cov` on a random port, runs the suite, then SIGINTs the binary to flush coverage. No Compose file is used. +- `tests/e2e/sdk/setup.ts`: `globalSetup`. Probes injected URLs, creates tables, refreshes schema, and bootstraps policy. Fails fast if URLs are unreachable. Warns if local Node major differs from `.nvmrc` to avoid transport bugs (see [#440](https://github.com/Wave-RF/WaveHouse/issues/440)). +- `tests/e2e/sdk/helpers.ts`: JWT factories, typed clients, async wait helpers, and ClickHouse query helpers. **Running E2E tests**: @@ -370,31 +353,27 @@ The primary E2E integration test suite lives in `tests/e2e/sdk/`. It uses the Ty make test-e2e ``` -`make test-e2e` builds `bin/wavehouse-cov` (coverage-instrumented) and runs the orchestrator under `scripts/orchestrator/` to wire ClickHouse + the cover binary into the suite. covdata flushes on SIGINT into `tmp/coverage/e2e/data/`. +`make test-e2e` builds `bin/wavehouse-cov` (instrumented) and runs the orchestrator. Coverage flushes to `tmp/coverage/e2e/data/`. The orchestrator provisions its own stack, so it won't collide with `make dev`. -The orchestrator always provisions its own stack — a fresh ClickHouse testcontainer plus `wavehouse-cov` on a random free port — so a running `make dev` on `:8080` is neither detected nor reused, and the two don't collide. To run vitest against a stack you manage yourself, start the server from the **repo root** with the E2E fixture config: +To run vitest against a manual stack, start the server from the **repo root** using the E2E fixture: ```bash WH_CONFIG=tests/e2e/fixtures/config.yaml go run ./cmd/wavehouse ``` -The fixture matters: the suite signs its tokens with its `sdk-dev-secret` and depends on its dedupe, DLQ, and 5s schema-refresh settings. Point the suite at a default `make dev` server (`jwt_secret: change-me-in-production`) and setup's schema calls are rejected, then global setup dies 30s later on a misleading `schema not refreshed within 30s`. The repo root matters too — the fixture's `policy.file_path` is relative to the working directory. The fixture pins no ClickHouse address, so the server looks for one on `localhost:9000`; point it elsewhere with `WH_CH_ADDR` / `WH_CH_HTTP_PORT` if yours isn't there. - -Prefixing the variable to `make dev` does **not** work: that recipe pins `WH_CONFIG=.config.local.yaml` inline, which overrides anything inherited from the environment. +The fixture is required: the suite signs tokens with its `sdk-dev-secret` and needs its dedupe, DLQ, and 5s schema-refresh settings. Point it at a default `make dev` server (`jwt_secret: change-me-in-production`) and setup's schema calls are rejected, then global setup dies 30s later on a misleading `schema not refreshed within 30s`. The repo root is required because `policy.file_path` is relative. Use `WH_CH_ADDR` / `WH_CH_HTTP_PORT` if ClickHouse isn't on `localhost:9000`. Note: Prefixing `make dev` with variables fails as it pins `WH_CONFIG=.config.local.yaml` inline. -Then set `CLICKHOUSE_URL` / `WAVEHOUSE_URL` and run `pnpm test` from `tests/e2e/sdk/`; teardown is a no-op on that path, so your stack survives between iterations. - -If a previous run was killed (harness timeout, stop button, `SIGKILL`), it can leave a `wavehouse-cov` behind. That process shares `tmp/data` and `tmp/wavehouse-cov.log` with the next run and will corrupt it, so the orchestrator kills any leftover before starting and says so. +Set `CLICKHOUSE_URL` / `WAVEHOUSE_URL` and run `pnpm test` from `tests/e2e/sdk/`. A run killed by a harness timeout, stop button, or `SIGKILL` can leave a `wavehouse-cov` behind that shares `tmp/data` and `tmp/wavehouse-cov.log` with the next run and corrupts it, so the orchestrator kills leftovers first and says so. **Environment knobs**: | Variable | Effect | |----------|--------| -| `V=1` | Stream the WaveHouse subprocess log live *in addition to* capturing it to `tmp/wavehouse-cov.log`. The on-failure log excerpt is then skipped — you have already seen it | -| `E2E_CH_QUERY_TIMEOUT_MS` | Per-request ceiling for the suite's direct ClickHouse queries (default `10000`) | -| `E2E_NO_COVERAGE=1` | Drop `--coverage` from the vitest run (skips v8 instrumentation and report generation) while chasing a flake. **Local debugging only** — no report is written. Ignored (with a log line) under `make ci` / `make test-all`, so an exported-and-forgotten var can't produce a green coverage gate with the TS e2e report missing | +| `V=1` | Streams WaveHouse logs live; skips on-failure log excerpt. | +| `E2E_CH_QUERY_TIMEOUT_MS` | Ceiling for direct ClickHouse queries (default `10000`). | +| `E2E_NO_COVERAGE=1` | Skips `--coverage` in vitest. Local debugging only; ignored in `make ci`/`test-all`. | -**Test files** (`tests/e2e/sdk/*.test.ts`): `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress`, plus `helpers` — a stack-free unit test of the harness's own `waitForCondition` poll helper rather than a pipeline test. +**Test files** (`tests/e2e/sdk/*.test.ts`): `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress`, plus `helpers` — a stack-free unit test of the harness's `waitForCondition` poll helper, not a pipeline test. ## Linting @@ -402,31 +381,31 @@ If a previous run was killed (harness timeout, stop button, `SIGKILL`), it can l make lint ``` -`golangci-lint` is installed separately (not in `go.mod` — its massive dependency tree causes conflicts). If not found, `make lint` prints install instructions. +`golangci-lint` is installed separately to avoid dependency conflicts; `make lint` provides install instructions if missing. Install options: - **macOS**: `brew install golangci-lint` -- **Binary**: See [golangci-lint.run/welcome/install/](https://golangci-lint.run/welcome/install/) +- **Binary**: [golangci-lint.run/welcome/install/](https://golangci-lint.run/welcome/install/) - **Go install**: `go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest` -The configuration is in `.golangci.yml` (v2 format with `default: none` for explicit control) — that file is the authoritative list of enabled linters. Highlights: +`.golangci.yml` (v2 format, `default: none`) is the authoritative list of enabled linters: -- **errcheck** — Unchecked error returns -- **govet** — Suspicious constructs -- **staticcheck** — Static analysis -- **unused** — Unused code -- **gosec** — Security issues -- **gocritic** — Opinionated style checks -- **revive** — Extensible linter (replaces golint) -- **ineffassign** — Ineffective assignments -- **misspell** — Spelling errors in comments/strings -- **bodyclose** — Unclosed HTTP response bodies -- **noctx** — HTTP requests without context -- **errorlint** — Proper error wrapping checks (`%w`, `errors.Is/As`) -- **tparallel** — Missing `t.Parallel()` in test subtests +- **errcheck**: Unchecked error returns +- **govet**: Suspicious constructs +- **staticcheck**: Static analysis +- **unused**: Unused code +- **gosec**: Security issues +- **gocritic**: Style checks +- **revive**: Extensible linter (replaces golint) +- **ineffassign**: Ineffective assignments +- **misspell**: Spelling errors +- **bodyclose**: Unclosed HTTP bodies +- **noctx**: HTTP requests without context +- **errorlint**: Error wrapping (`%w`, `errors.Is/As`) +- **tparallel**: Missing `t.Parallel()` in subtests -Formatting (**gofumpt** — strict superset of gofmt — and **goimports** import grouping) is enforced through the v2 `formatters:` section rather than as linters. +Formatting (**gofumpt** and **goimports**) is enforced via the v2 `formatters:` section. ## Project Structure @@ -472,65 +451,65 @@ WaveHouse/ ## Code Conventions -- **Strict Go formatting**: Use `gofumpt` (a stricter superset of `gofmt`, enforced by CI). Run `make fmt` to format. -- **Interface-first design**: Core behaviors (`Cache`, `Deduplicator`, `Publisher`, `Subscriber`) are defined as interfaces so implementations can be swapped behind a stable contract. -- **Package boundaries**: The `internal/` directory ensures packages are private to this module. -- **Error handling**: Return errors to callers. Use `slog` for structured logging. -- **Schema-driven**: ClickHouse is the schema source of truth. WaveHouse discovers and validates against real table schemas. +- **Formatting**: Use `gofumpt` (stricter than `gofmt`, CI-enforced). Run `make fmt`. +- **Design**: Core behaviors (`Cache`, `Deduplicator`, `Publisher`, `Subscriber`) use interfaces for swappable implementations. +- **Boundaries**: `internal/` keeps packages private to this module. +- **Errors & Logging**: Return errors; use `slog` for structured logging. +- **Schema**: ClickHouse is the source of truth; WaveHouse validates against real schemas. ## Makefile Targets -Run `make help` to see all targets. Key ones: +Run `make help` to see all targets. | Target | Description | | ------ | ----------- | -| `make help` | Show all targets with descriptions (always the source of truth) | -| `make tools` | Bootstrap: install pinned tools (`golangci-lint`, `air`), Go modules, pnpm deps | +| `make help` | Show all targets with descriptions (source of truth) | +| `make tools` | Bootstrap: install pinned tools (`golangci-lint`, `air`), Go modules, and pnpm deps | | **Dev** | | -| `make dev` | Hot-reload dev server: ClickHouse via Compose + WaveHouse under air on `:8080` | -| `make deps-up` | Start ClickHouse alone (idempotent; blocks until healthy) | +| `make dev` | Hot-reload server: ClickHouse via Compose + WaveHouse under air on `:8080` | +| `make deps-up` | Start ClickHouse (idempotent; blocks until healthy) | | `make deps-down` | Stop ClickHouse (preserves data volume) | | `make deps-logs` | Tail ClickHouse logs | | `make deps-shell` | `clickhouse-client` REPL on the running container | -| `make deps-wipe` | Stop ClickHouse AND destroy its data volume (DESTRUCTIVE) | +| `make deps-wipe` | Stop ClickHouse and destroy its data volume (DESTRUCTIVE) | | **Observability** | | -| `make obs-aspire` | Prebuilt 0-config o11y UI to show WaveHouse metrics, logs, and traces locally | -| `make obs-grafana` | Grafana alternative to aspire, more advanced and complicated | -| `make obs-front` | Custom graphs like grafana, but is simpler and easier to configure like aspire | +| `make obs-aspire` | 0-config o11y UI for WaveHouse metrics, logs, and traces locally | +| `make obs-grafana` | Advanced Grafana alternative to aspire | +| `make obs-front` | Simple custom graphs; easier to configure than grafana | | **Static checks** | | -| `make fmt` | Check formatting across Go (`gofumpt`) + TS (Biome). Run `make fix` to apply. | -| `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | -| `make lint` | Run linters across Go (`golangci-lint`) + TS (Biome) | -| `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | -| `make verify` | Repo-wide static checks: Go (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | -| `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`) and TS (Biome `--write`) | +| `make fmt` | Check Go (`gofumpt`) and TS (Biome) formatting. Use `make fix` to apply. | +| `make tidy` | Verify `go.mod`/`go.sum` are tidy (use `make fix` to apply) | +| `make lint` | Run linters for Go (`golangci-lint`) and TS (Biome) | +| `make vulncheck` | Run `govulncheck` (`V=1` for full call stacks) | +| `make verify` | Repo-wide checks: Go (tidy, fmt, vulncheck, lint) + TS (Biome, `tsc`). Parallel-safe: `make -j verify` | +| `make fix` | Auto-fix Go (`tidy`, `gofumpt`, `goimports`, `lint --fix`) and TS (Biome `--write`) | | **Build** | | -| `make build` | Compile `wavehouse` → `bin/wavehouse` (debug symbols kept) | -| `make build-release` | Stripped release-style build → `bin/wavehouse-release` | +| `make build` | Compile `wavehouse` → `bin/wavehouse` (keeps debug symbols) | +| `make build-release` | Stripped release build → `bin/wavehouse-release` | | `make build-cover` | Coverage-instrumented build → `bin/wavehouse-cov` (used by E2E) | | `make build-ts` | Build TypeScript SDK → `clients/ts/dist/` | | **Test** | | | `make test` | Alias for `test-unit` | -| `make test-unit` | Go unit tests + render coverage + gate suite threshold | +| `make test-unit` | Go unit tests + coverage render + threshold gate | | `make test-integration` | Go integration tests (requires Docker) + coverage gate | -| `make test-ts` | SDK vitest unit tests + v8 coverage + gate against `suites.ts-unit` (matches Go's "always coverage" pattern) | -| `make cov` | Merge Go + TS coverage and gate against thresholds. Auto-runs after `make test-all` and `make ci`; standalone `make cov` is "show me the merged numbers without re-running." Each side skips silently if its data is missing, but `make cov` fails if *both* are empty (you ran it before any test target). | +| `make test-ts` | SDK vitest unit tests + v8 coverage + `suites.ts-unit` gate | +| `make cov` | Merge Go + TS coverage and gate against thresholds. Fails if both are empty; otherwise skips missing data. | | `make test-e2e` | E2E SDK suite against `bin/wavehouse-cov` + coverage gate | | `make test-all` | All four suites sequentially + merged coverage gate | -| `make ci` | Full pipeline: parallel `verify` + builds + unit/SDK tests, then integration + E2E + cov | -| **Analysis** (informational, not in CI) | | -| `make size` | Binary size analysis → `tmp/analysis/` (text + SVG + interactive HTML) | +| `make ci` | Pipeline: parallel `verify`, builds, unit/SDK tests, then integration, E2E, and cov | +| **Analysis** (informational) | | +| `make size` | Binary size analysis → `tmp/analysis/` (text, SVG, HTML) | | `make audit-cgo` | Audit dependency tree for C files (builds use `CGO_ENABLED=0`) | | `make deadcode` | Find unreachable functions | | `make dep-cut` | Top cuttable deps by transitive weight (`LIMIT=N` to override) | -| `make binary-analysis` | Combined: `size` + `audit-cgo` + `deadcode` | -| **Cleanup** (tiered — compose explicitly for partial resets) | | -| `make clean` | Build outputs only (`bin/`, `dist/`, `clients/ts/dist/`, `docs/dist/`, `docs/.dev-dist/`) | -| `make clean-test` | Test outputs only (`tmp/` — coverage data, logs, NATS state) | -| `make clean-tools` | Installed tools and pnpm deps (`.bin/`, `node_modules/`) | -| `make clean-all` | Full reset: above + `data/` + Docker volumes | +| `make binary-analysis` | Combined: `size`, `audit-cgo`, and `deadcode` | +| **Cleanup** | | +| `make clean` | Remove build outputs (`bin/`, `dist/`, `clients/ts/dist/`, `docs/dist/`, `docs/.dev-dist/`) | +| `make clean-test` | Remove test outputs (`tmp/` coverage, logs, NATS state) | +| `make clean-tools` | Remove installed tools and pnpm deps (`.bin/`, `node_modules/`) | +| `make clean-all` | Full reset: all above + `data/` + Docker volumes | -All test targets accept `ARGS="..."` for pass-through `go test` flags. Build targets accept `TAGS="..."` for Go build tags. `V=1` switches to verbose `gotestsum` output. +Test targets accept `ARGS="..."` for `go test` flags. Build targets accept `TAGS="..."` for Go build tags. `V=1` enables verbose `gotestsum` output. ## Dependency Management @@ -543,32 +522,34 @@ go mod tidy # Remove unused, add missing ### Vulnerability Scanning -`govulncheck` analyzes your actual call graph — not just the module graph — so it only reports vulnerabilities in code paths you use. +`govulncheck` analyzes the actual call graph, reporting only vulnerabilities in used code paths. ```bash make vulncheck ``` -For a combined security scan, run `make verify` — it runs `vulncheck` alongside `lint`, and `gosec` is one of the linters enabled in `.golangci.yml`. This is also what CI runs on every push and pull request. +Run `make verify` for a combined scan; it executes `vulncheck`, `lint`, and `gosec` (via `.golangci.yml`). CI runs this on every push and pull request. ### Dependabot -Dependabot is configured in `.github/dependabot.yml` to open weekly grouped PRs for three update configs: +Configured in `.github/dependabot.yml`, Dependabot opens weekly grouped PRs: -- **Go modules** (root) — outdated or vulnerable Go dependencies, commit prefix `deps:` -- **GitHub Actions** (root) — outdated action versions tracked against the SHA pins in `ci.yml` / `release.yml`, commit prefix `ci:` -- **npm — pnpm workspace** (root) — covers all three TypeScript packages (the docs site, the SDK, and the E2E tests) in one grouped PR, commit prefix `deps:` +- **Go modules** (root): Outdated or vulnerable dependencies; prefix `deps:` +- **GitHub Actions** (root): Outdated versions against SHA pins in `ci.yml` / `release.yml`; prefix `ci:` +- **npm — pnpm workspace** (root): All three TypeScript packages (docs site, SDK, E2E tests) in one PR; prefix `deps:` -PRs are grouped per config to reduce noise. The npm config is pointed at the workspace **root** (`directory: /`), not the individual member directories. The repo has a single root `pnpm-lock.yaml`, and Dependabot only updates a lockfile co-located with the manifest it targets — so a per-member config (the previous setup) bumped a member's `package.json` without regenerating the root lockfile, and every such PR then failed CI's `pnpm install --frozen-lockfile` with `ERR_PNPM_OUTDATED_LOCKFILE`. Pointing at the root lets Dependabot read `pnpm-workspace.yaml`, walk every member, and update the one lockfile. +The npm config targets the root (`directory: /`) because Dependabot only updates lockfiles co-located with the target manifest. Previous per-member configs failed CI's `pnpm install --frozen-lockfile` with `ERR_PNPM_OUTDATED_LOCKFILE` as they didn't regenerate the root `pnpm-lock.yaml`. Root targeting allows Dependabot to use `pnpm-workspace.yaml` to update all members and the single lockfile. -**No auto-merge.** Dependabot PRs go through the same merge gate as any other PR — an approval from the `@Wave-RF/wavehouse-admins` team (the ruleset's `required_reviewers` rule) plus the required checks. (The former `dependabot-automerge.yml`, which auto-approved and merged patch/minor bumps hands-off, was removed — every bump now gets a human admin review.) +**No auto-merge.** All PRs require an approval from `@Wave-RF/wavehouse-admins` (via ruleset `required_reviewers`) and passing checks. The `dependabot-automerge.yml` was removed; every bump now requires human admin review. ## Releasing the SDK -The TypeScript SDK (`@wavehouse/sdk`, in `clients/ts/`) publishes to npm via `.github/workflows/publish-npm.yml` using OIDC trusted publishing — no `NPM_TOKEN`. It is independent of the server's Go/Docker release (`release.yml`): the `v*` (server) and `sdk-v*` (SDK) tag globs are disjoint, so the two never collide. There are two channels: +The TypeScript SDK (`@wavehouse/sdk`, in `clients/ts/`) publishes to npm via `.github/workflows/publish-npm.yml` using OIDC trusted publishing (no `NPM_TOKEN`). It is independent of the server's Go/Docker release (`release.yml`); `v*` (server) and `sdk-v*` (SDK) tag globs are disjoint. + +Two channels exist: -- **Dev snapshots.** Every push to `main` publishes `0.0.0-dev.` under the `dev` dist-tag — but only when the built `dist/` actually changed (the version is a hash of the build output, so an unchanged build resolves to an already-published version and is skipped). Install the bleeding edge with `npm install @wavehouse/sdk@dev`. -- **Tagged releases.** Pushing a `sdk-vX.Y.Z` tag publishes that version and creates a GitHub Release. A stable version goes to the `latest` dist-tag; a prerelease (`sdk-v0.2.0-rc.1`) is published under `alpha`/`beta`/`rc`/`next` — derived from the suffix — and marked as a GitHub pre-release. The tag **must** match `clients/ts/package.json`'s `version`, or the job fails fast. +- **Dev snapshots.** Pushes to `main` publish `0.0.0-dev.` under the `dev` dist-tag if `dist/` changed. Install via `npm install @wavehouse/sdk@dev`. +- **Tagged releases.** Pushing a `sdk-vX.Y.Z` tag publishes that version and creates a GitHub Release. Stable versions use the `latest` dist-tag; prereleases (e.g., `sdk-v0.2.0-rc.1`) use tags like `alpha`/`beta`/`rc`/`next` based on the suffix and are marked as GitHub pre-releases. The tag **must** match `clients/ts/package.json`'s `version`. To cut a release: @@ -580,97 +561,93 @@ git push origin sdk-v0.1.0 ``` :::caution[The first tagged release promotes `latest`] -npm sets a package's `latest` dist-tag on its *first* publish even under `--tag dev`, so until the first `sdk-v*` release a bare `npm install @wavehouse/sdk` (and the bare CDN URLs) resolve to a `0.0.0-dev.*` snapshot. The first tagged stable release moves `latest` to a real version and fixes this for every consumer. +npm sets `latest` on the first publish, even under `--tag dev`. Until the first `sdk-v*` release, `npm install @wavehouse/sdk` and bare CDN URLs resolve to a `0.0.0-dev.*` snapshot. The first stable release fixes this. ::: ## CI & review automation -This repo has three tiers of AI automation sitting alongside the normal CI checks. Full detail lives in `AGENTS.md`; this section covers the contributor-facing behavior. +This repo uses three tiers of AI automation alongside standard CI checks. Full details are in `AGENTS.md`. ### PR title and Conventional Commits -PR titles must match Conventional Commits format and stay ≤ 72 characters — the title becomes the squash-merge commit subject. Both rules are enforced by the `PR title` job under the required `CI` check (`.github/workflows/ci.yml`); validate locally with `scripts/lint-pr-title.sh ""`: +PR titles must follow Conventional Commits format and be $\le$ 72 characters, as they become the squash-merge commit subject. The `PR title` job in `.github/workflows/ci.yml` enforces this; validate locally via `scripts/lint-pr-title.sh "<title>"`. ```text <type>(optional-scope)(optional-!): <lowercase subject, no trailing period> ``` -Allowed types: `feat`, `fix`, `docs`, `refactor`, `test`, `chore`, `ci`, `deps`, `build`, `perf`, `revert`, `style`. - -The `!` before `:` marks a breaking change per Conventional Commits 1.0.0 (e.g., `feat!: remove deprecated endpoint`, `refactor(api)!: rename handlers`). Titles are also capped at **72 characters** — they become squash-merge commit subjects (Dependabot PRs are exempt from the cap). +Allowed types: `feat`, `fix`, `docs`, `refactor`, `test`, `chore`, `ci`, `deps`, `build`, `perf`, `revert`, `style`. Use `!` before `:` for breaking changes per Conventional Commits 1.0.0 (e.g. `feat!: remove deprecated endpoint`, `refactor(api)!: rename handlers`). Dependabot PRs are exempt from the length cap. -If the title doesn't match, a sticky comment posts on the PR explaining the format (from the `PR housekeeping` workflow, which mirrors the same script); it auto-removes once the title is fixed. Fixing the title needs no new push — the edit triggers housekeeping, which re-runs the failed `PR title` job (the job re-reads the title from the API, not the stale event payload). +If invalid, the `PR housekeeping` workflow posts a sticky comment explaining the format. Editing the title triggers an automatic re-run of the `PR title` job without requiring a new push. ### Required status checks -The `main branch protection` ruleset requires one status check to pass before any PR can merge: +The `main branch protection` ruleset requires the `CI` aggregator job (`.github/workflows/ci.yml`) to pass before merging. This DAG runs: -- `CI` — the aggregator job of `.github/workflows/ci.yml`. The workflow is a job DAG over the same Makefile targets local `make ci` runs: `lint` (`make verify`), `unit` (`make test-unit test-ts`), `integration` (`make test-integration`), `e2e` (`make -j test-e2e` — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), `coverage` (`make cov` over every suite's uploaded coverage fragment + threshold gates, like local `make ci`'s final step), `docs-build` (`make build-docs` when docs-affecting files changed, uploading the docs dist artifact), `PR title` (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run's Summary page gets a per-job wall-clock table from the non-gating `Timing summary` job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md). +- `lint` (`make verify`) +- `unit` (`make test-unit test-ts`) +- `integration` (`make test-integration`) +- `e2e` (`make -j test-e2e`) +- `coverage` (`make cov`) +- `docs-build` (`make build-docs`) +- `PR title` (Conventional Commits) +- Docs preview/deploy jobs -The `PR housekeeping` workflow still runs on every PR (labels + the title explainer comment) but is no longer a required check. +The aggregator fails if any job fails or is canceled; skipped jobs are treated as passing. Fork PRs skip secret-bearing docs deploys, and docs-only PRs skip Go tests. A non-gating `Timing summary` job provides wall-clock data on the Summary page. See [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md) for architecture and cache policy. -The ruleset also requires an approval from the `@Wave-RF/wavehouse-admins` team (the `required_reviewers` rule — this is what mandates an admin sign-off, replacing the old `Admin approval` status-check workflow), plus 1 approving review, approval of the most recent push by someone other than its author, resolution of all review threads, linear history, no branch deletion, no force-push, and squash-merge only. Repository admins may bypass these requirements when merging their own PR (e.g. a trivial `.github` change) but still cannot push directly to `main`. +The ruleset also mandates: -Approved, green PRs land through a **merge queue** ("Merge when ready"): the queue re-runs the required `CI` check against the PR merged with *current* main (a `merge_group` event — the CI workflow runs the full test suite for these) and fast-forwards only on green. That integration re-test replaces the old "branch is out-of-date with the base branch" requirement — queued PRs don't need manual branch updates, and the queue never pushes to the PR branch. +- Approval from `@Wave-RF/wavehouse-admins` (via `required_reviewers`). +- One additional approving review. +- Approval of the most recent push by a non-author. +- Resolution of all review threads. +- Linear history, no force-push, and squash-merge only. -Dependabot PRs go through the same admin review as any other PR — there is no auto-merge (see the Dependabot section above). +Admins may bypass these for their own PRs but cannot push directly to `main`. Approved PRs use a **merge queue** ("Merge when ready"), which re-runs `CI` against the current `main` via a `merge_group` event before fast-forwarding. This replaces manual branch updates. Dependabot PRs require standard admin review; there is no auto-merge. ### Merge behavior -Squash-only merges. The **PR title** becomes the commit subject (with `(#NN)` appended automatically), the **PR body** becomes the commit message. Keep PR bodies tight — they land in `git log` on `main`. The PR template gives the right shape (Summary / Test plan / Related Issues). - -Include `Closes #NN` in the PR body to auto-close the related issue on merge. Alternatively, link the issue in the sidebar's **Development** section — that triggers auto-close even without the keyword. - -Auto-merge is enabled repo-wide: click "Enable auto-merge (squash)" on a PR and it merges once checks + approvals land. +Only squash-merges are permitted. The PR title becomes the commit subject (with `(#NN)` appended) and the body becomes the commit message. Use the PR template (Summary / Test plan / Related Issues). Include `Closes #NN` in the body or link the issue in the **Development** sidebar to auto-close it on merge. Auto-merge is enabled repo-wide via "Enable auto-merge (squash)". ### AI reviewers -Advisory PR review comes from marketplace apps configured at the org/repo level: +Marketplace apps provide advisory reviews: -- **CodeRabbit** — automated PR review; auto-reviews on open + push, re-trigger with `@coderabbitai review`. -- **Copilot** — tied to individual reviewer subscriptions; shows up on PRs where a maintainer with Copilot Pro is listed as a reviewer. +- **CodeRabbit**: Reviews on open/push; re-trigger with `@coderabbitai review`. +- **Copilot**: Appears when a maintainer with Copilot Pro is a reviewer. -Both are **advisory** — the ruleset's `required_reviewers` rule (an `@Wave-RF/wavehouse-admins` approval) plus its thread-resolution / linear-history / required-check rules are the actual merge-gate. +These are non-gating; the `required_reviewers` rule and thread resolution remain the actual merge gates. ### Reviewer assignment and the Task Board -Reviewer assignment is GitHub-native, and so is the board: - -- **Reviewer assignment**: the `main branch protection` ruleset's `required_reviewers` rule requests the `@Wave-RF/wavehouse-admins` team on every PR, and the team's **code-review assignment** (configured on the team) auto-assigns and load-balances a specific member. No workflow is involved; `dismiss_stale_reviews_on_push` clears approvals on new commits, and GitHub re-requests per its own rules. -- **Merge gate**: the ruleset's `required_reviewers` rule requires an `APPROVED` review from the `@Wave-RF/wavehouse-admins` team; it also adds review-thread resolution, linear history, and squash-only. Auto-merge (squash) takes over once checks and approvals land. -- **Task Board** (Projects v2, project #7): card placement and status are handled by GitHub-native Projects v2 automation configured in the project UI — there is no workflow-driven board state machine. Priority lives on the board's `Priority` field (set during issue triage, below). - -Dependabot PRs go through the same admin review as any other PR — there is no auto-merge (see the Dependabot section above). +- **Assignment**: The ruleset requests `@Wave-RF/wavehouse-admins`, and GitHub's team code-review assignment load-balances members. `dismiss_stale_reviews_on_push` clears approvals on new commits. +- **Merge gate**: Requires an `APPROVED` review from the admin team, thread resolution, linear history, and squash-only merges. +- **Task Board** (Projects v2, project #7): Managed via native Projects v2 automation; no workflow is used for state transitions. Priority is set in the board's `Priority` field during triage. ### Invoking bots manually -- **CodeRabbit**: comment `@coderabbitai review` to re-trigger a review, or `@coderabbitai <question>` to ask it something. Works in both top-level and inline review comments. -- **Copilot**: the re-request-review button on the PR page sends a fresh request. +- **CodeRabbit**: Comment `@coderabbitai review` to re-trigger or `@coderabbitai <question>` for inquiries. +- **Copilot**: Use the "re-request-review" button on the PR page. ### Review-response expectations -Every review comment (human or AI) must get a substantive reply before merge — not "fixed" alone. The ruleset's `required_review_thread_resolution: true` means unresolved conversations literally block merge. Agents working on PRs follow the pattern documented in `AGENTS.md` §"Review Response": accept / push back / defer, reply with detail, resolve when settled. - -When pushing back on a bot's suggestion, end the reply with the bot's mention (e.g. `@coderabbitai`) to invite a counter-reply so the dialog actually loops. +All comments must receive a substantive reply; `required_review_thread_resolution: true` blocks merges until resolved. Agents follow the "Review Response" pattern in `AGENTS.md`. When pushing back against bots, end replies with their mention (e.g., `@coderabbitai`) to ensure a response loop. ### Issue triage -`.github/workflows/triage.yml` classifies new and edited issues via GitHub Models (`gpt-4o-mini`) and applies: +`.github/workflows/triage.yml` uses GitHub Models (`gpt-4o-mini`) to apply: -- `area/*` labels based on the issue body (areas pulled dynamically from the `area/*` repo labels — adding a new `area/foo` label with a description is all you need; no workflow edit) -- `security` if the model flags a security concern -- `breaking-change` if the model flags a public-API break -- Priority on the **Task Board** project #7 via the board's `Priority` field (requires `PROJECT_BOARD_TOKEN` secret — labels apply with or without it) +- `area/*` labels based on the body (pulled from existing repo label descriptions). +- `security` and `breaking-change` labels if flagged. +- Priority in project #7 via `PROJECT_BOARD_TOKEN`. ### Auto-labeling PRs -The `PR housekeeping` workflow (`.github/workflows/housekeeping.yml`) runs `actions/labeler` with `.github/labeler.yml` to apply `area/*`, `dependencies`, `github_actions`, `go`, and `documentation` labels to PRs based on the files they change. Sync-mode: labels follow the current changed-file set. +The `PR housekeeping` workflow (`.github/workflows/housekeeping.yml`) runs `actions/labeler` with `.github/labeler.yml` to apply `area/*`, `dependencies`, `github_actions`, `go`, and `documentation` labels based on changed files. ### When adding a new `internal/<pkg>/` package -Follow the checklist in `AGENTS.md` §"Common Tasks / Adding a new internal package" — the automation-relevant steps are: - -1. Create a matching `area/<pkg>` repo label with a meaningful description (triage reads the description as the classifier's per-area hint). -2. Add the path → label mapping to `.github/labeler.yml` so PRs touching the new package get auto-labeled. +Per `AGENTS.md`: -Triage picks up the new label automatically; no workflow edit needed. +1. Create an `area/<pkg>` repo label with a description (used as a classifier hint). +2. Add the path $\to$ label mapping to `.github/labeler.yml`. diff --git a/docs/src/content/docs/durability.md b/docs/src/content/docs/durability.md index f1b56173..646433da 100644 --- a/docs/src/content/docs/durability.md +++ b/docs/src/content/docs/durability.md @@ -5,62 +5,56 @@ sidebar: order: 11 --- -WaveHouse buffers every ingested event in embedded NATS JetStream before the [ingest worker](/ingest-pipeline) drains it into ClickHouse. That buffer lives on disk at `<data_dir>/nats`, and **WaveHouse runs JetStream in its strictest durability mode**: every publish is `fsync`'d to non-volatile storage before the producer is acknowledged. +WaveHouse buffers ingested events in embedded NATS JetStream at `<data_dir>/nats` before the [ingest worker](/ingest-pipeline) drains them into ClickHouse. **WaveHouse runs JetStream in its strictest durability mode**: every publish is `fsync`'d to non-volatile storage before acknowledgment. -This is a deliberate, strong guarantee — but it makes your ingest latency a direct function of your storage's `fsync` latency. On managed cloud block storage that is effectively free; on some commodity or virtualized substrates the `fsync` tail balloons into seconds and ingest visibly suffers. This page explains the contract, where it is cheap versus expensive, and how to measure your storage before you trust it. +Ingest latency is a direct function of your storage's `fsync` latency. On managed cloud block storage, this is typically negligible; on some commodity or virtualized substrates, `fsync` tails can balloon into seconds, degrading ingest performance. ## The durability contract -The embedded server is started with `SyncAlways: true` (`internal/mq/embedded.go`). Concretely: +The embedded server starts with `SyncAlways: true` (`internal/mq/embedded.go`). > When a client receives `200` from `POST /v1/ingest`, the event has already been `fsync`'d to disk on the WaveHouse node. -Ingestion is still [asynchronous](/architecture) end-to-end — the `200` means *durably buffered in JetStream*, not yet *written to ClickHouse* (the worker flushes to ClickHouse later, and the [worker's own ack](/ingest-pipeline#backpressure-and-durability-knobs) is what records "now in ClickHouse"). But the buffering step itself is hard-durable: an event that got a `200` survives an immediate, uncontrolled power loss on the node. - -This is the strongest mode JetStream offers. It is stronger than the default, where a publish is acked once the write reaches the OS page cache and the data is flushed to disk later by a periodic background sync — fast, but a hard crash can lose the not-yet-flushed window. +Ingestion remains [asynchronous](/architecture): `200` means *durably buffered in JetStream*, not yet *written to ClickHouse*. The buffering step is hard-durable; an event receiving a `200` survives immediate, uncontrolled power loss. This is stronger than the default JetStream mode, where publishes are acked once they reach the OS page cache. | Mode | Ack means | Crash exposure | Throughput | | --- | --- | --- | --- | -| **`SyncAlways` (WaveHouse today)** | data is `fsync`'d to disk | none for acked events | bounded by `fsync` latency | -| Periodic group commit (default JetStream) | data is in the OS page cache | up to one sync interval of acked-but-unflushed events | bounded by memory/CPU | +| **`SyncAlways` (WaveHouse)** | data is `fsync`'d to disk | none for acked events | bounded by `fsync` latency | +| Periodic group commit (default) | data is in OS page cache | up to one sync interval | bounded by memory/CPU | -WaveHouse does not currently expose a knob to relax this — `SyncAlways` is always on. Exposing a configurable group-commit interval (`mq.sync_interval`) is tracked in [#139](https://github.com/Wave-RF/WaveHouse/issues/139). +WaveHouse does not currently expose a knob to relax this; a configurable group-commit interval (`mq.sync_interval`) is tracked in [#139](https://github.com/Wave-RF/WaveHouse/issues/139). ## Why the fsync tail is your ingest floor -Because the publish blocks on `fsync`, **your typical ingest latency is your storage's typical `fsync` latency, and your worst-case publish is your storage's worst-case `fsync`.** When that tail is healthy (sub-millisecond to single-digit milliseconds) the guarantee is essentially free. When it is not, the same code path that handles every production message stalls: +Because publishes block on `fsync`, typical and worst-case ingest latency mirror your storage's `fsync` performance. If the tail is unhealthy: -- Publishes block for the duration of the `fsync`, so a multi-second `fsync` tail is a multi-second ingest tail. -- The embedded server's stream/consumer setup and every publish run under the JetStream client's request timeout; a slow-enough substrate makes them exceed it. The boot-time symptom is `create stream: ... context deadline exceeded`. -- If the worker cannot drain to ClickHouse faster than producers publish, the stream fills toward `mq.max_bytes_gb` and the API returns `503` ([backpressure by construction](/ingest-pipeline#backpressure-and-durability-knobs)). +- Publishes block for the duration of the `fsync`. +- Slow substrates may cause JetStream client request timeouts, manifesting at boot as `create stream: ... context deadline exceeded`. +- If the worker cannot drain to ClickHouse faster than producers publish, the stream hits `mq.max_bytes_gb` and the API returns `503` ([backpressure by construction](/ingest-pipeline#backpressure-and-durability-knobs)). ## Where `SyncAlways` is cheap vs. expensive -The strict guarantee translates well to managed cloud infrastructure — the presumed production target — and to enterprise-grade local disks. It is the commodity and virtualized substrates that bite. - -**Healthy — `SyncAlways` is effectively free:** +**Healthy (effectively free):** | Substrate | Why | | --- | --- | -| Cloud block storage (gp3/io2 EBS, GCP pd-ssd, Azure Premium SSD) | Battery-backed cache acks sync writes from non-volatile DRAM, not NAND. Sub-millisecond at typical load. | -| Enterprise NVMe with power-loss protection (Optane, Samsung PM-series, Solidigm D7) | The PLP capacitor lets the controller ack a sync write from DRAM — the `fsync` ≈ `memcpy`. | -| Local `ext4` on consumer NVMe | Single-device journal commit, 1–10 ms typical. Tail spikes under heavy concurrent dirty data, but bounded. | +| Cloud block storage (gp3/io2 EBS, GCP pd-ssd, Azure Premium SSD) | Battery-backed cache acks sync writes from non-volatile DRAM. | +| Enterprise NVMe with PLP (Optane, Samsung PM, Solidigm D7) | PLP capacitor lets the controller ack a sync write from DRAM — `fsync` ≈ `memcpy`. | +| Local `ext4` on consumer NVMe | Single-device journal commit; 1–10 ms typical. | -**Problematic — measure before you trust it:** +**Problematic (measure before trusting):** | Substrate | Failure mode | | --- | --- | -| ZFS without a SLOG, consumer NVMe | Every sync write hits the ZIL, gated by transaction-group commit cadence that serializes across all pool consumers. Single-digit-ms idle, **5–25 s under concurrent load**. | -| Loopback / qcow2 on `ext4` inside a VM | Stacks a second journaling layer; often 10× slower than direct `ext4` and highly variable. | -| Spinning disks | Mechanical seek on the NAND-equivalent program path: multi-millisecond baseline, multi-second tail. | +| ZFS without SLOG, consumer NVMe | Sync writes hit the ZIL; gated by transaction-group commit cadence. **5–25 s under load**. | +| Loopback / qcow2 on `ext4` in VM | Double journaling layer; often 10× slower than direct `ext4`. | +| Spinning disks | Mechanical seek: multi-millisecond baseline, multi-second tail. | -The tell for a commit-cadence problem (ZFS-without-SLOG, noisy-neighbor VM host) is that a single-threaded benchmark looks fine while a concurrent one is far worse — so always benchmark with multiple writers, and benchmark the guest **and** the host if virtualized. +Benchmark with multiple writers and check both guest and host if virtualized to surface commit-cadence problems. ## Check your storage before you trust it -Replicate JetStream's exact pattern — a 4 KiB write followed by a flush, in a tight loop — and report the percentiles. The numbers that matter are **p99** and **max**: those are your worst-case publish latency. - -On Linux, [`fio`](https://fio.readthedocs.io/) (packaged on every distro) is the honest, standard tool. Point it at the volume that backs `<data_dir>/nats`, ideally before WaveHouse is running: +Replicate JetStream's pattern—a 4 KiB write followed by a flush in a tight loop. Use [`fio`](https://fio.readthedocs.io/) on the volume backing `<data_dir>/nats` under representative load: ```bash # 8 concurrent writers — the variant that surfaces commit-cadence problems @@ -69,35 +63,31 @@ fio --name=jetstream-fsync --directory=/var/lib/wavehouse/nats \ --numjobs=8 --group_reporting ``` -Run it under representative load, not on an idle box — idle benchmarks understate real-world tails. - -Read the measured p99 against these bands, which track WaveHouse's `SyncAlways` default: - | p99 `fsync` | Verdict for `SyncAlways: true` | | ---: | --- | | < 1 ms | **Ideal** | | 1–5 ms | **Good** | | 5–50 ms | **Workable** — watch bursty load | -| 50 ms – 1 s | **Marginal** — relax durability once `mq.sync_interval` ([#139](https://github.com/Wave-RF/WaveHouse/issues/139)) lands, or move to faster storage | -| > 1 s | **Broken** — `create stream` will time out under load; fix the storage substrate | +| 50 ms – 1 s | **Marginal** — relax durability via `mq.sync_interval` ([#139](https://github.com/Wave-RF/WaveHouse/issues/139)) or upgrade storage | +| > 1 s | **Broken** — `create stream` will time out; fix substrate | :::caution[macOS `fsync` lies by default] -A plain `fsync()` on macOS returns once data is in the drive's volatile cache — it does **not** force a flush to NAND; only `fcntl(fd, F_FULLFSYNC)` does (NATS, Postgres, and SQLite all use it). On a Mac, any per-flush number under ~1 ms is almost certainly not a real flush — the gap between plain `fsync()` and `F_FULLFSYNC` can be ~180× on the same consumer NVMe. `fio` on macOS calls plain `fsync()`, so don't trust Mac `fio` numbers for tail-latency planning. This mostly matters when benchmarking a dev machine; production WaveHouse runs on Linux, where `fio` is honest. +Plain `fsync()` on macOS returns once data is in volatile cache, not NAND; only `fcntl(fd, F_FULLFSYNC)` forces a flush. `fio` on macOS uses plain `fsync()`, making its numbers unreliable for tail-latency planning. Production WaveHouse runs on Linux, where `fio` is honest. ::: -A self-contained `wavehouse storage-check` preflight subcommand that bakes this measurement and verdict into the binary — including the per-platform honest flush — is tracked in [#84](https://github.com/Wave-RF/WaveHouse/issues/84). +A `wavehouse storage-check` preflight subcommand is tracked in [#84](https://github.com/Wave-RF/WaveHouse/issues/84). ## Symptoms of storage that can't keep up -If you see any of these, benchmark the `<data_dir>/nats` volume as above: +Benchmark `<data_dir>/nats` if you observe: - `create stream: ... context deadline exceeded` at startup. -- Ingest p99 latency in the seconds, or occasional `200`s that take multiple seconds to return. -- Intermittent `503 Service Unavailable` from `/v1/ingest` when ClickHouse is healthy (the worker can't drain fast enough because acking is `fsync`-bound). -- Flaky CI or load tests that pass on fast storage and fail on a shared/virtualized host. +- Ingest p99 latency or occasional `200` responses taking multiple seconds. +- Intermittent `503 Service Unavailable` from `/v1/ingest` while ClickHouse is healthy. +- Flaky CI/load tests that pass on fast storage but fail on shared/virtualized hosts. ## See also -- [Configuration → Message Queue (NATS)](/configuration#message-queue-nats) — the `mq.*` knobs (`gap_window_minutes`, `max_bytes_gb`). -- [Deployment → Persistent Storage](/deployment#persistent-storage-required-for-containers) — `data_dir` must resolve to a host-backed volume. -- [Ingest Pipeline → Backpressure and durability knobs](/ingest-pipeline#backpressure-and-durability-knobs) — the worker-side ack cost and the in-flight backpressure layers. +- [Configuration → Message Queue (NATS)](/configuration#message-queue-nats) — `mq.*` knobs (`gap_window_minutes`, `max_bytes_gb`). +- [Deployment → Persistent Storage](/deployment#persistent-storage-required-for-containers) — `data_dir` requirements. +- [Ingest Pipeline → Backpressure and durability knobs](/ingest-pipeline#backpressure-and-durability-knobs) — worker-side ack costs. diff --git a/docs/src/content/docs/getting-started.md b/docs/src/content/docs/getting-started.md index 5ae0e978..a36a7462 100644 --- a/docs/src/content/docs/getting-started.md +++ b/docs/src/content/docs/getting-started.md @@ -5,17 +5,17 @@ sidebar: order: 2 --- -Run WaveHouse locally in under five minutes. WaveHouse ships as a single binary with ClickHouse as the only external dependency; this walkthrough covers ingest, query, and real-time streaming. +Run WaveHouse locally in under five minutes. It ships as a single binary with ClickHouse as the only external dependency. ## Prerequisites -- **Docker** — for running ClickHouse (and optionally WaveHouse itself). -- **curl** and **jq** (optional) — for poking the API. -- **Go 1.26+** — only required if you want to build from source; skip it for the Docker path below. +- **Docker** — for ClickHouse (and optionally WaveHouse). +- **curl** and **jq** (optional) — for API testing. +- **Go 1.26+** — only if building from source. ## 1. Start WaveHouse -The fastest path uses Docker Compose — it launches ClickHouse and a single `wavehouse` process. +Use Docker Compose to launch ClickHouse and `wavehouse`: ```bash git clone https://github.com/Wave-RF/WaveHouse.git @@ -23,16 +23,16 @@ cd WaveHouse docker compose -f deployments/compose/standalone.yaml up -d ``` -This exposes: +Exposed ports: -- WaveHouse API on `http://localhost:8080` -- ClickHouse on ports `8123` (HTTP) and `9000` (native) +- WaveHouse API: `http://localhost:8080` +- ClickHouse: `8123` (HTTP) and `9000` (native) -WaveHouse is **fail-closed** — with no policy loaded, every request is denied. So the standalone stack ships a permissive **trial policy** (`deployments/compose/dev-policy.yaml`, mounted read-only and wired in via `WH_POLICY_FILE_PATH`): a non-admin [`public` role](/access-control#default_role--public-unauthenticated-access) that can read and write the demo tables (`clicks`, `events`) with no token, so the quickstart just works. It's *not* admin — it can't run raw SQL or manage policy/pipes — and it names specific tables, so it grants nothing in a real deployment (those tables won't exist there). It seeds into NATS KV on first boot; after that KV is authoritative (see [Access Control — Bootstrapping](/access-control#bootstrapping-and-the-policy-lifecycle)). It's deliberately lenient for trialing — a real deployment should [tune it](/access-control): your own roles, real tables, scoped columns, and usually tokens instead of a public default. +WaveHouse is **fail-closed** — with no policy loaded, every request is denied. So the standalone stack ships a permissive **trial policy** (`deployments/compose/dev-policy.yaml`, mounted read-only, wired via `WH_POLICY_FILE_PATH`): a non-admin [`public` role](/access-control#default_role--public-unauthenticated-access) with read/write on the demo tables (`clicks`, `events`), no token needed. It's *not* admin — it can't run raw SQL or manage policy/pipes — and it names specific tables, so it grants nothing in a real deployment. It seeds into NATS KV on first boot; thereafter KV is authoritative ([Access Control — Bootstrapping](/access-control#bootstrapping-and-the-policy-lifecycle)). For production, [tune it](/access-control): your own roles, real tables, scoped columns, and tokens instead of a public default. ## 2. Create a ClickHouse table -WaveHouse uses a **Bring Your Own Schema** model — you create tables in ClickHouse, and WaveHouse discovers them automatically via `system.columns`. +WaveHouse uses **Bring Your Own Schema**; it discovers tables via `system.columns`. ```bash docker compose -f deployments/compose/standalone.yaml exec clickhouse \ @@ -47,11 +47,11 @@ docker compose -f deployments/compose/standalone.yaml exec clickhouse \ " ``` -Schemas refresh every 60 seconds by default, or on demand via `POST /v1/schema/refresh` (admin-only). If the first ingest below returns `404 unknown table: clicks`, the refresh simply hasn't picked the new table up yet — wait and retry (worst case the next refresh is a full 60 seconds out). +Schemas refresh every 60 seconds or via `POST /v1/schema/refresh` (admin-only). If ingest returns `404 unknown table: clicks`, wait for the next refresh. ## 3. Ingest an event -The JWT middleware always runs, but with no secret configured (the default) every request resolves to the policy `default_role` — which the trial policy from step 1 maps to the `public` role (granted insert on the demo tables). So you can POST straight to `/v1/ingest?table=clicks` with no token: +Without a configured secret, requests resolve to `default_role`, which the trial policy maps to `public`. POST directly to `/v1/ingest?table=clicks`: ```bash curl -s -X POST "http://localhost:8080/v1/ingest?table=clicks" \ @@ -60,11 +60,11 @@ curl -s -X POST "http://localhost:8080/v1/ingest?table=clicks" \ # → {"ok":true} ``` -WaveHouse validates the body against the ClickHouse schema before acknowledging. Unknown fields, type mismatches, and missing required columns are rejected with a `400`. +WaveHouse validates the body against the ClickHouse schema; unknown fields, type mismatches, and missing required columns return `400`. ## 4. Query -The trial `public` role can read the demo tables, so query `clicks` with the structured-query endpoint — no token needed: +Query `clicks` using the structured-query endpoint: ```bash # Wait ~5 seconds for the batch flush to ClickHouse, then query: @@ -73,15 +73,15 @@ curl -s -X POST "http://localhost:8080/v1/query?table=clicks" \ -d '{"columns": ["page", "button", "score"], "limit": 10}' ``` -`POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached in-process (L1 Ristretto) with singleflight coalescing — duplicate concurrent queries hit ClickHouse once. For raw SQL there's `POST /v1/admin/query` (an admin escape hatch that never caches, emitting `Cache-Control: no-store`), but it's **admin-only** — the trial `public` role can't reach it. To use it, swap the public default for real auth: configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). +`POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached in-process (L1 Ristretto) with singleflight coalescing — duplicate concurrent queries hit ClickHouse once. Raw SQL goes through `POST /v1/admin/query`, an admin escape hatch that never caches and emits `Cache-Control: no-store`; the trial `public` role can't reach it. To use it, configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). :::tip[Prefer a type-safe client?] -The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). +The [TypeScript SDK](/sdk) provides a chainable query builder with autocomplete. See the [structured query reference](/api#post-v1querytabletable--structured-query) for raw shapes. ::: ## 5. Subscribe to real-time updates -Every ingested event is broadcast to SSE subscribers **before** it's flushed to ClickHouse, so dashboards see new data with zero perceived lag. +Events are broadcast via SSE **before** flushing to ClickHouse: ```bash # Specific table (?table= is required) @@ -93,24 +93,22 @@ curl -N "http://localhost:8080/v1/stream?table=clicks&since=2026-03-24T11:00:00Z ## Troubleshooting first runs -The handful of things that most often trip up a first session — each is expected behavior with a quick fix: - -- **`404 unknown table: clicks` on the first ingest.** Schema discovery refreshes every 60 seconds (`WH_SCHEMA_REFRESH_INTERVAL`), so a just-created table may not be visible yet. Wait and retry — worst case the next refresh is a full 60 seconds out. (`POST /v1/schema/refresh` forces it, but that endpoint is admin-only — the trial `public` role can't call it.) -- **The query returns `[]` right after an ingest succeeded.** Ingest acknowledges as soon as the event is durable in the WAL; the batch worker flushes to ClickHouse every few seconds. If you query within that window the rows simply aren't in ClickHouse yet — re-query after ~5 seconds. (The [SSE stream](#5-subscribe-to-real-time-updates) sees events *immediately* — it's broadcast before the flush.) -- **`403` on a table you created yourself.** WaveHouse is fail-closed and the trial policy grants the `public` role access to the *named demo tables only* (`clicks`, `events`). A new table needs a policy entry — see [Access Control](/access-control) for granting roles per table. -- **A port is already taken.** The stack binds `8080` (WaveHouse) and `8123`/`9000` (ClickHouse). Stop whatever holds the port or edit the `ports:` mappings in `deployments/compose/standalone.yaml`. -- **Errors right at first boot.** `docker compose -f deployments/compose/standalone.yaml ps` should show both services up — ClickHouse takes a few seconds to initialize on a cold start, so give the stack a moment before the first request. +- **`404 unknown table: clicks`**: Schema discovery refreshes every 60s (`WH_SCHEMA_REFRESH_INTERVAL`). Wait and retry. +- **Query returns `[]` after ingest**: Ingest is durable in the WAL immediately, but batch workers flush to ClickHouse every few seconds. Re-query after ~5 seconds or use [SSE stream](#5-subscribe-to-real-time-updates). +- **`403` on custom tables**: The trial policy only grants access to `clicks` and `events`. See [Access Control](/access-control) to grant permissions for new tables. +- **Port conflict**: Stop processes using `8080`, `8123`, or `9000`, or edit `deployments/compose/standalone.yaml`. +- **Boot errors**: `docker compose -f deployments/compose/standalone.yaml ps` should show both services up; ClickHouse takes a few seconds to initialize on a cold start. ## Next steps -- **[Architecture](/architecture)** — how ingest, query, cache, and streaming fit together. -- **[API Reference](/api)** — every endpoint, request/response shape, and error code. -- **[TypeScript SDK](/sdk)** — zero-dependency client with query builder, live queries, and codegen. -- **[Configuration](/configuration)** — full YAML + environment variable reference. -- **[Deployment](/deployment)** — Docker images, releases, health checks. -- **[Development](/development)** — building from source, running tests, hot-reload workflow. +- **[Architecture](/architecture)** — Ingest, query, cache, and streaming flow. +- **[API Reference](/api)** — Endpoints, shapes, and error codes. +- **[TypeScript SDK](/sdk)** — Client with query builder and codegen. +- **[Configuration](/configuration)** — YAML and environment variables. +- **[Deployment](/deployment)** — Images, releases, and health checks. +- **[Development](/development)** — Building from source and testing. ## Going further -- **Validate JWTs**: set `WH_AUTH_JWT_SECRET=<secret>` (the middleware always runs; without a secret every request is the policy `default_role`) and replace the shipped trial policy (`deployments/compose/dev-policy.yaml`) with a least-privilege one — see [API Reference — Authentication](/api#authentication) and [Access Control](/access-control). -- **Enable deduplication**: set `WH_DEDUPE_ENABLED=true` and `WH_DEDUPE_ID_FIELD=event_id` — see [Configuration — Deduplication](/configuration#deduplication). +- **Validate JWTs**: Set `WH_AUTH_JWT_SECRET=<secret>` and replace the trial policy with a least-privilege one (see [Authentication](/api#authentication) and [Access Control](/access-control)). +- **Enable deduplication**: Set `WH_DEDUPE_ENABLED=true` and `WH_DEDUPE_ID_FIELD=event_id` ([Configuration — Deduplication](/configuration#deduplication)). diff --git a/docs/src/content/docs/ingest-pipeline.md b/docs/src/content/docs/ingest-pipeline.md index a4e6a9c9..36ff1ff9 100644 --- a/docs/src/content/docs/ingest-pipeline.md +++ b/docs/src/content/docs/ingest-pipeline.md @@ -15,24 +15,15 @@ goroutine / channel / timer interplay is subtle. | File | Contents | | --- | --- | -| `worker.go` | `StartIngestWorker`, the `dispatchLoop`, the per-table `tableBatcher`/`tableLoop`, `flushTable` (bulk insert with a row-by-row poison-isolation fallback), `insertToClickHouse`, `handleSuccess` (cache invalidation + acks), `sendToDLQ` | -| `sweeper.go` | The **Active Sweeper** — purges stream messages that are both written to ClickHouse and past the SSE gap window | -| `types.go` | `EventMessage` wire format and the `BufferConsumerName` constant | +| `worker.go` | `StartIngestWorker`, `dispatchLoop`, per-table `tableBatcher`/`tableLoop`, `flushTable` (bulk insert with row-by-row poison-isolation fallback), `insertToClickHouse`, `handleSuccess` (cache invalidation + acks), `sendToDLQ` | +| `sweeper.go` | **Active Sweeper**: purges stream messages written to ClickHouse and past the SSE gap window | +| `types.go` | `EventMessage` wire format, `BufferConsumerName` constant | -The pipeline is **insert-only**. The wire format carries -`{table_name, received_timestamp, data}` and nothing else; the worker parses -the envelope and bulk-`INSERT`s — schema validation already happened at the -HTTP ingest handler, before publish. Non-insert mutations go through a -different admin path. +The pipeline is **insert-only**. The `{table_name, received_timestamp, data}` wire format is parsed by the worker for bulk-`INSERT`. HTTP ingest handlers validate schemas before publishing. Non-insert mutations use a separate admin path. ## High-level shape -One process consumes a single durable JetStream consumer and fans events out to -a goroutine per table. Each table batches independently and POSTs to ClickHouse -over the HTTP interface (`JSONEachRow`). On a bulk-insert failure the batch is -re-inserted row by row, so a single poison row can't sink it: clean rows ack, -and only the rows that fail again go to the dead-letter stream. A separate -sweeper reclaims stream storage. +One process consumes a single durable JetStream consumer and fans events to one goroutine per table. Each table batches independently and POSTs to ClickHouse via HTTP (`JSONEachRow`). On bulk-insert failure, rows are re-inserted individually: clean rows ack, while failing rows go to the dead-letter stream. A separate sweeper reclaims storage. ```mermaid flowchart LR @@ -62,29 +53,10 @@ flowchart LR Stream -.->|"DeliverByStartTime gap-fill"| Hub["hub-bridge consumer<br/>(SSE fan-out)"] ``` -Note the stream is **dual-use**: it is both the durable buffer feeding the -worker and the replay buffer that SSE clients gap-fill from. That is why a -custom sweeper exists instead of plain work-queue auto-deletion (see -[Scaling out](#scaling-to-multiple-instances)). +The stream is **dual-use**: a durable buffer for the worker and a replay buffer for SSE client gap-fills. Thus, a custom sweeper replaces work-queue auto-deletion (see [Scaling out](#scaling-to-multiple-instances)). :::note[ClickHouse timestamp parsing] -Inserts pin `date_time_input_format=best_effort` — the server default since -ClickHouse 26.5, but on older servers the `basic` default rejects the canonical -RFC 3339 form's `Z` suffix ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). -The ordinary spellings (zone-less date-times, 9–10-digit Unix-seconds strings) -parse identically under both settings, so pre-canonical messages still in the -stream replay unchanged. Bare digit-strings of other lengths are the exception: -`best_effort` reads them as ClickHouse's calendar/epoch shapes, where `basic` -read a plain `DateTime` column's digit string of five or more digits as Unix -seconds (shorter runs it rejected outright, where `best_effort` reads `"2026"` -as a year): under `best_effort` `"20260711"` stores 2026-07-11, where `basic` -stored 1970-08-23. `DateTime64` columns diverge the same way on calendar-shaped -runs, and additionally whenever an epoch run's unit doesn't match the column -scale (under `basic`, runs longer than 10 digits are ticks at the column's -own scale; `best_effort` unit-detects 13/16/19-digit runs as ms/µs/ns). A -producer relying on the old `basic` reading changes meaning as soon as this -WaveHouse version is deployed — the pin, not a ClickHouse upgrade, is what -flips the parse. +Inserts pin `date_time_input_format=best_effort`—the server default since ClickHouse 26.5; older servers' `basic` default rejects RFC 3339 `Z` suffixes ([#372](https://github.com/Wave-RF/WaveHouse/issues/372)). Zone-less date-times and 9–10-digit Unix strings parse identically under both. However, `best_effort` reads calendar shapes (e.g., `"20260711"`) as dates, whereas `basic` read them as Unix seconds. For `DateTime64`, they diverge on calendar shapes and when epoch run units mismatch column scale (`basic` treats >10 digits as column-scale ticks; `best_effort` detects 13/16/19-digit runs as ms/µs/ns). Producers relying on `basic` parsing change meaning upon deploying this WaveHouse version via the pin. ::: ## The journey of one event @@ -112,8 +84,7 @@ sequenceDiagram ## Goroutine topology -The design rule is **single-owner state, lock-free**: each piece of mutable -state is touched by exactly one goroutine. There are no mutexes in the hot path. +Design rule: **single-owner state, lock-free**. One goroutine touches each piece of mutable state; no mutexes exist in the hot path. ```mermaid flowchart TD @@ -127,34 +98,21 @@ flowchart TD F2 --> A2["ack goroutines (ackWg)"] ``` -Three `WaitGroup`s form a strict containment hierarchy, which is what makes -shutdown correct (below): +Three `WaitGroup`s ensure correct shutdown via a strict containment hierarchy: -- **`wg`** tracks the `dispatchLoop` goroutine. -- **`tableWg`** (owned by `dispatchLoop`) tracks the per-table `tableLoop`s. -- **`ackWg`** tracks the background `DoubleAck` goroutines. +- **`wg`**: tracks `dispatchLoop`. +- **`tableWg`** (owned by `dispatchLoop`): tracks per-table `tableLoop`s. +- **`ackWg`**: tracks background `DoubleAck` goroutines. ## Why per table? The bug this design fixes -A single shared batch across all tables couples them: a high-volume table can -trip the size trigger and strand a low-volume table's rows in a batch that then -waits for the time trigger, and vice-versa. Routing each table to its own -`tableLoop` gives every table an **independent** size trigger and timer, so one -table's traffic never delays another's. (`dispatchLoop` does no batching itself -— it only parses enough to pick the route key.) +Shared batches couple tables: high-volume tables can trip size triggers, stranding low-volume rows until time triggers fire. Routing each table to its own `tableLoop` provides **independent** size triggers and timers; one table's traffic never delays another's. (`dispatchLoop` does no batching—it only parses enough to pick the route key.) ## The `tableBatcher` state machine -Each `tableLoop` owns a `tableBatcher`. It has exactly two flush **triggers** — -the batch reaching `maxBatch` (checked in `add`) and the `maxWait` deadline -timer — plus a rule that **at most one insert runs per table at a time** -("coalescing"). A flush *completing* is **not** a trigger. +Each `tableLoop` owns a `tableBatcher`. Flushes are triggered by reaching `maxBatch` (checked in `add`) or the `maxWait` deadline timer. Only one insert runs per table at a time ("coalescing"); a flush completing is not a trigger. -The `flushing` channel signals "an insert is in flight" (it is `nil` when idle — -and receiving from a `nil` channel blocks forever, so the loop's `<-flushing` -arm is automatically inert while idle). The `flushQueued` flag **latches** a -trigger that fires while an insert is already running, so the deferred flush runs -the moment the slot frees. +The `flushing` channel signals an active insert; it is `nil` when idle, making the loop's `<-flushing` arm inert. The `flushQueued` flag latches triggers that fire during an active insert, ensuring a deferred flush runs once the slot frees. ```mermaid stateDiagram-v2 @@ -170,23 +128,14 @@ stateDiagram-v2 FlushingQueued --> Flushing: done — deferred flush ``` -Two consequences worth internalizing: +Two key consequences: -- **`maxBatch` is a "try to flush" threshold, not a hard cap.** If rows keep - arriving while a flush is in flight, the next batch coalesces and can exceed - `maxBatch`, flushing as one larger insert when the slot frees. That is *good* - for ClickHouse part pressure (bigger batches when hot) and is bounded upstream - by `maxAckPending`. -- **A partial leftover after a size flush waits for its own size/timer.** When - 500 rows flush and 100 remain, those 100 do **not** flush just because the - first insert finished — they wait for their own `maxBatch` or `maxWait`. The - `flushQueued` latch is also what prevents a stranding bug: if the leftover's - timer fires *during* the in-flight insert, the latch remembers it so the rows - still flush when the slot frees (rather than the tick being silently lost). +- **`maxBatch` is a "try to flush" threshold, not a hard cap.** If rows arrive during a flush, the next batch coalesces and may exceed `maxBatch`, flushing as one larger insert. This reduces ClickHouse part pressure and is bounded upstream by `maxAckPending`. +- **Leftovers wait for their own size/timer.** If 500 rows flush and 100 remain, those 100 wait for `maxBatch` or `maxWait`. The `flushQueued` latch prevents stranding: if a leftover's timer fires during an insert, the latch ensures they flush when the slot frees. ### Why there are no data races -When a flush starts, the batcher hands the goroutine a **private snapshot**: +Flushes use a **private snapshot**: ```go rows := b.batch // snapshot the slice header @@ -194,16 +143,11 @@ b.batch = nil // fresh batch here; appends allocate a new backing array go func() { w.flushTable(ctx, b.table, rows) }() ``` -The flush goroutine only ever touches `rows` and the worker's -concurrency-safe collaborators (HTTP client, cache, `ackWg`); it never touches -`b.batch`, `b.timer`, or `b.flushing`. Those are touched solely by the -`tableLoop` goroutine. `b.batch = nil` (rather than `b.batch[:0]`) is load-bearing -— reusing the array would let new appends overwrite rows the flush is still -reading. The race detector (`go test -race`) guards this. +The flush goroutine only touches `rows` and concurrency-safe collaborators (HTTP client, cache, `ackWg`). It never touches `b.batch`, `b.timer`, or `b.flushing`, which are exclusive to the `tableLoop` goroutine. Setting `b.batch = nil` instead of `b.batch[:0]` prevents new appends from overwriting rows being read. The race detector (`go test -race`) guards this. ## Contexts -There are three contexts, each with one job. +Three contexts each perform one job. ```mermaid flowchart LR @@ -213,28 +157,17 @@ flowchart LR FC -.->|"passed down; never watched"| TL["tableLoops + flushes"] ``` -- **`workerCtx`** is the stop signal. **Only `dispatchLoop` watches it.** Every - downstream goroutine stops via channel-close instead, which gives a - deterministic drain with no select race that could abandon buffered rows. -- **`flushCtx` = `context.WithoutCancel(workerCtx)`** carries trace values but is - never canceled. A flush that has started must finish, so data already written - to ClickHouse gets acked rather than redelivered. It is bounded by the HTTP - client timeout (30s); shutdown bounds the *wait* for it with a deadline. -- A separate **shutdown-deadline context** lives only in `main`'s signal handler - and is rooted in `context.Background()` (so it survives `workerCtx` being - canceled) — it caps how long shutdown waits. +- **`workerCtx`**: The stop signal. Only `dispatchLoop` watches it. Downstream goroutines stop via channel-close to ensure deterministic drain without select races abandoning buffered rows. +- **`flushCtx` = `context.WithoutCancel(workerCtx)`**: Carries trace values; never canceled. Started flushes must finish so ClickHouse data is acked, not redelivered. It is bounded by the 30s HTTP client timeout; shutdown bounds the wait via a deadline. +- **Shutdown-deadline context**: Rooted in `context.Background()` within `main`'s signal handler to survive `workerCtx` cancellation and cap shutdown wait time. -The principle: **`ctx` cancellation is the stop mechanism for long-running -loops; `Close()`/stop-funcs are the mechanism for resources.** +Principle: **`ctx` cancellation stops long-running loops; `Close()`/stop-funcs manage resources.** ## Lifecycle and shutdown -Startup: `StartIngestWorker` creates the consumer, builds the worker, and -launches `dispatchLoop`. It returns a `stopFunc` closure that `main` holds and -calls during graceful shutdown. +Startup: `StartIngestWorker` creates the consumer, builds the worker, and launches `dispatchLoop`, returning a `stopFunc` closure for `main` to call during graceful shutdown. -Shutdown drains **bottom-up through the containment hierarchy**, under a single -deadline: +Shutdown drains **bottom-up through the containment hierarchy** under one deadline: ```mermaid sequenceDiagram @@ -255,53 +188,35 @@ sequenceDiagram SF-->>M: waitOrDeadline returns nil (or deadline error) ``` -Why this ordering is correct: every `ackWg.Add` happens inside a `tableLoop`'s -lifetime, so all of them complete before `tableWg.Wait()` returns — which means -`dispatchLoop` can safely `ackWg.Wait()` afterward with no `Add` racing `Wait`. -The old code relied on "flush runs synchronously" for this; the hierarchy makes -it structural instead. +This ordering is correct because every `ackWg.Add` occurs within a `tableLoop`'s lifetime; all complete before `tableWg.Wait()` returns, ensuring `dispatchLoop` can safely call `ackWg.Wait()` without racing `Add`. This structural hierarchy replaces the old reliance on synchronous flushes. -If the deadline fires first, `waitOrDeadline` returns the deadline error and the -in-flight goroutines are abandoned — the process is exiting anyway, and anything -un-acked is redelivered on the next boot (at-least-once). +If the deadline fires, `waitOrDeadline` returns a deadline error and in-flight goroutines are abandoned; un-acked data is redelivered next boot (at-least-once). -Messages still sitting in `msgChan` or the consumer's prefetch buffer at shutdown -are **not** flushed; they are simply redelivered next boot. Graceful shutdown -flushes the in-hand per-table batches, not the entire in-flight pipeline. +Messages in `msgChan` or the consumer's prefetch buffer at shutdown are **not** flushed—only in-hand per-table batches are. ## Backpressure and durability knobs -Several layers throttle the pipeline, inner to outer: +Pipeline throttling layers (inner to outer): -1. **`batch`** flushes at `maxBatch` rows or `maxWait`. -2. **`msgChan`** (cap `maxBatch*2`) — when full, the consume callback blocks and - delivery pauses. -3. **`pullMaxMessages`** — nats.go's client-side prefetch buffer in front of - `msgChan`. -4. **`maxAckPending`** — the server suspends delivery once this many messages are - delivered-but-unacked. The outermost in-memory bound. -5. **`MaxBytes` + `DiscardNew`** on the stream — when disk fills (e.g. ClickHouse - is down so nothing acks/purges), new publishes are rejected and the API - returns 503. +1. **`batch`**: flushes at `maxBatch` rows or `maxWait`. +2. **`msgChan`** (cap `maxBatch*2`): when full, consume callback blocks and delivery pauses. +3. **`pullMaxMessages`**: nats.go client-side prefetch buffer before `msgChan`. +4. **`maxAckPending`**: server suspends delivery once this many messages are delivered-but-unacked; the outermost in-memory bound. +5. **`MaxBytes` + `DiscardNew`** (stream): if disk fills (e.g. ClickHouse down), new publishes are rejected via 503 API errors. | Knob | Default | Meaning / invariant | | --- | --- | --- | -| `maxBatch` | 500 | rows that trigger a flush (soft — coalescing can exceed it) | -| `maxWait` | 5s | max time a row waits before its batch flushes | -| `ackWait` | 60s | server redelivery timeout; **must exceed `maxWait` + flush time** or in-flight rows get redelivered → duplicate inserts | +| `maxBatch` | 500 | rows triggering flush (soft; coalescing may exceed) | +| `maxWait` | 5s | max row wait before batch flushes | +| `ackWait` | 60s | server redelivery timeout; **must exceed `maxWait` + flush time** to prevent duplicate inserts | | `pullMaxMessages` | 500 | client prefetch; keep `<= maxAckPending` | -| `maxAckPending` | 10,000 | server cap on unacked messages (backpressure) | +| `maxAckPending` | 10,000 | server unacked message cap (backpressure) | -`DoubleAck` is used (not fire-and-forget `Ack`) because acking is what records -"this data is durably in ClickHouse." With the embedded server's `SyncAlways`, -every ack is an fsync and therefore *slow*, which is exactly why acks run in the -background (`ackWg`) off the insert path. +`DoubleAck` records durable ClickHouse storage. With embedded server `SyncAlways`, every ack is a slow fsync; thus, acks run backgrounded via `ackWg` off the insert path. ## The Active Sweeper -The worker advances the consumer's `AckFloor` by acking; the sweeper observes it -to decide what is safe to purge. They never call each other — the consumer's -`AckFloor` is their only contract. +The worker advances the consumer's `AckFloor` by acking; the sweeper observes it to decide what to purge. They communicate only via `AckFloor`. ```mermaid flowchart TD @@ -312,19 +227,11 @@ flowchart TD Purge -->|"deletes msgs that are BOTH<br/>written to ClickHouse AND past the gap window"| Stream[("WAVEHOUSE stream")] ``` -`MIN(ackFloor+1, gapSeq)` is the safety argument: never purge past what is in -ClickHouse, and never past the SSE replay window. If ClickHouse is down the -`AckFloor` stops advancing, purging freezes, and the stream fills toward -`MaxBytes` — backpressure by construction. The sweeper is launched fire-and-forget -(`go sweeper.Start(ctx)`); an interrupted sweep is harmless and idempotent, so it -needs no drain on shutdown — the opposite posture from the worker. +`MIN(ackFloor+1, gapSeq)` ensures no purging past ClickHouse data or the SSE replay window. If ClickHouse fails, `AckFloor` stops, purging freezes, and the stream hits `MaxBytes`, creating backpressure. The sweeper starts via `go sweeper.Start(ctx)`; as an idempotent process, it requires no shutdown drain. ## Scaling to multiple instances -Today this is a **single-process** design (embedded, in-process NATS — the -"connection" cannot blip independently of the process, so there is intentionally -no reconnect logic). Running multiple instances against a real/clustered NATS -changes several things: +The current **single-process** design uses embedded NATS; since the connection cannot fail independently, there is no reconnect logic. Moving to clustered NATS requires several changes: ```mermaid flowchart TD @@ -341,48 +248,18 @@ flowchart TD IB --> CH ``` -What will need to change, and the trade-offs (discussed at length on the -batching work): - -- **Work distribution.** Either a *shared* durable pull consumer (competing - consumers — coordination-free, but a hot table's rows spread across instances, - shrinking per-instance batches), or **partitioned consumer groups** that hash - by table-name subject token so a table always lands on one owner (pinned - consumer → per-table affinity + automatic failover, at the cost of an - assignment layer). -- **Idempotent inserts become mandatory.** At-least-once + redelivery-on-crash - means another instance can re-insert a batch the dead one had written but not - acked. Use `ReplacingMergeTree` (or a dedup key). The single-instance design - hides this today. -- **NATS resilience.** Remote NATS needs explicit reconnect/backoff and a - `Consume` error handler — none of which the embedded path needs. -- **The sweeper.** Its single-`AckFloor` model assumes one consumer. With - per-table/partition consumers you either rework it to purge below the *minimum* - AckFloor across consumers, or — cleaner — **split the dual-use stream**: a - `WorkQueuePolicy` work stream (auto-deletes on ack, no sweeper) plus a - `MaxAge` replay stream (server-expired by time, no sweeper), joined by stream - sourcing. That deletes the sweeper and its leader-election problem entirely, - at the cost of duplicating the in-flight overlap on disk. +Required changes and trade-offs: + +- **Work distribution.** Use either a *shared* durable pull consumer (competing consumers; coordination-free, but shrinks per-instance batches) or **partitioned consumer groups** hashing by table-name subject token (pinned consumer; provides per-table affinity and automatic failover via an assignment layer). +- **Idempotent inserts.** Mandatory due to at-least-once delivery and redelivery-on-crash. Use `ReplacingMergeTree` or a dedup key. +- **NATS resilience.** Remote NATS requires explicit reconnect/backoff and a `Consume` error handler. +- **The sweeper.** Replace the single-`AckFloor` model by **splitting the dual-use stream**: use a `WorkQueuePolicy` work stream (auto-deletes on ack) and a `MaxAge` replay stream (server-expired). This removes the sweeper and leader-election issues, though it duplicates in-flight overlap on disk. ## Deferred / not yet implemented Tracked under [#191](https://github.com/Wave-RF/WaveHouse/issues/191): -- **Pipelining beyond coalescing** — more than one insert in flight per table - (with a documented bound), once benchmarks justify the added concurrency. -- **`tableLoop` reaping** — loops are spawned per distinct table and never - reaped; safe while table names are bounded (schema-validated, in-process - publishers only). Needs idle-reaping before untrusted/remote publishers can - create unbounded cardinality. -- **Per-table / partitioned consumers** and the **two-stream retention - redesign**. -- **Parallel e2e test files.** The e2e suite now isolates tables **per file** - (`tests/e2e/sdk/tables.ts` — each file gets its own - `clicks_<suite>`/`events_<suite>`/`users_<suite>`), so cross-file *data* - contamination is structurally impossible. Running the files in parallel - (dropping `maxWorkers: 1` in `vitest.config.ts`) is still deferred: several - files do read-modify-write on the **single global policy document** and - `streaming.test.ts` flips the global `default_role`, so concurrent files would - race those writes. Parallelism needs per-table policy storage with atomic - per-table updates first — tracked in - [#214](https://github.com/Wave-RF/WaveHouse/issues/214). +- **Pipelining beyond coalescing**: multiple in-flight inserts per table (with documented bound), pending benchmark justification. +- **`tableLoop` reaping**: loops spawn per distinct table and aren't reaped; safe for bounded, schema-validated, in-process publishers. Idle-reaping is required before allowing untrusted/remote publishers to create unbounded cardinality. +- **Per-table / partitioned consumers** and the **two-stream retention redesign**. +- **Parallel e2e test files**: tables are isolated per file (`tests/e2e/sdk/tables.ts` — each file gets its own `clicks_<suite>`/`events_<suite>`/`users_<suite>`), so data contamination is structurally impossible. Parallel execution (removing `maxWorkers: 1` in `vitest.config.ts`) is deferred; files race on the global policy document and `streaming.test.ts` flips the global `default_role`. Requires per-table policy storage with atomic updates ([#214](https://github.com/Wave-RF/WaveHouse/issues/214)). diff --git a/docs/src/content/docs/sdk/admin.md b/docs/src/content/docs/sdk/admin.md index 2a4971b5..48e8739b 100644 --- a/docs/src/content/docs/sdk/admin.md +++ b/docs/src/content/docs/sdk/admin.md @@ -3,11 +3,7 @@ title: "SDK Admin & System" description: "Schema introspection, access-control policy, DLQ stats, and health checks in @wavehouse/sdk." --- -Operational surfaces of `@wavehouse/sdk`. Everything here except -`wh.sys.health()` requires the admin role (`policy.admin_role`) — see -[Access Control](/access-control) for how roles resolve. -Examples import from `@wavehouse/sdk`; using the CDN instead, import from -`https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). +Operational surfaces of `@wavehouse/sdk`. Except for `wh.sys.health()`, all require the admin role (`policy.admin_role`)—see [Access Control](/access-control). Examples import from `@wavehouse/sdk`; for CDN, use `https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). ## Schema — `wh.schema` @@ -22,9 +18,9 @@ const { data: schemas } = await wh.schema.list(); await wh.schema.refresh(); ``` -Individual table schema is also available via `wh.from('clicks').schema()`. +Individual table schema is available via `wh.from('clicks').schema()`. -> `wh.schema.list()`, `wh.schema.refresh()`, and `wh.from(t).schema()` hit `/v1/schema*`, which are **admin-only** endpoints. Against any non-dev policy (anything but `default_role: admin`), construct the client with an admin-role token or these calls return `403`. +> `wh.schema.list()`, `wh.schema.refresh()`, and `wh.from(t).schema()` hit **admin-only** `/v1/schema*` endpoints. Use an admin-role token or these return `403` against non-dev policies. --- @@ -72,7 +68,7 @@ const { data } = await wh.dlq.list(); const { data } = await wh.dlq.table('clicks'); ``` -`wh.dlq.stream()` exists in the API but is **not yet functional**: there is no server-side DLQ stream today (the SSE bridge only carries `ingest.>` subjects), so it connects and receives no events — live DLQ streaming is tracked in [#197](https://github.com/Wave-RF/WaveHouse/issues/197). +`wh.dlq.stream()` is **not yet functional**; there is no server-side DLQ stream today (SSE only carries `ingest.>` subjects). Live streaming is tracked in [#197](https://github.com/Wave-RF/WaveHouse/issues/197). --- @@ -90,4 +86,4 @@ if (result.ok) { // on failure, result.error carries the reason (network vs. server error) ``` -> Readiness (`/readyz`) is intentionally **not** exposed through the SDK — it runs a ClickHouse query per call and is a load-balancer / reverse-proxy concern, not the client's. Probe `/readyz` directly from your orchestrator if you need it. +> Readiness (`/readyz`) is **not** exposed via SDK as it runs a ClickHouse query per call; probe `/readyz` directly from your orchestrator. diff --git a/docs/src/content/docs/sdk/pipes.md b/docs/src/content/docs/sdk/pipes.md index 3e1cca2f..be312bce 100644 --- a/docs/src/content/docs/sdk/pipes.md +++ b/docs/src/content/docs/sdk/pipes.md @@ -3,15 +3,11 @@ title: "SDK Pipes" description: "Execute and manage named query pipes with @wavehouse/sdk." --- -Named pipes are server-defined, parameterized queries — the -[Named Pipes guide](/pipes) covers defining them. The SDK executes pipes for -any allowed role and manages their definitions under the admin role. -Examples import from `@wavehouse/sdk`; using the CDN instead, import from -`https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). +Named pipes are server-defined, parameterized queries (see [Named Pipes guide](/pipes)). The SDK executes them for allowed roles and manages definitions via the admin role. Import from `@wavehouse/sdk` or `https://esm.sh/@wavehouse/sdk` ([Imports & Runtimes](/sdk#imports--runtimes)). ## Named Pipes — `wh.pipe(name, params?)` -Execute a pre-defined named query pipe. Returns a `PipeRef` which is **PromiseLike**. +Execute a pre-defined named query pipe. Returns a **PromiseLike** `PipeRef`. ```ts // These are equivalent (PipeRef is PromiseLike): @@ -25,7 +21,7 @@ Execute and return results. ### `.stream(opts?)` -Open a live stream. See [Streaming](/sdk/streaming). +Open a live stream (see [Streaming](/sdk/streaming)). --- diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index 0da64b91..e24a25ae 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -14,7 +14,7 @@ Examples import from `@wavehouse/sdk`; using the CDN instead, import from ## Tables — `wh.from(table)` -`from()` returns a `TableRef` — a reference to a table. It is **NOT thenable**, so it's safe to pass around or store in a variable without triggering requests. +`from()` returns a `TableRef`. **NOT thenable** — safe to store without triggering requests. ```ts const clicks = wh.from('clicks'); @@ -22,9 +22,9 @@ const clicks = wh.from('clicks'); ### `.fetch(opts?)` -Shortcut for "select every column", with a default limit of 1000. When an access-control policy restricts your role's columns, the server returns only the columns your role is allowed to read — `.fetch()` is never a way around `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). +Shortcut for "select every column", default limit 1000. Server-side `deny_columns`/`allow_columns` policies restrict returned columns and `.fetch()` cannot bypass them ([Access control](/access-control#column-permissions)). -To paginate, chain an explicit `.orderBy()` — a bare `.fetch()` sends no default order (see [Pagination](#pagination)). Ordering, grouping, or filtering by a column your role can't read is rejected, so a column-restricted role must reference only readable columns in those clauses. +Chain `.orderBy()` to paginate — bare `.fetch()` sends no default order (see [Pagination](#pagination)). Roles may only reference readable columns when ordering, grouping, or filtering. ```ts const { data, error, hasMore, next } = await clicks.fetch(); @@ -33,7 +33,7 @@ const { data } = await clicks.fetch({ limit: 50, signal: controller.signal }); ### `.insert(data, opts?)` -Insert one row or many. A single object is sent as a JSON `POST /v1/ingest?table={table}`. An **array** is serialized to NDJSON (one record per line) and sent as a single `application/x-ndjson` request, so a bad record no longer fails or hides the rest of the batch — per-record outcomes come back in the result. +Insert one or many rows. Single objects use JSON `POST /v1/ingest?table={table}`. Arrays serialize to NDJSON in one `application/x-ndjson` request returning per-record outcomes, so bad records don't fail the batch. ```ts // Single row → { ok: true } (or { ok: true, duplicate: true } when dedup skips it) @@ -47,13 +47,13 @@ const { data } = await clicks.insert([ // data: { ok, total, succeeded, failed, duplicates, results? } ``` -For an array insert, `data.ok` is `true` only when every record succeeded (`failed === 0`). Inspect `data.failed` and `data.results` (each `{ index, ok|duplicate|error }`, 1-based `index`) for partial failures — the call's top-level `error` is reserved for whole-request failures (network, `404` unknown table, `403` forbidden, `503` backpressure). An empty array is a no-op and sends no request. The array path sends one request regardless of size; bounded-concurrency chunking of very large arrays is tracked in [#196](https://github.com/Wave-RF/WaveHouse/issues/196). +For arrays, `data.ok` is `true` only if `failed === 0`. Use `data.failed` and `data.results` (each `{ index, ok|duplicate|error }`, 1-based `index`) for partial failures. Top-level `error` covers whole-request failures (network, `404` unknown table, `403` forbidden, `503` backpressure). Empty arrays are no-ops; chunking large arrays is tracked in [#196](https://github.com/Wave-RF/WaveHouse/issues/196). -> The server itself is format-agnostic: `POST /v1/ingest` also accepts a raw JSON array or a single object directly (the `Content-Type` is only a hint), so non-SDK clients can send whichever shape is convenient. See the [API reference](/api#post-v1ingesttabletable--ingest-data). +> The server is format-agnostic: `POST /v1/ingest` also accepts a raw JSON array or a single object (`Content-Type` is only a hint). See the [API reference](/api#post-v1ingesttabletable--ingest-data). ### `.insertNDJSON(source, opts?)` -Insert pre-formatted NDJSON you already have — a `.ndjson` file, a byte stream, or a string — without first parsing it into objects. Accepts a `string`, `Uint8Array`, `Blob`/`File`, or `ReadableStream<Uint8Array>`; non-string sources are read fully into memory before sending. Returns the same per-record summary as an array `insert`. +Insert pre-formatted NDJSON (`string`, `Uint8Array`, `Blob`/`File`, or `ReadableStream<Uint8Array>`) without parsing into objects. Non-string sources are read fully into memory. Returns the same per-record summary as array `insert`. ```ts // From a string @@ -69,7 +69,7 @@ await clicks.insertNDJSON(await openAsBlob('events.ndjson')); ### `.schema(opts?)` -Fetch the table's column definitions from ClickHouse. +Fetch table column definitions from ClickHouse. ```ts const { data } = await clicks.schema(); @@ -88,7 +88,7 @@ const { data } = await clicks.select('page', 'button').where('page', '=', '/home ### `.selectAll()` -Start a query that selects **every column your role is allowed to read** — the explicit form of what a bare `.fetch()` does. Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.); the server expands it to your allowed columns (never a raw `SELECT *`) and never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). +Selects every column your role may read — the explicit `.fetch()`. Mutually exclusive with `.select(...)` and aggregations (`.count()`, `.sum()`, etc.). The server expands it to allowed columns, never a raw `SELECT *` ([Column permissions](/access-control#column-permissions)). ```ts const { data } = await clicks.selectAll().where('country', '=', 'US').limit(10); @@ -102,11 +102,9 @@ Open a real-time event subscription. See [Streaming](/sdk/streaming). const stream = clicks.stream({ since: '2026-01-01T00:00:00Z' }); ``` ---- - ## Query Builder -Returned by `tableRef.select()`. Immutable — every chain method returns a new `QueryBuilder`. The builder is **PromiseLike**, so `await builder` auto-executes `.fetch()`. +Returned by `tableRef.select()`. Immutable — every chain method returns a new `QueryBuilder`. **PromiseLike**: `await builder` auto-executes `.fetch()`. ```ts // These are equivalent: @@ -116,11 +114,11 @@ const result = await clicks.select('page').limit(10); // PromiseLike shortcut ### Chain Methods -All methods return a new `QueryBuilder` — the original is unchanged. +All return a new `QueryBuilder`; the original is unchanged. #### `.select(...columns)` -Append columns to the SELECT clause. A literal `'*'` is the column *named* `*`, not a wildcard — use `.selectAll()` for all columns. +Append columns to the SELECT clause. A literal `'*'` is a column *named* `*`; use `.selectAll()` for all columns. ```ts const q = clicks.select('page').select('button'); // SELECT page, button @@ -128,7 +126,7 @@ const q = clicks.select('page').select('button'); // SELECT page, button #### `.selectAll()` -Select every column your role may read (the all-columns wildcard, expanded server-side to your allowed columns). Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.). +Selects every column your role may read (expanded server-side). Mutually exclusive with `.select(...)` and aggregations (`.count()`, `.sum()`, etc.). ```ts const q = clicks.selectAll().where('country', '=', 'US'); @@ -136,7 +134,7 @@ const q = clicks.selectAll().where('country', '=', 'US'); #### `.where(column, op, value)` -Add a filter condition. SDK operators are translated to backend format. +Add a filter condition. SDK operators translate to backend format. ```ts clicks.select('page').where('score', '>', 10).where('page', 'like', '/home%') @@ -152,7 +150,7 @@ clicks.select('page').where('score', '>', 10).where('page', 'like', '/home%') | `'<='` | `lte` | Less than or equal | | `'in'` | `in` | Value in array | | `'like'` | `like` | SQL LIKE pattern | -| `'not_like'` | — | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | +| `'not_like'` | — | SQL NOT LIKE — **client-side only** (live-query/stream); `/v1/query` rejects it | #### Aggregations @@ -167,8 +165,7 @@ clicks.select('page') .aggregate('uniqExact', 'user_id', 'unique_users') // custom fn ``` -Each aggregation method signature: `(column: string, alias?: string)`. -`count()` defaults to `column='*'`, `alias='count'`. +Signature: `(column: string, alias?: string)`. `count()` defaults to `column='*'`, `alias='count'`. #### `.groupBy(...columns)` @@ -190,11 +187,11 @@ clicks.select('page').count('*', 'total').orderBy('total', 'desc') clicks.select().limit(100) ``` -If no limit is specified, `QueryBuilder.DEFAULT_LIMIT` (1000) is applied automatically to prevent unbounded result sets. The server also enforces the configured maximum (`query.default_max_rows`, default 10,000 rows). +Defaults to `QueryBuilder.DEFAULT_LIMIT` (1000); server max is `query.default_max_rows` (10,000). #### `.timeRange(column, since, until?)` -Filter by a time window. `since` and `until` accept RFC3339 timestamps or relative durations (`'1h'`, `'30m'`, `'7d'`, `'2w'` — day and week suffixes expand to hours, so `'7d'` is `'168h'`). +Filter by time window. `since` and `until` take RFC3339 timestamps or relative durations (`'1h'`, `'30m'`, `'7d'`, `'2w'`; day/week suffixes expand to hours, so `'7d'` is `'168h'`). ```ts clicks.select('page').timeRange('received_timestamp', '1h') @@ -205,7 +202,7 @@ clicks.select('page').timeRange( #### `.cacheTTL(seconds)` -Records a desired result-cache TTL on the builder. **Currently client-side state only** — the value is never sent to the server, which derives each result's cache TTL adaptively from query execution time. Wiring it through the wire format is tracked in [#280](https://github.com/Wave-RF/WaveHouse/issues/280). +Desired result-cache TTL. **Client-side only**; the server derives TTL adaptively from execution time. See [#280](https://github.com/Wave-RF/WaveHouse/issues/280). ```ts clicks.select('page').count().cacheTTL(300) // not yet honored server-side — see #280 @@ -213,7 +210,7 @@ clicks.select('page').count().cacheTTL(300) // not yet honored server-side — s ### `.fetch(opts?)` -Execute the query. Returns `Result<Row[]>` with optional pagination. +Execute query. Returns `Result<Row[]>` with optional pagination. ```ts const { data, error, hasMore, next } = await clicks.select('page').limit(50).fetch(); @@ -227,16 +224,16 @@ if (hasMore && next) { | Field | Type | Description | |-------|------|-------------| -| `signal` | `AbortSignal` | Cancel the request | +| `signal` | `AbortSignal` | Cancel request | | `limit` | `number` | Override builder limit for this fetch | ### `.stream(opts?)` -Open a live stream from the builder's table. See [Streaming](/sdk/streaming). +Open a live stream from the table. See [Streaming](/sdk/streaming). ### Pagination -When `limit` is set and the result contains at least `limit` rows, `hasMore` is `true`. Cursor-based pagination's `next()` walks an **order column** — it adds a filter on that column using the last row's value — so `next()` is only attached when the query has an explicit `.orderBy()`. With no order column the result still reports `hasMore` honestly, but `next` is `undefined` (there is no deterministic cursor to build) — add an `.orderBy()` to paginate. +`hasMore` is `true` when results meet the set `limit`. Cursor pagination's `next()` needs an explicit `.orderBy()`; without it `next` is `undefined` though `hasMore` stays true. ```ts let result = await clicks.select().orderBy('received_timestamp', 'desc').limit(100).fetch(); @@ -248,16 +245,14 @@ while (result.hasMore && result.next) { } ``` ---- - ## Raw SQL — `wh.sql(query, opts?)` -Execute a raw SQL query. `/v1/admin/query` is admin-only: the caller's JWT must resolve to the policy admin role (`admin_role`, `"admin"` by default). A request with no token, or an invalid/expired one, falls back to the `default_role` and is rejected. +Execute raw SQL. `/v1/admin/query` needs a JWT resolving to the policy admin role (`admin_role`, default `"admin"`). Missing, invalid, or expired tokens fall back to `default_role` and are rejected. ```ts const { data, error } = await wh.sql('SELECT page, count() FROM clicks GROUP BY page LIMIT 10'); ``` :::note[No parameter binding through the SDK] -Positional `?` substitution is not supported, and the SDK has no way to forward ClickHouse-style named params (the `WHERE id = {id:UInt32}` + `param_id=42` query-string combo) — the proxy doesn't forward arbitrary query-string params and `wh.sql()` doesn't expose a hook to add them. Inline literals into the SQL, or — for safe binding from user-supplied input — use the structured query builder (`wh.from(table)…`). +Positional `?` substitution is unsupported. The SDK can't forward ClickHouse named params (`WHERE id = {id:UInt32}` + `param_id=42`): the proxy blocks arbitrary query-string params and `wh.sql()` has no hook for them. Use inline literals or the structured builder (`wh.from(table)…`) to bind user input safely. ::: diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index 040c4044..05059489 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -3,13 +3,11 @@ title: "SDK Reference & CLI" description: "Error codes, AbortController, the full API tree, the codegen CLI, and E2E testing with @wavehouse/sdk." --- -Cross-cutting reference for `@wavehouse/sdk`: cancellation, the error model -behind every [`Result<T>`](/sdk#result-type), the complete API tree at a -glance, and the tooling that ships in the package. +Cross-cutting reference for `@wavehouse/sdk`: cancellation, the error model behind [`Result<T>`](/sdk#result-type), the API tree, and tooling. ## AbortController Support -All async operations accept an `AbortSignal` for cancellation: +All async operations accept an `AbortSignal`: ```ts const controller = new AbortController(); @@ -25,7 +23,7 @@ if (error?.code === 'ABORTED') { ## Error Handling -The SDK **never throws** for anything the server returns — all API errors come back in `Result.error`. It does throw on caller and environment errors: a non-absolute `baseURL` (REST calls reject with a `TypeError`; streams report `SSE_CONNECT_ERROR` to the subscriber's `error` callback — see [Serving under a path prefix](/sdk#serving-under-a-path-prefix)), `.stream()` / `.liveQuery()` in a runtime with no `EventSource` (see [Runtime support](/sdk#runtime-support)), and an `auth` callback that rejects — a token-refresh failure propagates out of the REST call, and surfaces on a stream as `SSE_CONNECT_ERROR`. +The SDK **never throws** for server responses; API errors land in `Result.error`. It throws only on caller/environment errors: non-absolute `baseURL` (REST rejects with `TypeError`, streams report `SSE_CONNECT_ERROR` — [Serving under a path prefix](/sdk#serving-under-a-path-prefix)), `.stream()`/`.liveQuery()` without `EventSource` ([Runtime support](/sdk#runtime-support)), and rejecting `auth` callbacks (refresh failures propagate from REST or surface as `SSE_CONNECT_ERROR` on streams). | Status | Code | Retryable | Description | |--------|------|-----------|-------------| @@ -35,12 +33,12 @@ The SDK **never throws** for anything the server returns — all API errors come | 404 | `HTTP_404` | No | Table or pipe not found | | 500 | `HTTP_500` | Yes | Server error (retried per `maxRetries`) | | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries with `Retry-After`) | -| 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | +| 0 | `NETWORK_ERROR` | Yes | Network failure (exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `AbortSignal` | -| 0 | `SSE_CONNECT_ERROR` | Yes | Stream failed to connect (e.g. a non-absolute `baseURL`) | +| 0 | `SSE_CONNECT_ERROR` | Yes | Stream failed to connect (e.g. non-absolute `baseURL`) | | 0 | `SSE_ERROR` | Yes | Stream connection error | -The two `SSE_*` codes arrive on the subscriber's `error` callback rather than in a `Result.error`, since a stream has no single result to carry them. Their `retryable: true` is advisory: unlike the REST codes above, the SDK never re-dials a stream itself. After the connection is open, drops surface through the `status` callback (`reconnecting` → `live`, or `closed`) while the native `EventSource` re-dials on its own; `SSE_ERROR` is a defensive fallback for a transport left in an unexpected state. A failure *before* the `EventSource` is constructed — a non-absolute `baseURL`, a rejecting `auth` callback — is terminal (`SSE_CONNECT_ERROR`), so fix the cause and start a new stream. +`SSE_*` codes arrive via the subscriber's `error` callback — streams have no single result object. Their `retryable: true` is advisory; the SDK never re-dials. Post-connection drops surface on the `status` callback (`reconnecting` → `live`, or `closed`) while `EventSource` re-dials natively; `SSE_ERROR` is a defensive fallback. Failures *before* `EventSource` construction (non-absolute `baseURL`, rejecting `auth`) are terminal (`SSE_CONNECT_ERROR`). --- @@ -94,7 +92,7 @@ StreamController (NOT thenable) ## Codegen CLI -Generate TypeScript types from a running WaveHouse instance. The package ships a `wavehouse-codegen` bin, so after installing `@wavehouse/sdk` you can run it with `npx`: +Generate TypeScript types from a running instance with `wavehouse-codegen`: ```bash npx wavehouse-codegen --url http://localhost:8080 --out ./src/db.d.ts @@ -103,7 +101,7 @@ npx wavehouse-codegen --url http://localhost:8080 --out ./src/db.d.ts pnpm codegen --url http://localhost:8080 --out ./src/db.d.ts ``` -Codegen reads `/v1/schema`, which is **admin-only**. Against a non-dev server, pass an admin-role token with `--auth <jwt>` or the request is denied with `403`. +Codegen reads the admin-only `/v1/schema`; on non-dev servers pass `--auth <jwt>` to avoid `403`. **Options:** @@ -145,7 +143,7 @@ export interface ClicksRow { ## E2E Testing -The SDK doubles as the E2E integration test harness. Tests in `tests/e2e/sdk/` exercise the full pipeline (ingest → ClickHouse → query) through the SDK, validating both the backend and the client library in one pass. +The SDK is the E2E integration test harness; tests in `tests/e2e/sdk/` validate the full pipeline (ingest → ClickHouse → query). ```bash # Run all E2E tests: the orchestrator boots a ClickHouse testcontainer + @@ -153,6 +151,6 @@ The SDK doubles as the E2E integration test harness. Tests in `tests/e2e/sdk/` e make test-e2e ``` -Test files live in `tests/e2e/sdk/` (each `*.test.ts`): `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress`, plus `helpers` — a stack-free unit test of the harness's own `waitForCondition` poll helper rather than a pipeline test. +Test files in `tests/e2e/sdk/` (each `*.test.ts`): `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress`, plus `helpers` — a stack-free unit test of the harness's `waitForCondition` poll helper, not a pipeline test. -See [Development Guide — E2E Tests via SDK](/development#e2e-tests-via-sdk) for architecture details and workflow tips. +See [Development Guide — E2E Tests via SDK](/development#e2e-tests-via-sdk). diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index 88250196..7f3b431e 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -3,19 +3,15 @@ title: "SDK Streaming & Live Queries" description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in @wavehouse/sdk." --- -Real-time consumption with `@wavehouse/sdk`: SSE event streams from tables, -builders, and pipes, plus live queries that backfill history before going -live. Builders and table refs come from [Queries](/sdk/queries). -Examples import from `@wavehouse/sdk`; using the CDN instead, import from -`https://esm.sh/@wavehouse/sdk` (see [Imports & Runtimes](/sdk#imports--runtimes)). +Real-time consumption with `@wavehouse/sdk`: SSE event streams from tables, builders, and pipes, plus live queries that backfill history before going live. Builders and table refs come from [Queries](/sdk/queries). Import from `@wavehouse/sdk` or the CDN `https://esm.sh/@wavehouse/sdk` ([Imports & Runtimes](/sdk#imports--runtimes)). ## Streaming -Streams use SSE (Server-Sent Events) for both unauthenticated connections and for authenticated ones. +Streams use SSE for both authenticated and unauthenticated connections. ### `StreamController` -Returned by `.stream()` on `TableRef`, `QueryBuilder`, `PipeRef`, and `DLQNamespace` (the DLQ variant is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). It is **NOT thenable**. +Returned by `.stream()` on `TableRef`, `QueryBuilder`, `PipeRef`, and `DLQNamespace` (DLQ is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). It is **NOT thenable**. ```ts const stream = wh.from('clicks').stream({ since: '2026-01-01T00:00:00Z' }); @@ -57,7 +53,7 @@ for await (const event of stream) { ### `.close()` -Explicitly close the stream and release all resources. +Close the stream and release all resources. ```ts stream.close(); @@ -84,25 +80,13 @@ interface StreamEvent<T> { } ``` -Row values of top-level `DateTime`/`DateTime64` columns inside `data` (not -timestamps nested in `Array`/`Map`/`Tuple` columns) arrive in canonical RFC 3339 -UTC (`2026-06-21T04:00:00.123Z`), matching what `/v1/query` returns for the -same row — `new Date(value)` parses correctly with no zone fix-up. -Values WaveHouse couldn't canonicalize (ingest is fail-open) stream in the -producer's original spelling, and the `/v1/query` match doesn't hold for them: -a spelling ClickHouse accepted anyway still queries back in canonical UTC (one -it rejected never lands in the table at all), and a zone-less date-time is -what `new Date()` reads as *local* time — though a date-only `YYYY-MM-DD` -string is read as UTC, an ECMAScript quirk -(see [Timestamp canonicalization](/api#timestamp-canonicalization)). +Top-level `DateTime`/`DateTime64` columns in `data` (not timestamps nested in `Array`/`Map`/`Tuple`) arrive in canonical RFC 3339 UTC (`2026-06-21T04:00:00.123Z`), matching `/v1/query`. Non-canonicalized values keep the producer's spelling; `new Date()` reads zone-less date-times as local, `YYYY-MM-DD` as UTC ([Timestamp canonicalization](/api#timestamp-canonicalization)). ### Transport Behavior | Transport | Reconnect | Protocol | | --------- | --------- | -------- | | SSE | Automatic (native `EventSource` with `Last-Event-ID`) | HTTP/2 recommended | -<!-- | TBD | Automatic (retries?) | HTTP/2 recommended | --> -<!-- TODO: Fill in above ^ for SSE fallback, likely polling? --> :::note[SSE connection limit] The SDK warns when more than 5 concurrent SSE connections are open (browser limit per domain). @@ -110,7 +94,7 @@ The SDK warns when more than 5 concurrent SSE connections are open (browser limi ### Client-Side Stream Filtering -When a `QueryBuilder` with `.where()` filters or `.select()` columns calls `.stream()`, the returned stream applies those filters client-side: +`QueryBuilder` `.stream()` applies `.where()`/`.select()` filters client-side: ```ts const stream = wh.from('clicks') @@ -121,13 +105,13 @@ const stream = wh.from('clicks') // Only events where page === '/home' are emitted, with only page + button columns ``` -Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, `not_like` — the same `FilterOp` set `.where()` takes everywhere (the SDK maps them to wire tokens such as `eq`/`neq` internally). +Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, `not_like` — the `FilterOp` set `.where()` takes everywhere (mapped to wire tokens `eq`/`neq`). --- ## Live Queries -Live queries combine a historical backfill (`.fetch()`) with a real-time stream, providing a seamless initial load + live updates experience. +Live queries combine a historical backfill (`.fetch()`) with a live stream: initial load plus live updates. ```ts const lq = wh.from('clicks') @@ -164,9 +148,7 @@ interface StreamSubscriber<T> { ### How it works -1. Opens the stream **immediately** and buffers incoming events. -2. Runs the `.fetch()` query for historical data, calls `subscriber.initial()` with the result. -3. Deduplicates buffered events by comparing timestamps against the latest historical timestamp. +1. Opens the stream immediately and buffers incoming events. +2. Runs `.fetch()` for historical data, calling `subscriber.initial()`. +3. Deduplicates buffered events against the latest historical timestamp. 4. Flushes remaining buffered events and switches to live mode. - -This "stream-first" approach ensures no events are lost between the fetch and stream start. diff --git a/docs/src/content/docs/why-wavehouse.md b/docs/src/content/docs/why-wavehouse.md index 9cb5b63d..283eb6d9 100644 --- a/docs/src/content/docs/why-wavehouse.md +++ b/docs/src/content/docs/why-wavehouse.md @@ -5,17 +5,15 @@ sidebar: order: 3 --- -WaveHouse is an API gateway purpose-built for fronting ClickHouse with user-facing traffic. This page is the engineering answer to "why not just point my clients at ClickHouse?" — or "why not Kafka + ClickHouse?" or "why not Tinybird?" — with the failure modes, the common DIY stacks, and where the cost falls. +WaveHouse is an API gateway for fronting ClickHouse with user-facing traffic. This page answers "why not point clients straight at ClickHouse?", "why not Kafka + ClickHouse?", and "why not Tinybird?" — failure modes, DIY stacks, and where the cost falls. ## Part I — Why ClickHouse alone breaks under user-facing writes ### The one-row-insert anti-pattern -ClickHouse is an OLAP database. Every `INSERT` writes a new *part* on disk; a background merger then consolidates parts over time. This design is phenomenal for bulk analytics ingestion (1M rows in a single block) and catastrophic for streaming ingest from many clients. +ClickHouse is an OLAP database: every `INSERT` creates a disk *part*, consolidated later by background mergers. Fine for bulk analytics, bad for streaming ingest from many clients. -ClickHouse's own documentation is unambiguous: **insert batches of 1,000–100,000 rows at a time, and no more often than once per second.** ([ClickHouse — Inserting Data Best Practices](https://clickhouse.com/docs/en/guides/inserting-data).) A frontend that POSTs one event at a time violates both rules by orders of magnitude. - -What goes wrong in concrete terms: +The docs mandate **batches of 1,000–100,000 rows, at most once per second** ([ClickHouse — Inserting Data Best Practices](https://clickhouse.com/docs/en/guides/inserting-data)). Single-event frontend POSTs violate that by orders of magnitude. ```mermaid flowchart TB @@ -38,28 +36,28 @@ flowchart TB end ``` -The `parts_to_delay_insert` and `parts_to_throw_insert` thresholds are documented MergeTree settings (historical defaults around 1,000 and 3,000 respectively; current values depend on version and are tunable). The failure mode is not hypothetical — it shows up in production any time insert rate outpaces merges, and the error text is literally `DB::Exception: Too many parts (N). Merges are processing significantly slower than inserts`. +MergeTree settings `parts_to_delay_insert` and `parts_to_throw_insert` (historically ~1,000 and 3,000) trigger failures when insert rates outpace merges, raising `DB::Exception: Too many parts (N). Merges are processing significantly slower than inserts`. -**The scenario in dollars.** A product shipping 10M events/day to ClickHouse directly — say, a user analytics widget on 20k daily actives — produces ~115 events/sec. With default merge settings, an unlucky hour of traffic spike pushes you over the delay threshold; inserts stall; the frontend starts 503'ing; engineers get paged; someone spends the weekend writing an ad-hoc batching service. The same 10M events/day via WaveHouse: ~17k bulk inserts/day instead of 10M, merge pressure flat, no incidents. +**The scenario in dollars.** 10M events/day (~115 events/sec) inserted directly: a spike stalls inserts, the frontend 503s, someone writes an emergency batching service. Through WaveHouse it's ~17k bulk inserts/day — merge pressure flat, no incidents. ### No backpressure, no DLQ, no validation at the edge -Even if you remember to batch client-side, a naive ingest path has no safe way to tell a client "slow down" or "that payload was malformed": +Naive ingest paths can't tell a client to slow down or reject a malformed payload: -- **Validation happens late.** ClickHouse will accept an insert with a `String` where you expected a `UInt32` — it just rejects the whole block at parse time, and only after the network round-trip. There is no "this field is unknown" signal at the HTTP boundary; you build that yourself. -- **No backpressure channel.** If the merger falls behind, ClickHouse raises an error at the *next* insert. The client has already left. -- **No DLQ.** Bad events that fail to insert are either lost or logged into ClickHouse's error log. Good luck replaying yesterday's dropped rows. +- **Late Validation:** ClickHouse accepts a `String` where you expected a `UInt32`, then rejects the whole block at parse time, after the network round-trip; no "unknown field" signal at the HTTP boundary. +- **No Backpressure:** if mergers lag, ClickHouse errors on the *next* insert — the client already left. +- **No DLQ:** failed events are lost or buried in error logs; replay is painful. -WaveHouse fixes all three at the gateway: validates every payload against the real `system.columns` schema before accepting, returns `503 Service Unavailable` with a `Retry-After` header when the NATS WAL fills, and routes failed batch inserts to a dedicated `WAVEHOUSE_DLQ` stream you can inspect via `GET /v1/dlq/stats`. +WaveHouse fixes all three at the gateway: validates against `system.columns`, returns `503 Service Unavailable` + `Retry-After` when the NATS WAL fills, and routes failed batches to a `WAVEHOUSE_DLQ` stream (`GET /v1/dlq/stats`). ### No real-time push -ClickHouse has no pub/sub. If a dashboard needs to see new events within a second of ingest, your choices are: +ClickHouse has no pub/sub. For sub-second dashboard updates: -- **Poll.** Every client issues `SELECT ... WHERE received_timestamp > ?` every N seconds. For a dashboard with 100 concurrent viewers polling every 2s, that's **3,000 queries/min against ClickHouse** — most returning zero rows. -- **Add another system** (Kafka, Pulsar, Redis pub/sub) and duplicate the event stream. +- **Poll:** every client issues `SELECT ... WHERE received_timestamp > ?` every N seconds; 100 viewers at 2s means **3,000 queries/min**, mostly zero rows. +- **Duplicate Streams:** Add Kafka, Pulsar, or Redis. -WaveHouse's SSE layer broadcasts events **before** they flush to ClickHouse, with NATS JetStream history for gap-fill when a client reconnects. Same binary, no separate pub/sub stack. +WaveHouse's SSE layer broadcasts **before** the ClickHouse flush, with NATS JetStream history for gap-fill on reconnect. ```mermaid flowchart TB @@ -82,9 +80,7 @@ flowchart TB ### Thundering-herd queries -A popular dashboard recomputes the same expensive aggregation for every viewer. ClickHouse has a built-in query cache, but it is **per-server** and there is no client-facing singleflight: 50 dashboards hitting refresh at once means up to 50 identical queries land on ClickHouse. - -WaveHouse coalesces identical queries with an in-process Ristretto cache and Go's `singleflight`, so only the first of N concurrent identical queries actually hits ClickHouse — the rest receive the same result without making the round trip. +ClickHouse's query cache is per-server with no client-facing singleflight: 50 simultaneous refreshes are 50 identical queries. WaveHouse coalesces them with an in-process Ristretto cache and Go `singleflight` — one query reaches ClickHouse. ```mermaid flowchart TB @@ -107,11 +103,11 @@ flowchart TB ### No row/column access control -ClickHouse has users and role grants, but nothing like row-level security driven by a JWT claim. If your product serves multiple tenants from a shared table, you're writing middleware to inject `WHERE tenant_id = ?` on every query — and hoping you never miss one. WaveHouse ships Hasura-style policies stored in NATS KV: per-role `allow_columns`, row-level `filter` with JWT claim templating (`{{ jwt.app_metadata.tenant_id }}`), bootstrapped from a YAML file, synced cluster-wide via KV Watch. +ClickHouse has no JWT-driven row-level security; multi-tenant apps hand-inject `WHERE tenant_id = ?` everywhere. WaveHouse keeps Hasura-style policies in NATS KV: per-role `allow_columns` and row-level `filter` with JWT claim templating (`{{ jwt.app_metadata.tenant_id }}`), bootstrapped from YAML, synced cluster-wide. ## Part II — What people actually build instead -The canonical DIY stack for user-facing analytics on ClickHouse looks like this: +The canonical DIY stack: <div class="diagram-pair"> @@ -143,41 +139,41 @@ flowchart TB </div> -**Ingredient count, cleanly:** +**Ingredient count:** | Capability | DIY stack | WaveHouse | | ---------- | --------- | --------- | | Durable ingest buffer | Kafka / Redpanda cluster (3+ brokers, Zookeeper/KRaft) | Embedded NATS JetStream | -| Batch consumer | Custom Go/Rust/Java service you write and operate | Built in | -| Query cache | Redis + singleflight middleware you write | Built in (Ristretto + singleflight) | -| Real-time push | WebSocket service + bridge from Kafka | Built in (`/v1/stream`) | +| Batch consumer | Custom Go/Rust/Java service | Built in | +| Query cache | Redis + singleflight middleware | Built in (Ristretto + singleflight) | +| Real-time push | WebSocket service + Kafka bridge | Built in (`/v1/stream`) | | Schema validation | Custom code in ingest API | Built in (discovers `system.columns`) | -| Row/column access control | Custom middleware or a dedicated service | Built in (Hasura-style, JWT-driven) | +| Row/column access control | Custom middleware or service | Built in (Hasura-style, JWT-driven) | | Dead letter queue | Custom retry + dead topic on Kafka | Built in (`WAVEHOUSE_DLQ`) | | Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript, zero-dep, codegen) | -The DIY path works — big teams run it — but the ops cost is not small. You're paying for a Kafka cluster (or Confluent bill), a second service you wrote from scratch, and all the debugging hours when the batching consumer stalls at 3 a.m. +The DIY path works for big teams; the ops cost is Kafka bills and 3 a.m. batch-consumer stalls. -**The scenario.** A seed-stage team building user-facing analytics picks "Kafka + ClickHouse + custom ingest". Six months in: two engineers are spending ~30% of their time on data-plane reliability (batching edge cases, DLQ replay tooling, Kafka upgrades, monitoring dashboards for all of it). That's roughly one full-time engineer of drag on a 3-person backend team. A drop-in gateway removes that line item. +**Scenario:** a seed-stage team picks "Kafka + ClickHouse + custom ingest". Six months on, two engineers spend ~30% of their time on data-plane reliability (batching edge cases, DLQ replay, Kafka upgrades) — one full-time engineer of drag on a 3-person backend team. A drop-in gateway removes it. ### Tinybird -Tinybird is the most direct commercial alternative — a hosted ClickHouse platform with SQL-based "pipes" for defining APIs. Genuinely good product for teams that want to pay to skip the plumbing. +Tinybird is the main commercial alternative: hosted ClickHouse with SQL "pipes" as APIs, for teams paying to skip the plumbing. -Where it differs from WaveHouse: +Differences: | Dimension | Tinybird | WaveHouse | | --------- | -------- | --------- | -| Hosting | SaaS only (managed tiers: Developer $49/mo → Enterprise custom) | Self-host, single binary | -| Pricing model | Pay for allocated vCPU/QPS/storage; egress fees for cross-region | Your infra; no per-query or per-GB fee | +| Hosting | SaaS only (Developer $49/mo → Enterprise) | Self-host, single binary | +| Pricing model | vCPU/QPS/storage; egress fees | Your infra; no per-query/GB fee | | Data residency | Their infrastructure | Your infrastructure | -| Source of truth for schema | Tinybird datasource definitions | Your ClickHouse tables (`system.columns`) | -| Deployment workflow | Tinybird CLI against Tinybird Cloud | `docker compose up` or any K8s | +| Schema source of truth | Tinybird datasource definitions | ClickHouse tables (`system.columns`) | +| Deployment workflow | Tinybird CLI against Cloud | `docker compose up` or K8s | | Real-time push | Pipe endpoints (request/response) | Native SSE | | Access control | Tinybird tokens (API-level) | JWT + Hasura-style row/column policies | -| Vendor lock-in | Queries run on Tinybird; moving off = rewriting | None — WaveHouse is Apache 2.0, ClickHouse is yours | +| Vendor lock-in | Rewriting queries | None — WaveHouse is Apache 2.0, ClickHouse is yours | -Tinybird wins on "zero ops to start." WaveHouse wins on "own your data plane and pay AWS, not a second vendor" — which gets more compelling at scale, on sensitive data, or for anyone who needs on-prem. +Tinybird wins on "zero ops to start". WaveHouse wins on owning the data plane and paying AWS instead of a second vendor — for scale, sensitive data, or on-prem. ## Part III — The feature matrix @@ -199,7 +195,7 @@ Tinybird wins on "zero ops to start." WaveHouse wins on "own your data plane and ## Part IV — End-to-end data journey -How an event actually moves through WaveHouse, end to end. The ingest path is split into a synchronous edge (everything before the `200 OK`) and two async tails — real-time broadcast and batched insert. +Events move through a synchronous edge (pre-`200 OK`) and two async tails: real-time broadcast and batched insert. **Ingest & broadcast path:** @@ -232,7 +228,7 @@ flowchart TB CH -. "miss: query latency" .-> Q ``` -**Latency budget at each stage** (representative): +**Latency budget at each stage:** | Stage | Typical p50 | Typical p99 | | ----- | ----------- | ----------- | @@ -248,17 +244,17 @@ flowchart TB The honest list: -- **Internal BI / data team workloads** — if the only clients of ClickHouse are analysts in a BI tool and ETL jobs that already batch, WaveHouse adds latency and an extra component for no gain. Point BI straight at ClickHouse. -- **Pure bulk ETL pipelines** — if your writes already arrive as 100k-row blocks from Airflow or dbt, the buffering layer is redundant. -- **ClickHouse-as-a-datalake** — using ClickHouse for cold analytics over S3/Iceberg. WaveHouse is about the hot path. -- **Kafka-shaped organizations** — if you already run a heavily-invested Kafka Connect ecosystem with custom sinks, the migration cost may exceed the benefit. Run Kafka → WaveHouse in front of ClickHouse if you want the real-time and cache layers without throwing out Kafka. +- **Internal BI / data team workloads** — if the clients are BI tools and batch ETL, point them straight at ClickHouse. +- **Pure bulk ETL** — redundant when writes already arrive as 100k-row blocks from Airflow or dbt. +- **ClickHouse-as-a-datalake** — WaveHouse targets the hot path, not cold analytics over S3/Iceberg. +- **Kafka-shaped organizations** — deep in Kafka Connect custom sinks, migration may cost too much; run Kafka → WaveHouse → ClickHouse for the real-time layer. ## Summary -The pitch in one sentence: **ClickHouse is a great database and a poor API**. Every product putting user-facing traffic on ClickHouse eventually builds WaveHouse — the question is whether you build it on the weekend before the merge backlog, after it, or skip the build by deploying ours. +**ClickHouse is a great database but a poor API**. Products serving user traffic from ClickHouse eventually build WaveHouse — deploy ours instead. Read next: -- **[Architecture](/architecture)** — the internal package map and how each of these capabilities is implemented. +- **[Architecture](/architecture)** — package map and implementation details. - **[Getting Started](/getting-started)** — five minutes to `200 OK`. -- **[API Reference](/api)** — every endpoint and the schema-validation contract. +- **[API Reference](/api)** — endpoints and schema-validation contracts.