From 834a0935d141ce2e2f7b7db1ab7f9440d80c0e0e Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 15:13:24 -0400 Subject: [PATCH 01/40] feat(sdk): add Go client SDK with full API-tree parity Complete Go SDK at clients/go/ with zero third-party runtime dependencies. Covers ingest, query (structured + SQL), streaming (SSE with reconnect), live queries, pipes, DLQ, schema, policy, and health endpoints. - Immutable QueryBuilder with generics (FetchTyped[Row], SQL[Row]) - SSE StreamController with reconnect/backoff and client-side filtering - LiveQuery drain-then-switch backfill (stream-first, dedup, go live) - Reflect-based Insert handling any []T, not just []map[string]any - wavehouse-codegen CLI for generating typed row structs from /v1/schema - 42 unit tests + 44 cross-language wire-format conformance cases - E2E test scaffolding (build tag e2e, 9 tests against live server) - Cross-language conformance runner for TS SDK (tests/conformance/) - Makefile targets: verify-go-sdk, test-go-sdk, test-go-sdk-e2e, lint-go-sdk --- CHANGELOG.md | 1 + Makefile | 45 +- clients/go/README.md | 235 ++++++++++ clients/go/client_test.go | 100 ++++ clients/go/cmd/wavehouse-codegen/main.go | 338 ++++++++++++++ clients/go/conformance_test.go | 377 +++++++++++++++ clients/go/dlq.go | 42 ++ clients/go/e2e_test.go | 436 ++++++++++++++++++ clients/go/errors.go | 81 ++++ clients/go/errors_test.go | 145 ++++++ clients/go/example_test.go | 71 +++ clients/go/go.mod | 3 + clients/go/http.go | 213 +++++++++ clients/go/http_test.go | 218 +++++++++ clients/go/live_query.go | 158 +++++++ clients/go/namespaces_test.go | 193 ++++++++ clients/go/pipes.go | 99 ++++ clients/go/policy.go | 42 ++ clients/go/query_builder.go | 310 +++++++++++++ clients/go/query_builder_test.go | 259 +++++++++++ clients/go/schema.go | 33 ++ clients/go/stream.go | 561 +++++++++++++++++++++++ clients/go/sys.go | 17 + clients/go/table.go | 205 +++++++++ clients/go/table_test.go | 209 +++++++++ clients/go/testdata/wire_cases.json | 547 ++++++++++++++++++++++ clients/go/types.go | 245 ++++++++++ clients/go/wavehouse.go | 142 ++++++ tests/conformance/conformance_ts.mjs | 279 +++++++++++ 29 files changed, 5601 insertions(+), 3 deletions(-) create mode 100644 clients/go/README.md create mode 100644 clients/go/client_test.go create mode 100644 clients/go/cmd/wavehouse-codegen/main.go create mode 100644 clients/go/conformance_test.go create mode 100644 clients/go/dlq.go create mode 100644 clients/go/e2e_test.go create mode 100644 clients/go/errors.go create mode 100644 clients/go/errors_test.go create mode 100644 clients/go/example_test.go create mode 100644 clients/go/go.mod create mode 100644 clients/go/http.go create mode 100644 clients/go/http_test.go create mode 100644 clients/go/live_query.go create mode 100644 clients/go/namespaces_test.go create mode 100644 clients/go/pipes.go create mode 100644 clients/go/policy.go create mode 100644 clients/go/query_builder.go create mode 100644 clients/go/query_builder_test.go create mode 100644 clients/go/schema.go create mode 100644 clients/go/stream.go create mode 100644 clients/go/sys.go create mode 100644 clients/go/table.go create mode 100644 clients/go/table_test.go create mode 100644 clients/go/testdata/wire_cases.json create mode 100644 clients/go/types.go create mode 100644 clients/go/wavehouse.go create mode 100644 tests/conformance/conformance_ts.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 3db0d718..e0ba485d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Go SDK (`clients/go/`) — official Go client with full API-tree parity against the TypeScript SDK, zero third-party runtime dependencies, cross-language wire-format conformance tests - **Non-JWT operator key for full-access admin + break-glass recovery** (`internal/auth/auth.go`, `internal/auth/context.go`, `internal/auth/auth_test.go`, `internal/api/router.go`, `internal/api/router_test.go`, `internal/config/config.go`, `internal/config/config_test.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `tests/e2e/fixtures/config.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/access-control.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/development.md`, `docs/src/content/docs/architecture.md`, `docs/src/content/docs/reverse-proxy.mdx`, `AGENTS.md`, `SECURITY.md`): closes #240; partially advances the auth-hardening epic [#228](https://github.com/Wave-RF/WaveHouse/issues/228) and the management/data-plane split [#359](https://github.com/Wave-RF/WaveHouse/issues/359). Adds an optional `auth.operator_key` (`WH_AUTH_OPERATOR_KEY`): a request presenting it — via an `Authorization: Operator ` header (forwarded verbatim by proxies, no collision with Bearer JWTs) or the `X-Operator-Key` alias — is authorized as a full-access platform operator — the whole data plane *and* the `/v1/admin/*` management surface — without minting a JWT and independently of the token verifier, giving the person running the deployment a role-free credential for bootstrap and break-glass. The middleware checks it before the Bearer token with a constant-time comparison (`crypto/subtle`) and stamps two things into the request context: the live `admin_role` (so the policy evaluator's admin bypass grants unrestricted data-plane access while a policy exists) and a platform-operator bit that `RequireAdmin` honors **even when the policy is `nil`/deleted** — the one HTTP path that can restore a wiped policy, which previously required SSH access and a reboot. Empty (the default) disables it, so existing deployments are unchanged; treat it as an admin secret (load from a secret store, serve only over TLS). A successful operator authentication is audit-logged at `INFO`; a request presenting a *non-matching* operator key is logged at `WARN` and counted by a new `wavehouse_auth_operator_key_failures_total` counter (a probing/brute-force signal on the most privileged credential in the system) before falling through to the normal token/default path — the middleware still never rejects. The `Authorization` auth-scheme is matched case-insensitively (RFC 7235) via a shared `authScheme` helper, which also makes the existing `Bearer` JWT scheme case-insensitive (previously it required the canonical `Bearer` casing). Explicitly out of scope, tracked in #359: scoping the operator credential to the management surface only, and capability-scoped admin permissions. - **Missing-dedupe-id observability + optional strict mode** (`internal/api/ingest.go`, `internal/api/ingest_test.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/architecture.md`): closes #219. With dedupe enabled, a row missing the configured `id_field` can't be deduped — previously it was published with idempotency silently disabled and *no* log or metric, so a producer bug that dropped the id turned off the guarantee for those rows unnoticed. Now every such row is logged at `WARN` and counted by a new `wavehouse_ingest_dedupe_missing_id_total` counter (labeled by `table`), making the loss observable server-side. A new opt-in `dedupe.require_id` (`WH_DEDUPE_REQUIRE_ID`, default `false`) turns that signal into enforcement: a row missing the id is rejected (`400` for a single insert; a per-record failure in a batch) instead of published — a tripwire for producers that must guarantee the id (complements the client-side [#202](https://github.com/Wave-RF/WaveHouse/issues/202)). Default behavior is unchanged. - **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes. diff --git a/Makefile b/Makefile index de1ebea2..3174e5c6 100644 --- a/Makefile +++ b/Makefile @@ -362,6 +362,10 @@ lint: lint-go lint-ts lint-md lint-prose ## Lint across Go (golangci-lint) + TS/ lint-go: $(GOLANGCI_LINT) go-mod-download $(call run,golangci-lint,$(GOLANGCI_LINT) run ./... --allow-parallel-runners,run make fix to auto-fix what is fixable) +.PHONY: lint-go-sdk +lint-go-sdk: $(GOLANGCI_LINT) + $(call run,golangci-lint (Go SDK),cd clients/go && $(GOLANGCI_LINT) run ./...,) + .PHONY: lint-ts lint-ts: pnpm-install $(call run,Biome (lint + format + imports),$(PNPM) -s -w run check,run make fix to auto-fix what is fixable) @@ -424,6 +428,18 @@ endif tidy: ## Verify go.mod/go.sum are tidy (run `make fix` to apply) $(call run,go.mod tidy,go mod tidy -diff,run make fix to tidy go.mod and go.sum) +# verify-go-sdk: static checks for clients/go/ — a nested Go module (its own +# go.mod), so it's invisible to `go list ./...` from the root and every leaf +# above (GO_DIRS, lint-go, vulncheck, tidy) silently skips it. Scoped and run +# explicitly here instead. `go vet` needs its own module context (cd +# clients/go), but gofumpt is a pure syntax formatter with no module +# resolution of its own, so the repo-pinned $(GOFUMPT) binary can format it +# directly by path from the root — no second tool pin needed. +.PHONY: verify-go-sdk +verify-go-sdk: ## Static checks for the Go SDK (clients/go, a nested module) — go vet + gofumpt + $(call run,go vet (Go SDK),cd clients/go && go vet ./...,) + $(call run,gofumpt (Go SDK),$(GOFUMPT) -l clients/go | (! grep .),run make fix to apply formatting) + # fix: apply auto-fixes everywhere, fanned out into three tracks that touch # disjoint files — Go (.go + go.mod/sum), TS/JS/JSON (Biome), Markdown — so they # run in parallel safely. The Go track is itself a serial chain (tidy → gofumpt → @@ -467,7 +483,8 @@ fix-prose: $(MISSPELL) # slowest tool, not the slowest *group* (e.g. golangci no longer drags Biome + # markdownlint along behind it). # -# Leaves (9): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck on the Go +# Leaves (10): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck, +# verify-go-sdk (go vet + gofumpt on the nested clients/go module) on the Go # side; lint-ts (biome check) + lint-md (markdownlint) + lint-prose (misspell, # docs spelling) for JS/TS + Markdown + prose; # check-docs (astro check — the only leaf that writes, to docs/.astro/, and @@ -483,7 +500,7 @@ verify: ## Run all static checks across the repo (Go + TS + docs, parallelized) @printf "$(GREEN)$(BOLD)✔ All static checks passed$(RESET)\n" .PHONY: verify-parallel -verify-parallel: tidy fmt-go lint-go lint-ts lint-md lint-prose lint-sh lint-gha test-classify-paths vulncheck check-docs typecheck-ts +verify-parallel: tidy fmt-go lint-go lint-go-sdk lint-ts lint-md lint-prose lint-sh lint-gha test-classify-paths vulncheck check-docs typecheck-ts verify-go-sdk # typecheck-ts: tsc --noEmit on the SDK. Its own target (was inline in verify's # recipe) so it can run as a parallel leaf of verify-parallel. @@ -713,6 +730,28 @@ test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against sui $(if $(COV_DEFER),,--coverage.thresholds.statements=$$(go run ./scripts/cov threshold ts-unit)) $(ARGS) @if [ -z "$(COV_DEFER)" ]; then printf "$(GREEN)==> ts-unit gate passed$(RESET) HTML: tmp/coverage/ts-unit/index.html\n"; fi +# test-go-sdk: unit tests for clients/go/ — a nested Go module (its own +# go.mod), so it's outside test-unit's ./internal/... ./cmd/... scope and +# needs its own leaf; a root `go test ./...` wouldn't reach it either. Zero +# third-party runtime or test deps (stdlib only, no go.sum), so no +# go-mod-download prereq. Not yet wired into the Go/TS coverage gate — see +# verify-go-sdk above for the same "nested module, own leaf" reasoning. +.PHONY: test-go-sdk +test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests + @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" + @cd clients/go && go test ./... + +# test-go-sdk-e2e: runs the Go SDK's E2E tests against a live WaveHouse +# instance. Requires a running server (e.g. `make dev` in the main repo). +# Env vars: +# WAVEHOUSE_URL base URL of the server (default: http://localhost:8080) +# WAVEHOUSE_AUTH bearer token for auth (optional; omit for default_role) +# The tests skip gracefully when the server is unreachable. +.PHONY: test-go-sdk-e2e +test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVEHOUSE_URL, WAVEHOUSE_AUTH) + @printf "$(CYAN)==> Running Go SDK E2E tests...$(RESET)\n" + @cd clients/go && go test -tags e2e -v -count=1 -timeout 60s ./... + # Aggregator: recipe-based with $(MAKE) calls so suites run sequentially even # under `make -j N`. The suites bind ports / spin testcontainers / start the # release binary, so concurrent execution is unsafe. @@ -749,7 +788,7 @@ cov: ## Consolidated coverage report (Go + TS) + gate against thresholds (auto-r # marker that standalone `make verify` writes is instead written by ci's own # `ci-marker.sh write` below — it touches both the ci and verify markers. .PHONY: ci-parallel -ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts +ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts test-go-sdk .PHONY: ci ci: ## Full pipeline — parallel checks, then sequential heavy suites + coverage diff --git a/clients/go/README.md b/clients/go/README.md new file mode 100644 index 00000000..e3af7bd4 --- /dev/null +++ b/clients/go/README.md @@ -0,0 +1,235 @@ +# WaveHouse Go SDK + +Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a schema-aware real-time API gateway for ClickHouse. + +**Zero third-party runtime dependencies** — stdlib only. + +**[Full SDK documentation on wavehouse.dev](https://wavehouse.dev/sdk/go/)** + +## Install + +```bash +go get github.com/Wave-RF/WaveHouse/clients/go +``` + +## Quick Start + +```go +package main + +import ( + "context" + "fmt" + "log" + + wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +) + +func main() { + // Create an unauthenticated client (uses the server's default_role). + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + }) + + // Health check. + if err := client.Sys.Health(context.Background()); err != nil { + log.Fatal(err) + } + + ctx := context.Background() + + // Insert a row. + _, err := client.From("clicks").Insert(ctx, map[string]any{ + "page": "/home", "button": "cta", + }) + if err != nil { + log.Fatal(err) + } + + // Query with the fluent builder. + page, err := client.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + OrderBy("page", "asc"). + Limit(10). + FetchUntyped(ctx) + if err != nil { + log.Fatal(err) + } + for _, row := range page.Data { + fmt.Println(row["page"], row["button"]) + } +} +``` + +## Authentication + +```go +// Static token. +client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), +}) + +// Dynamic token (e.g. rotated). +client = wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: func(ctx context.Context) (string, error) { + return fetchFreshToken(ctx) + }, +}) +``` + +## Typed Queries (Generics) + +```go +type ClickRow struct { + Page string `json:"page"` + Button string `json:"button"` + DurationMS int `json:"duration_ms"` +} + +page, err := wavehouse.FetchTyped[ClickRow](ctx, + client.From("clicks").Select("page", "button", "duration_ms").Limit(100), +) +// page.Data is []ClickRow +``` + +## Batch Insert (NDJSON) + +```go +// Array of maps — serialized to NDJSON automatically. +result, _ := client.From("clicks").Insert(ctx, []map[string]any{ + {"page": "/a", "button": "cta"}, + {"page": "/b", "button": "nav"}, +}) +// result.OK, result.Total, result.Succeeded, result.Failed + +// Pre-formatted NDJSON string. +result, _ = client.From("clicks").InsertNDJSON(ctx, + `{"page":"/a"}`+"\n"+`{"page":"/b"}`, +) +``` + +## Streaming (SSE) + +```go +stream := client.From("clicks").Stream(&wavehouse.StreamOptions{ + Since: "2026-01-01T00:00:00Z", +}) +defer stream.Close() + +// Channel-based consumption. +for event := range stream.Events() { + fmt.Println(event.Table, event.Data) +} + +// Or callback-based. +unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ + Next: func(e wavehouse.StreamEvent) { fmt.Println(e.Data) }, + Status: func(s wavehouse.StreamStatus) { fmt.Println("status:", s) }, +}) +defer unsub() +``` + +## Live Queries + +```go +lq := client.From("clicks"). + SelectAll(). + OrderBy("received_timestamp", "desc"). + Limit(100). + LiveQuery(&wavehouse.StreamSubscriber{ + Initial: func(rows []map[string]any, err error) { + // Historical backfill. + fmt.Println("initial rows:", len(rows)) + }, + Next: func(e wavehouse.StreamEvent) { + // Live events after backfill. + fmt.Println("live:", e.Data) + }, + }, nil) +defer lq.Close() +``` + +## Named Pipes + +```go +// Execute a pipe. +rows, _ := wavehouse.Fetch[map[string]any](ctx, + client.Pipe("top_pages", map[string]any{"limit": 10}), +) + +// Admin: manage pipes. +client.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ + SQL: "SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}", + AllowedRoles: []string{"viewer", "admin"}, +}) +pipes, _ := client.Pipes.List(ctx) +client.Pipes.Delete(ctx, "old_pipe") +``` + +## Admin + +```go +// Schema introspection (admin-only). +schemas, _ := client.Schema.List(ctx) +client.Schema.Refresh(ctx) + +// Policy management (admin-only). +policy, _ := client.Policy.Get(ctx) +client.Policy.Set(ctx, policy) +result, _ := client.Policy.Validate(ctx, policy) + +// DLQ stats (admin-only). +stats, _ := client.DLQ.List(ctx) + +// Raw SQL (admin-only). +rows, _ := wavehouse.SQL[map[string]any](ctx, client, "SELECT count() FROM clicks") +``` + +## Codegen + +Generate Go structs from a running WaveHouse instance: + +```bash +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ + --url http://localhost:8080 \ + --auth \ + --out ./db_types.go \ + --package myapp +``` + +The CLI reads `/v1/schema` (admin-only) and maps ClickHouse types to Go types: + +| ClickHouse | Go | +|---|---| +| `String`, `UUID`, `DateTime*`, `Date*`, `Enum*`, `IPv4/6` | `string` | +| `UInt8/16/32/64` | `uint8/16/32/64` | +| `Int8/16/32/64` | `int8/16/32/64` | +| `Float32/64` | `float32/64` | +| `Bool` | `bool` | +| `Nullable(T)` | `*T` | +| `Array(T)` | `[]T` | +| `Map(K,V)` | `map[K]V` | +| `UInt128/256`, `Int128/256`, `Decimal*` | `string` | + +## Error Handling + +All SDK operations return `(T, error)`. Errors are `*wavehouse.Error` (use `errors.As`): + +```go +page, err := client.From("clicks").Fetch(ctx) +if err != nil { + var whErr *wavehouse.Error + if errors.As(err, &whErr) { + fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } +} +``` + +The HTTP layer retries 5xx and network errors with exponential backoff (default 2 retries). 503 with `Retry-After` is honored. Context cancellation returns immediately with code `ABORTED`. + +## License + +Apache-2.0 diff --git a/clients/go/client_test.go b/clients/go/client_test.go new file mode 100644 index 00000000..4f57507f --- /dev/null +++ b/clients/go/client_test.go @@ -0,0 +1,100 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" +) + +func TestNewClient_Defaults(t *testing.T) { + c := NewClient(Config{BaseURL: "http://localhost:8080"}) + if c.ctx.maxRetries != 2 { + t.Fatalf("want default maxRetries=2, got %d", c.ctx.maxRetries) + } + if c.ctx.baseURL != "http://localhost:8080" { + t.Fatalf("want baseURL, got %s", c.ctx.baseURL) + } +} + +func TestNewClient_StripsTrailingSlashes(t *testing.T) { + c := NewClient(Config{BaseURL: "http://localhost:8080///"}) + if c.ctx.baseURL != "http://localhost:8080" { + t.Fatalf("want stripped URL, got %s", c.ctx.baseURL) + } +} + +func TestNewClient_CustomMaxRetries(t *testing.T) { + c := NewClient(Config{ + BaseURL: "http://localhost:8080", + Options: &ClientOptions{MaxRetries: 5}, + }) + if c.ctx.maxRetries != 5 { + t.Fatalf("want 5, got %d", c.ctx.maxRetries) + } +} + +func TestNewClient_HasNamespaces(t *testing.T) { + c := NewClient(Config{BaseURL: "http://localhost:8080"}) + if c.Schema == nil { + t.Fatal("Schema namespace nil") + } + if c.Policy == nil { + t.Fatal("Policy namespace nil") + } + if c.DLQ == nil { + t.Fatal("DLQ namespace nil") + } + if c.Sys == nil { + t.Fatal("Sys namespace nil") + } + if c.Pipes == nil { + t.Fatal("Pipes namespace nil") + } +} + +func TestClient_From(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + // Verify the table name appears in the URL. + if r.URL.Query().Get("table") != "events" { + t.Errorf("want table=events, got %s", r.URL.Query().Get("table")) + } + json.NewEncoder(w).Encode([]map[string]any{}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + _, _ = c.From("events").Fetch(context.Background()) +} + +func TestClient_SQL(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/admin/query" { + t.Errorf("want /v1/admin/query, got %s", r.URL.Path) + } + var body map[string]string + json.NewDecoder(r.Body).Decode(&body) + if body["sql"] != "SELECT 1" { + t.Errorf("want sql=SELECT 1, got %s", body["sql"]) + } + json.NewEncoder(w).Encode([]map[string]any{{"x": 1}}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + rows, err := SQL[map[string]any](context.Background(), c, "SELECT 1") + if err != nil { + t.Fatal(err) + } + if len(rows) != 1 { + t.Fatalf("want 1 row, got %d", len(rows)) + } +} + +func TestStaticToken(t *testing.T) { + fn := StaticToken("abc") + token, err := fn(context.Background()) + if err != nil { + t.Fatal(err) + } + if token != "abc" { + t.Fatalf("want abc, got %s", token) + } +} diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go new file mode 100644 index 00000000..1df138b9 --- /dev/null +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -0,0 +1,338 @@ +// Command wavehouse-codegen reads a WaveHouse server's /v1/schema endpoint +// and generates Go struct definitions for use with the wavehouse SDK. +// +// Usage: +// +// wavehouse-codegen --url http://localhost:8080 --out ./db.go --auth +package main + +import ( + "context" + "encoding/json" + "fmt" + "go/format" + "net/http" + "os" + "slices" + "strings" + "unicode" +) + +type cliArgs struct { + url string + out string + auth string + pkg string +} + +func parseArgs() cliArgs { + args := cliArgs{url: "http://localhost:8080", out: "./wavehouse_types.go", pkg: "main"} + for i := 1; i < len(os.Args); i++ { + switch os.Args[i] { + case "--url", "-u": + i++ + if i < len(os.Args) { + args.url = os.Args[i] + } + case "--out", "-o": + i++ + if i < len(os.Args) { + args.out = os.Args[i] + } + case "--auth", "-a": + i++ + if i < len(os.Args) { + args.auth = os.Args[i] + } + case "--package", "-p": + i++ + if i < len(os.Args) { + args.pkg = os.Args[i] + } + case "--help", "-h": + fmt.Println(`wavehouse-codegen — Generate Go types from WaveHouse schema + +Options: + --url, -u WaveHouse base URL (default: http://localhost:8080) + --out, -o Output .go file path (default: ./wavehouse_types.go) + --auth, -a Bearer token for authenticated /v1/schema endpoint + --package, -p Go package name (default: main) + --help, -h Show this help`) + os.Exit(0) + } + } + return args +} + +type column struct { + Name string `json:"name"` + Type string `json:"type"` + IsNullable bool `json:"is_nullable"` + HasDefault bool `json:"has_default"` +} + +type tableSchema struct { + Name string `json:"name"` + Columns []column `json:"columns"` +} + +func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSchema, error) { + url := strings.TrimRight(baseURL, "/") + "/v1/schema" + req, err := http.NewRequestWithContext(ctx, "GET", url, nil) + if err != nil { + return nil, err + } + if auth != "" { + req.Header.Set("Authorization", "Bearer "+auth) + } + resp, err := http.DefaultClient.Do(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + if resp.StatusCode != 200 { + return nil, fmt.Errorf("schema fetch failed: HTTP %d", resp.StatusCode) + } + + // Server returns either []tableSchema or map[string]tableSchema. + var raw json.RawMessage + if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil { + return nil, err + } + // Try array first. + var arr []tableSchema + if err := json.Unmarshal(raw, &arr); err == nil { + m := make(map[string]tableSchema, len(arr)) + for _, t := range arr { + m[t.Name] = t + } + return m, nil + } + var m map[string]tableSchema + if err := json.Unmarshal(raw, &m); err != nil { + return nil, err + } + return m, nil +} + +// chTypeToGo maps a ClickHouse type string (as reported by /v1/schema) to a +// Go type name suitable for a JSON struct field. +// +// We deliberately don't import clickhouse-go's type catalog +// (github.com/ClickHouse/clickhouse-go/v2/lib/column) for this. It's public +// and does expose a real ClickHouse-type-string parser — +// column.Type(chType).Column(name, sc).ScanType() — but it answers a +// different question than the one we're asking. That catalog maps to the Go +// types the *driver* scans query results into over the native protocol +// (time.Time for Date/DateTime*, uuid.UUID for UUID, decimal.Decimal for +// Decimal, net.IP for IPv4/IPv6, *big.Int for [U]Int128/256), not the types +// that round-trip cleanly through the JSON the /v1/schema and query +// endpoints actually speak. ClickHouse's JSON output renders DateTime as +// "2024-01-15 10:30:00" (no "T", no offset), which fails Go's default +// time.Time JSON unmarshaling; big integers and decimals are similarly +// rendered as JSON strings, not driver-native types. Adopting the driver's +// ScanType() as-is would produce generated structs that don't unmarshal the +// server's actual JSON, and would drag uuid/decimal/orb/net imports into +// generated output that today has zero non-stdlib dependencies. So we keep +// the hand-rolled JSON-oriented mapping below, informed by (but not bound +// to) the type set clickhouse-go's lib/column recognizes. +func chTypeToGo(chType string) string { + // Unwrap Nullable → pointer. + if strings.HasPrefix(chType, "Nullable(") && strings.HasSuffix(chType, ")") { + inner := chType[9 : len(chType)-1] + return "*" + chTypeToGo(inner) + } + // Unwrap LowCardinality. + if strings.HasPrefix(chType, "LowCardinality(") && strings.HasSuffix(chType, ")") { + return chTypeToGo(chType[15 : len(chType)-1]) + } + // Unwrap SimpleAggregateFunction(func, InnerType) — readable columns in + // AggregatingMergeTree/SummingMergeTree rollup tables. The value on the + // wire is just InnerType; the aggregate function name only describes how + // merges combine rows. + if strings.HasPrefix(chType, "SimpleAggregateFunction(") && strings.HasSuffix(chType, ")") { + inner := chType[len("SimpleAggregateFunction(") : len(chType)-1] + if comma := findTopLevelComma(inner); comma != -1 { + return chTypeToGo(strings.TrimSpace(inner[comma+1:])) + } + return "any" + } + // String-like. + switch { + case chType == "String", + strings.HasPrefix(chType, "FixedString("), + chType == "UUID", + strings.HasPrefix(chType, "DateTime"), + strings.HasPrefix(chType, "Date"), + // Time/Time64 are ClickHouse's newer time-of-day types (distinct + // from DateTime); same JSON-string-not-RFC3339 story applies. + strings.HasPrefix(chType, "Time"), + strings.HasPrefix(chType, "Enum8("), + strings.HasPrefix(chType, "Enum16("), + chType == "IPv4", + chType == "IPv6": + return "string" + case chType == "Bool", chType == "Boolean": + return "bool" + } + // Numeric — map widths honestly. + switch { + case chType == "UInt8": + return "uint8" + case chType == "UInt16": + return "uint16" + case chType == "UInt32": + return "uint32" + case chType == "UInt64": + return "uint64" + case chType == "Int8": + return "int8" + case chType == "Int16": + return "int16" + case chType == "Int32": + return "int32" + case chType == "Int64": + return "int64" + case chType == "Float32": + return "float32" + case chType == "Float64": + return "float64" + case chType == "BFloat16": + return "float32" + case strings.HasPrefix(chType, "Decimal"), + strings.HasPrefix(chType, "UInt128"), + strings.HasPrefix(chType, "UInt256"), + strings.HasPrefix(chType, "Int128"), + strings.HasPrefix(chType, "Int256"): + return "string" // big numbers are strings in JSON + } + // Array. + if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { + inner := chType[6 : len(chType)-1] + return "[]" + chTypeToGo(inner) + } + // Map. + if strings.HasPrefix(chType, "Map(") && strings.HasSuffix(chType, ")") { + inner := chType[4 : len(chType)-1] + comma := findTopLevelComma(inner) + if comma != -1 { + k := chTypeToGo(strings.TrimSpace(inner[:comma])) + v := chTypeToGo(strings.TrimSpace(inner[comma+1:])) + return "map[" + k + "]" + v + } + return "map[string]any" + } + return "any" +} + +func findTopLevelComma(s string) int { + depth := 0 + for i := range len(s) { + switch s[i] { + case '(': + depth++ + case ')': + depth-- + case ',': + if depth == 0 { + return i + } + } + } + return -1 +} + +func pascalCase(s string) string { + parts := strings.FieldsFunc(s, func(r rune) bool { + return r == '_' || r == '-' || r == ' ' || r == '.' + }) + var sb strings.Builder + for _, p := range parts { + if len(p) == 0 { + continue + } + runes := []rune(p) + runes[0] = unicode.ToUpper(runes[0]) + sb.WriteString(string(runes)) + } + result := sb.String() + if result == "" { + return result + } + // Go identifiers can't start with a digit (e.g. a table named + // "2fa_events" would otherwise produce the invalid identifier + // "2faEvents"). Prefix with "X" to keep it a valid, exported name. + if unicode.IsDigit([]rune(result)[0]) { + result = "X" + result + } + return result +} + +func generate(schemas map[string]tableSchema, pkg string) string { + var sb strings.Builder + fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) + + names := make([]string, 0, len(schemas)) + for name := range schemas { + names = append(names, name) + } + slices.Sort(names) + + for _, name := range names { + schema := schemas[name] + typeName := pascalCase(name) + "Row" + fmt.Fprintf(&sb, "// %s represents a row in the %q table.\ntype %s struct {\n", typeName, name, typeName) + for _, col := range schema.Columns { + goType := chTypeToGo(col.Type) + fieldName := pascalCase(col.Name) + jsonTag := col.Name + if col.HasDefault { + jsonTag += ",omitempty" + } + fmt.Fprintf(&sb, "\t%s %s `json:%q`\n", fieldName, goType, jsonTag) + } + sb.WriteString("}\n\n") + } + + return sb.String() +} + +func main() { + args := parseArgs() + fmt.Printf("Fetching schema from %s...\n", args.url) + + schemas, err := fetchSchemas(context.Background(), args.url, args.auth) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + if len(schemas) == 0 { + fmt.Fprintln(os.Stderr, "No tables found. Is WaveHouse running with tables in ClickHouse?") + os.Exit(1) + } + + names := make([]string, 0, len(schemas)) + for name := range schemas { + names = append(names, name) + } + slices.Sort(names) + fmt.Printf("Found %d table(s): %s\n", len(schemas), strings.Join(names, ", ")) + + output := generate(schemas, args.pkg) + + // gofmt the output. A failure here means the generated source is not + // valid Go (e.g. a table/column name produced an invalid identifier); + // don't write unusable output and claim success. + formatted, err := format.Source([]byte(output)) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: generated code is not valid Go: %v\n", err) + os.Exit(1) + } + + if err := os.WriteFile(args.out, formatted, 0o644); err != nil { + fmt.Fprintf(os.Stderr, "Error writing %s: %v\n", args.out, err) + os.Exit(1) + } + + fmt.Printf("✓ Types written to %s\n", args.out) +} diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go new file mode 100644 index 00000000..5f5edf74 --- /dev/null +++ b/clients/go/conformance_test.go @@ -0,0 +1,377 @@ +package wavehouse + +import ( + "context" + _ "embed" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "reflect" + "strings" + "testing" +) + +// wireCasesJSON embeds the shared wire-format conformance fixture so the +// test binary is self-contained: it works from a module archive or a +// standalone checkout without depending on paths outside the Go module. +// +//go:embed testdata/wire_cases.json +var wireCasesJSON []byte + +// wireCase is one entry in the shared wire_cases.json fixture. +type wireCase struct { + Name string `json:"name"` + Endpoint string `json:"endpoint"` + Table string `json:"table"` + Operations []wireOp `json:"operations"` + PipeName string `json:"pipe_name"` + PipeParams map[string]any `json:"pipe_params"` + PipeDefBody json.RawMessage `json:"pipe_def"` + PolicyBody json.RawMessage `json:"policy_body"` + SQL string `json:"sql"` + ExpectedPath string `json:"expected_path"` + ExpectedMethod string `json:"expected_method"` + ExpectedContentType string `json:"expected_content_type"` + ExpectedBody json.RawMessage `json:"expected_body"` + ExpectedRawBody *string `json:"expected_raw_body"` +} + +type wireOp struct { + Method string `json:"method"` + Args []any `json:"args"` +} + +func loadWireCases(t *testing.T) []wireCase { + t.Helper() + var cases []wireCase + if err := json.Unmarshal(wireCasesJSON, &cases); err != nil { + t.Fatalf("parse wire_cases.json: %v", err) + } + return cases +} + +// captured holds the HTTP request details from a single SDK call. +type captured struct { + method string + path string // path + query string + contentType string + body string +} + +func TestConformance_WireFormat(t *testing.T) { + cases := loadWireCases(t) + + for _, tc := range cases { + t.Run(tc.Name, func(t *testing.T) { + var cap captured + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + cap.method = r.Method + cap.path = r.URL.RequestURI() + cap.contentType = r.Header.Get("Content-Type") + raw, _ := io.ReadAll(r.Body) + cap.body = string(raw) + + // Return valid JSON so the SDK doesn't error on decode. + w.Header().Set("Content-Type", "application/json") + switch { + case strings.HasPrefix(r.URL.Path, "/v1/dlq"): + json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{}, Total: 0}) + case strings.HasPrefix(r.URL.Path, "/v1/schema") && r.Method == "GET": + json.NewEncoder(w).Encode([]TableSchema{}) + case r.URL.Path == "/v1/admin/policy/validate" && r.Method == "POST": + json.NewEncoder(w).Encode(ValidationResult{Valid: true}) + case strings.HasPrefix(r.URL.Path, "/v1/admin/policy") && r.Method == "GET": + json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) + case strings.HasPrefix(r.URL.Path, "/v1/admin/pipes/") && r.Method == "GET": + json.NewEncoder(w).Encode(Pipe{Name: "test", SQL: "SELECT 1"}) + case r.URL.Path == "/v1/admin/pipes" && r.Method == "GET": + json.NewEncoder(w).Encode([]Pipe{}) + default: + json.NewEncoder(w).Encode([]map[string]any{}) + } + })) + defer srv.Close() + + c := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{MaxRetries: 0}, + }) + ctx := context.Background() + + // Execute the case. + switch tc.Endpoint { + case "query": + q := c.From(tc.Table).Select() + q = applyOps(t, q, tc.Table, c, tc.Operations) + _, _ = q.FetchUntyped(ctx) + + case "ingest": + if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { + data := tc.Operations[0].Args[0] + _, _ = c.From(tc.Table).Insert(ctx, data) + } + + case "ingest_batch": + if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { + rawArr, ok := tc.Operations[0].Args[0].([]any) + if !ok { + t.Fatalf("batch insert args[0] is not an array") + } + rows := make([]map[string]any, len(rawArr)) + for i, r := range rawArr { + rows[i] = toStringMap(r) + } + _, _ = c.From(tc.Table).Insert(ctx, rows) + } + + case "pipe": + p := c.Pipe(tc.PipeName, tc.PipeParams) + _, _ = p.FetchUntyped(ctx) + + case "sql": + _, _ = SQL[map[string]any](ctx, c, tc.SQL) + + case "health": + _ = c.Sys.Health(ctx) + + case "schema_list": + _, _ = c.Schema.List(ctx) + + case "schema_refresh": + _ = c.Schema.Refresh(ctx) + + case "policy_get": + _, _ = c.Policy.Get(ctx) + + case "policy_set": + var pol Policy + if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { + t.Fatalf("parse policy_body: %v", err) + } + _ = c.Policy.Set(ctx, &pol) + + case "policy_validate": + var pol Policy + if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { + t.Fatalf("parse policy_body: %v", err) + } + _, _ = c.Policy.Validate(ctx, &pol) + + case "dlq_list": + _, _ = c.DLQ.List(ctx) + + case "dlq_table": + _, _ = c.DLQ.Table(ctx, tc.Table) + + case "pipes_list": + _, _ = c.Pipes.List(ctx) + + case "pipes_get": + _, _ = c.Pipes.Get(ctx, tc.PipeName) + + case "pipes_set": + var def PipeDef + if err := json.Unmarshal(tc.PipeDefBody, &def); err != nil { + t.Fatalf("parse pipe_def: %v", err) + } + _ = c.Pipes.Set(ctx, tc.PipeName, def) + + case "pipes_delete": + _ = c.Pipes.Delete(ctx, tc.PipeName) + + default: + t.Skipf("unhandled endpoint: %s", tc.Endpoint) + } + + // Verify method. + if tc.ExpectedMethod != "" && cap.method != tc.ExpectedMethod { + t.Errorf("method: want %s, got %s", tc.ExpectedMethod, cap.method) + } + + // Verify path. + if tc.ExpectedPath != "" { + // Normalize: the SDK may use different encoding (+ vs %20). + wantPath := normalizePath(tc.ExpectedPath) + gotPath := normalizePath(cap.path) + if wantPath != gotPath { + t.Errorf("path: want %s, got %s", tc.ExpectedPath, cap.path) + } + } + + // Verify content type. + if tc.ExpectedContentType != "" && cap.contentType != tc.ExpectedContentType { + t.Errorf("content-type: want %s, got %s", tc.ExpectedContentType, cap.contentType) + } + + // Verify raw body (for NDJSON). + if tc.ExpectedRawBody != nil { + if cap.body != *tc.ExpectedRawBody { + t.Errorf("raw body:\n want: %s\n got: %s", *tc.ExpectedRawBody, cap.body) + } + return + } + + // Verify JSON body. + if tc.ExpectedBody != nil && string(tc.ExpectedBody) != "null" { + var want, got any + if err := json.Unmarshal(tc.ExpectedBody, &want); err != nil { + t.Fatalf("parse expected_body: %v", err) + } + if err := json.Unmarshal([]byte(cap.body), &got); err != nil { + t.Fatalf("parse captured body: %v (body: %s)", err, cap.body) + } + if !deepEqualJSON(want, got) { + wantJSON, _ := json.MarshalIndent(want, "", " ") + gotJSON, _ := json.MarshalIndent(got, "", " ") + t.Errorf("body mismatch:\n want: %s\n got: %s", wantJSON, gotJSON) + } + } + }) + } +} + +// applyOps replays the operation chain from the fixture onto a QueryBuilder. +// Fixtures always put select first (mirroring real usage), so rebuilding on +// select is safe and keeps this simple. +func applyOps(t *testing.T, _ *QueryBuilder, table string, c *Client, ops []wireOp) *QueryBuilder { + t.Helper() + q := c.From(table).Select() + + for _, op := range ops { + switch op.Method { + case "select": + q = c.From(table).Select(toStringSlice(op.Args)...) + case "selectAll": + q = q.SelectAll() + case "where": + if len(op.Args) != 3 { + t.Fatalf("where needs 3 args, got %d", len(op.Args)) + } + col := op.Args[0].(string) + opStr := FilterOp(op.Args[1].(string)) + val := op.Args[2] + q = q.Where(col, opStr, val) + case "count": + col, alias := stringArg(op.Args, 0, "*"), stringArg(op.Args, 1, "count") + q = q.Count(col, alias) + case "sum": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Sum(col, alias) + case "avg": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Avg(col, alias) + case "min": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Min(col, alias) + case "max": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Max(col, alias) + case "countDistinct": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.CountDistinct(col, alias) + case "aggregate": + fn := stringArg(op.Args, 0, "") + col := stringArg(op.Args, 1, "") + alias := stringArg(op.Args, 2, "") + q = q.Aggregate(fn, col, alias) + case "groupBy": + cols := toStringSlice(op.Args) + q = q.GroupBy(cols...) + case "orderBy": + col := stringArg(op.Args, 0, "") + dir := stringArg(op.Args, 1, "asc") + q = q.OrderBy(col, dir) + case "limit": + n := intArg(op.Args, 0) + q = q.Limit(n) + case "timeRange": + col := stringArg(op.Args, 0, "") + since := stringArg(op.Args, 1, "") + until := stringArg(op.Args, 2, "") + q = q.TimeRange(col, since, until) + case "cacheTTL": + n := intArg(op.Args, 0) + q = q.CacheTTL(n) + } + } + return q +} + +func stringArg(args []any, i int, fallback string) string { + if i >= len(args) { + return fallback + } + s, ok := args[i].(string) + if !ok { + return fallback + } + return s +} + +func intArg(args []any, i int) int { + if i >= len(args) { + return 0 + } + switch v := args[i].(type) { + case float64: + return int(v) + case int: + return v + default: + return 0 + } +} + +func toStringSlice(args []any) []string { + out := make([]string, len(args)) + for i, a := range args { + out[i], _ = a.(string) + } + return out +} + +func toStringMap(v any) map[string]any { + m, ok := v.(map[string]any) + if ok { + return m + } + return nil +} + +// deepEqualJSON compares two JSON-decoded values, treating float64 ints as equal +// to ints (JSON numbers decode as float64 in Go). +func deepEqualJSON(a, b any) bool { + return reflect.DeepEqual(normalizeJSON(a), normalizeJSON(b)) +} + +func normalizeJSON(v any) any { + switch val := v.(type) { + case map[string]any: + m := make(map[string]any, len(val)) + for k, v := range val { + m[k] = normalizeJSON(v) + } + return m + case []any: + s := make([]any, len(val)) + for i, v := range val { + s[i] = normalizeJSON(v) + } + return s + case float64: + // Normalize integer-valued floats to int for comparison. + if val == float64(int64(val)) { + return int64(val) + } + return val + default: + return val + } +} + +func normalizePath(p string) string { + // Normalize URL encoding differences (+ vs %20 for spaces). + return strings.ReplaceAll(p, "+", "%20") +} diff --git a/clients/go/dlq.go b/clients/go/dlq.go new file mode 100644 index 00000000..01248b2c --- /dev/null +++ b/clients/go/dlq.go @@ -0,0 +1,42 @@ +package wavehouse + +import ( + "context" + "net/url" +) + +// DLQNamespace provides admin-only dead-letter-queue statistics. +type DLQNamespace struct { + ctx httpContext + createStream func(table string, opts *StreamOptions) *StreamController +} + +// List returns DLQ statistics (message counts per table). Admin-only. +func (d *DLQNamespace) List(ctx context.Context) (*DLQStats, error) { + var stats DLQStats + if err := doRequest(d.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/dlq/stats", + }, &stats); err != nil { + return nil, err + } + return &stats, nil +} + +// Table returns DLQ stats filtered by table name. Admin-only. +func (d *DLQNamespace) Table(ctx context.Context, name string) (*DLQStats, error) { + var stats DLQStats + if err := doRequest(d.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/dlq/stats", + params: url.Values{"table": {name}}, + }, &stats); err != nil { + return nil, err + } + return &stats, nil +} + +// Stream subscribes to live DLQ events. Not yet functional server-side (#197). +func (d *DLQNamespace) Stream(opts *StreamOptions) *StreamController { + return d.createStream("dlq", opts) +} diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go new file mode 100644 index 00000000..ea8aa7bc --- /dev/null +++ b/clients/go/e2e_test.go @@ -0,0 +1,436 @@ +//go:build e2e + +package wavehouse + +import ( + "context" + "fmt" + "net/http" + "os" + "strings" + "testing" + "time" +) + +// e2eClient builds a Client pointing at the live WaveHouse instance. +// It reads WAVEHOUSE_URL (default http://localhost:8080) and the optional +// WAVEHOUSE_AUTH bearer token. The test is skipped when the server is +// unreachable — so `go test -tags e2e` degrades gracefully on a dev +// machine that isn't running the stack. +func e2eClient(t *testing.T) *Client { + t.Helper() + + base := os.Getenv("WAVEHOUSE_URL") + if base == "" { + base = "http://localhost:8080" + } + + cfg := Config{ + BaseURL: base, + Options: &ClientOptions{MaxRetries: 1}, + } + if tok := os.Getenv("WAVEHOUSE_AUTH"); tok != "" { + cfg.Auth = StaticToken(tok) + } + + // Probe the server before committing to the test. + probe, err := http.NewRequestWithContext( + context.Background(), "GET", base+"/v1/health", nil, + ) + if err != nil { + t.Skipf("e2e: bad WAVEHOUSE_URL %q: %v", base, err) + } + resp, err := http.DefaultClient.Do(probe) + if err != nil { + t.Skipf("e2e: server unreachable at %s: %v", base, err) + } + resp.Body.Close() + + return NewClient(cfg) +} + +// marker returns a unique string for the running test, useful for +// inserting distinguishable rows that won't collide across parallel runs. +func marker(t *testing.T) string { + t.Helper() + // Replace slashes in subtest names so it's a clean string value. + safe := strings.ReplaceAll(t.Name(), "/", "_") + return fmt.Sprintf("%s_%d", safe, time.Now().UnixNano()) +} + +// firstTable discovers a usable table from the schema list. Many E2E tests +// need a real table to insert/query — this avoids hardcoding a name. +func firstTable(t *testing.T, c *Client) string { + t.Helper() + ctx := context.Background() + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Skipf("e2e: cannot list schemas (auth?): %v", err) + } + for name := range schemas { + return name + } + t.Skip("e2e: no tables found — server has an empty schema") + return "" +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +func TestE2E_HealthCheck(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + if err := c.Sys.Health(ctx); err != nil { + t.Fatalf("Health check failed: %v", err) + } +} + +func TestE2E_SchemaList(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Fatalf("Schema.List failed: %v", err) + } + if len(schemas) == 0 { + t.Fatal("Schema.List returned zero tables — expected at least one") + } + // Quick sanity: every table should have columns. + for name, ts := range schemas { + if len(ts.Columns) == 0 { + t.Errorf("table %q has no columns", name) + } + } +} + +func TestE2E_InsertAndQuery(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table := firstTable(t, c) + mk := marker(t) + + // Discover columns so we can build a valid row. We need at least one + // string-ish column to inject our marker. Fall back to skipping if the + // table's schema doesn't have one we can use. + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Fatalf("Schema.List: %v", err) + } + ts, ok := schemas[table] + if !ok { + t.Skipf("table %q vanished between discovery and use", table) + } + + row := buildMarkerRow(t, ts, mk) + markerCol := markerColumn(t, ts) + + res, err := c.From(table).Insert(ctx, row) + if err != nil { + t.Fatalf("Insert into %s failed: %v", table, err) + } + if !res.OK { + t.Fatalf("Insert into %s: OK=false", table) + } + + // Allow a moment for async ingestion to settle. + time.Sleep(500 * time.Millisecond) + + // Query it back. + page, err := c.From(table).Select(markerCol). + Where(markerCol, OpEq, mk). + Limit(1). + FetchUntyped(ctx) + if err != nil { + t.Fatalf("Query failed: %v", err) + } + if len(page.Data) == 0 { + t.Fatal("Query returned zero rows — expected the inserted marker row") + } + got, _ := page.Data[0][markerCol].(string) + if got != mk { + t.Errorf("marker mismatch: want %q, got %q", mk, got) + } +} + +func TestE2E_BatchInsert(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table := firstTable(t, c) + + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Fatalf("Schema.List: %v", err) + } + ts := schemas[table] + + mk := marker(t) + markerCol := markerColumn(t, ts) + + // Build 3 rows, each with the same marker so we can count them. + rows := make([]map[string]any, 3) + for i := range rows { + rows[i] = buildMarkerRow(t, ts, mk) + } + + res, err := c.From(table).Insert(ctx, rows) + if err != nil { + t.Fatalf("Batch insert failed: %v", err) + } + if !res.OK { + t.Fatalf("Batch insert: OK=false") + } + + time.Sleep(500 * time.Millisecond) + + page, err := c.From(table).Select(markerCol). + Where(markerCol, OpEq, mk). + Limit(10). + FetchUntyped(ctx) + if err != nil { + t.Fatalf("Query after batch insert failed: %v", err) + } + if len(page.Data) < 3 { + t.Fatalf("expected >= 3 rows for marker %q, got %d", mk, len(page.Data)) + } +} + +func TestE2E_QueryBuilder(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table := firstTable(t, c) + + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Fatalf("Schema.List: %v", err) + } + ts := schemas[table] + + // Pick two columns for a minimal projection. + var cols []string + for _, col := range ts.Columns { + cols = append(cols, col.Name) + if len(cols) >= 2 { + break + } + } + + page, err := c.From(table). + Select(cols...). + OrderBy(cols[0], "asc"). + Limit(5). + FetchUntyped(ctx) + if err != nil { + t.Fatalf("QueryBuilder chain failed: %v", err) + } + // We can't assert exact data, but the chain should execute without error + // and return at most 5 rows. + if len(page.Data) > 5 { + t.Errorf("Limit(5) returned %d rows", len(page.Data)) + } +} + +func TestE2E_TypedFetch(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table := firstTable(t, c) + + q := c.From(table).SelectAll().Limit(3) + page, err := FetchTyped[map[string]any](ctx, q) + if err != nil { + t.Fatalf("FetchTyped failed: %v", err) + } + // If the table has data we should get rows; if it's empty that's still + // a valid result. The important thing is no error and correct type. + for i, row := range page.Data { + if row == nil { + t.Errorf("row %d is nil", i) + } + } +} + +func TestE2E_SQLQuery(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + rows, err := SQL[map[string]any](ctx, c, "SELECT 1 AS n") + if err != nil { + // SQL requires admin role — skip gracefully if forbidden. + if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { + t.Skipf("e2e: SQL query requires admin auth: %v", err) + } + t.Fatalf("SQL query failed: %v", err) + } + if len(rows) != 1 { + t.Fatalf("expected 1 row, got %d", len(rows)) + } + // ClickHouse returns numbers as strings or floats depending on format; + // accept either. + n := rows[0]["n"] + switch v := n.(type) { + case float64: + if v != 1 { + t.Errorf("expected n=1, got %v", v) + } + case string: + if v != "1" { + t.Errorf("expected n=1, got %q", v) + } + default: + t.Errorf("unexpected type for n: %T = %v", n, n) + } +} + +func TestE2E_PolicyGetSet(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + pol, err := c.Policy.Get(ctx) + if err != nil { + if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { + t.Skipf("e2e: Policy.Get requires admin auth: %v", err) + } + t.Fatalf("Policy.Get failed: %v", err) + } + + // Round-trip: set the same policy back. + if err := c.Policy.Set(ctx, pol); err != nil { + t.Fatalf("Policy.Set (round-trip) failed: %v", err) + } + + // Read again and verify tables still match. + pol2, err := c.Policy.Get(ctx) + if err != nil { + t.Fatalf("Policy.Get (after set) failed: %v", err) + } + if len(pol2.Tables) != len(pol.Tables) { + t.Errorf("policy table count changed: %d -> %d", len(pol.Tables), len(pol2.Tables)) + } +} + +func TestE2E_PipesCRUD(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + pipeName := fmt.Sprintf("e2e_test_%d", time.Now().UnixNano()) + + // Create + def := PipeDef{ + SQL: "SELECT 1 AS ok", + Description: "E2E test pipe — safe to delete", + } + if err := c.Pipes.Set(ctx, pipeName, def); err != nil { + if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { + t.Skipf("e2e: Pipes.Set requires admin auth: %v", err) + } + t.Fatalf("Pipes.Set (create) failed: %v", err) + } + + // Cleanup: always attempt delete so we don't litter. + t.Cleanup(func() { + _ = c.Pipes.Delete(context.Background(), pipeName) + }) + + // Get + pipe, err := c.Pipes.Get(ctx, pipeName) + if err != nil { + t.Fatalf("Pipes.Get failed: %v", err) + } + if pipe.SQL != def.SQL { + t.Errorf("pipe SQL mismatch: want %q, got %q", def.SQL, pipe.SQL) + } + + // List — verify it appears + pipes, err := c.Pipes.List(ctx) + if err != nil { + t.Fatalf("Pipes.List failed: %v", err) + } + found := false + for _, p := range pipes { + if p.Name == pipeName { + found = true + break + } + } + if !found { + t.Errorf("Pipes.List: created pipe %q not found in list of %d pipes", pipeName, len(pipes)) + } + + // Delete + if err := c.Pipes.Delete(ctx, pipeName); err != nil { + t.Fatalf("Pipes.Delete failed: %v", err) + } + + // Verify gone — Get should fail. + _, err = c.Pipes.Get(ctx, pipeName) + if err == nil { + t.Error("Pipes.Get after delete: expected error, got nil") + } +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +// markerColumn finds the first String/LowCardinality(String) column in the +// schema that we can use to inject a test marker value. +func markerColumn(t *testing.T, ts TableSchema) string { + t.Helper() + for _, col := range ts.Columns { + ct := strings.ToLower(col.Type) + if ct == "string" || strings.Contains(ct, "string") { + return col.Name + } + } + t.Skipf("e2e: table %q has no string column for marker injection", ts.Name) + return "" +} + +// buildMarkerRow constructs a minimal valid row for the table, injecting the +// marker into the first string column and using sensible defaults for other +// required columns. +func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { + t.Helper() + row := make(map[string]any) + markerSet := false + for _, col := range ts.Columns { + if col.HasDefault { + continue // let the server fill defaults + } + ct := strings.ToLower(col.Type) + switch { + case !markerSet && strings.Contains(ct, "string"): + row[col.Name] = mk + markerSet = true + case strings.Contains(ct, "string"): + row[col.Name] = "e2e" + case strings.Contains(ct, "int"): + row[col.Name] = 0 + case strings.Contains(ct, "float") || strings.Contains(ct, "decimal"): + row[col.Name] = 0.0 + case strings.Contains(ct, "date") || strings.Contains(ct, "datetime"): + row[col.Name] = time.Now().UTC().Format(time.RFC3339) + case strings.Contains(ct, "bool"): + row[col.Name] = false + default: + row[col.Name] = "" + } + } + if !markerSet { + t.Skipf("e2e: table %q has no non-default string column for marker", ts.Name) + } + return row +} + +// isHTTPStatus checks whether err is a wavehouse.Error with the given status. +func isHTTPStatus(err error, status int) bool { + if err == nil { + return false + } + if e, ok := err.(*Error); ok { + return e.Status == status + } + return false +} diff --git a/clients/go/errors.go b/clients/go/errors.go new file mode 100644 index 00000000..1a161a1b --- /dev/null +++ b/clients/go/errors.go @@ -0,0 +1,81 @@ +package wavehouse + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "net/http" +) + +// Error is the structured error returned by all SDK operations. Use +// [errors.As] to extract it from wrapped errors. +type Error struct { + // Status is the HTTP status code (0 for network/abort errors). + Status int `json:"status"` + // Code is a machine-readable error code (e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED"). + Code string `json:"code"` + // Message is a human-readable description. + Message string `json:"message"` + // Details contains the full parsed error body, if available. + Details map[string]any `json:"details,omitempty"` + // Retryable indicates whether the request can be retried. + Retryable bool `json:"retryable"` +} + +func (e *Error) Error() string { + if e.Status > 0 { + return fmt.Sprintf("wavehouse: %s (%d): %s", e.Code, e.Status, e.Message) + } + return fmt.Sprintf("wavehouse: %s: %s", e.Code, e.Message) +} + +// IsRetryable reports whether err wraps a retryable [*Error]. +func IsRetryable(err error) bool { + var e *Error + if errors.As(err, &e) { + return e.Retryable + } + return false +} + +// parseErrorResponse creates an Error from an HTTP response. +func parseErrorResponse(res *http.Response) *Error { + var body map[string]any + if res.Body != nil { + raw, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20)) // cap at 1 MiB + _ = json.Unmarshal(raw, &body) + } + + msg := "" + if s, ok := body["error"].(string); ok { + msg = s + } else if s, ok := body["message"].(string); ok { + msg = s + } else { + msg = http.StatusText(res.StatusCode) + } + + retryable := res.StatusCode == http.StatusServiceUnavailable || res.StatusCode >= 500 + return &Error{ + Status: res.StatusCode, + Code: fmt.Sprintf("HTTP_%d", res.StatusCode), + Message: msg, + Details: body, + Retryable: retryable, + } +} + +// networkError creates an Error from a transport-level failure. +func networkError(cause error) *Error { + msg := "unknown network error" + if cause != nil { + msg = cause.Error() + } + return &Error{ + Status: 0, + Code: "NETWORK_ERROR", + Message: msg, + Retryable: true, + } +} diff --git a/clients/go/errors_test.go b/clients/go/errors_test.go new file mode 100644 index 00000000..0c9dd05f --- /dev/null +++ b/clients/go/errors_test.go @@ -0,0 +1,145 @@ +package wavehouse + +import ( + "errors" + "io" + "net/http" + "strings" + "testing" +) + +func TestParseErrorResponse_JSONError(t *testing.T) { + res := &http.Response{ + StatusCode: 404, + Body: io.NopCloser(strings.NewReader(`{"error":"unknown table: foo"}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Status != 404 { + t.Fatalf("want status 404, got %d", e.Status) + } + if e.Code != "HTTP_404" { + t.Fatalf("want code HTTP_404, got %s", e.Code) + } + if e.Message != "unknown table: foo" { + t.Fatalf("want message 'unknown table: foo', got %s", e.Message) + } + if e.Retryable { + t.Fatal("4xx should not be retryable") + } +} + +func TestParseErrorResponse_MessageField(t *testing.T) { + res := &http.Response{ + StatusCode: 400, + Body: io.NopCloser(strings.NewReader(`{"message":"bad request"}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Message != "bad request" { + t.Fatalf("want 'bad request', got %s", e.Message) + } +} + +func TestParseErrorResponse_FallsBackToStatusText(t *testing.T) { + res := &http.Response{ + StatusCode: 500, + Body: io.NopCloser(strings.NewReader(`{"code":123}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Message != "Internal Server Error" { + t.Fatalf("want status text fallback, got %s", e.Message) + } +} + +func TestParseErrorResponse_NonJSONBody(t *testing.T) { + res := &http.Response{ + StatusCode: 502, + Body: io.NopCloser(strings.NewReader("plain text")), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Message != "Bad Gateway" { + t.Fatalf("want 'Bad Gateway', got %s", e.Message) + } + if e.Details != nil { + t.Fatal("details should be nil for non-JSON body") + } +} + +func TestParseErrorResponse_5xxRetryable(t *testing.T) { + tests := []struct { + status int + retryable bool + }{ + {400, false}, + {403, false}, + {500, true}, + {503, true}, + } + for _, tt := range tests { + res := &http.Response{ + StatusCode: tt.status, + Body: io.NopCloser(strings.NewReader(`{"error":"test"}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Retryable != tt.retryable { + t.Errorf("status %d: want retryable=%v, got %v", tt.status, tt.retryable, e.Retryable) + } + } +} + +func TestNetworkError(t *testing.T) { + e := networkError(errors.New("connection refused")) + if e.Code != "NETWORK_ERROR" { + t.Fatalf("want NETWORK_ERROR, got %s", e.Code) + } + if e.Message != "connection refused" { + t.Fatalf("want 'connection refused', got %s", e.Message) + } + if !e.Retryable { + t.Fatal("network errors should be retryable") + } + if e.Status != 0 { + t.Fatalf("want status 0, got %d", e.Status) + } +} + +func TestError_ErrorMethod(t *testing.T) { + e := &Error{Status: 404, Code: "HTTP_404", Message: "not found"} + got := e.Error() + if !strings.Contains(got, "HTTP_404") || !strings.Contains(got, "not found") { + t.Fatalf("unexpected Error() output: %s", got) + } + + e2 := &Error{Status: 0, Code: "NETWORK_ERROR", Message: "timeout"} + got2 := e2.Error() + if !strings.Contains(got2, "NETWORK_ERROR") { + t.Fatalf("unexpected Error() output: %s", got2) + } +} + +func TestIsRetryable(t *testing.T) { + if !IsRetryable(&Error{Retryable: true}) { + t.Fatal("want true for retryable error") + } + if IsRetryable(&Error{Retryable: false}) { + t.Fatal("want false for non-retryable error") + } + if IsRetryable(errors.New("plain error")) { + t.Fatal("want false for non-wavehouse error") + } +} + +func TestErrorsAs(t *testing.T) { + err := error(&Error{Status: 403, Code: "HTTP_403", Message: "forbidden"}) + var e *Error + if !errors.As(err, &e) { + t.Fatal("errors.As should find *Error") + } + if e.Status != 403 { + t.Fatalf("want 403, got %d", e.Status) + } +} diff --git a/clients/go/example_test.go b/clients/go/example_test.go new file mode 100644 index 00000000..03d8b518 --- /dev/null +++ b/clients/go/example_test.go @@ -0,0 +1,71 @@ +package wavehouse_test + +import ( + "context" + "fmt" + "log" + + wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +) + +// ExampleNewClient demonstrates creating an unauthenticated client and +// performing a health check. The Output assertion is omitted because the +// example needs a running server. +func ExampleNewClient() { + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + }) + + // Health check — returns nil when the server is reachable. + err := client.Sys.Health(context.Background()) + _ = err +} + +func ExampleNewClient_withAuth() { + _ = wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("my-jwt-token"), + }) +} + +func ExampleClient_From() { + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + }) + + // Insert a row. + _, _ = client.From("clicks").Insert(context.Background(), map[string]any{ + "page": "/home", + "button": "cta", + }) + + // Query with the builder. + page, _ := client.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + OrderBy("page", "asc"). + Limit(10). + FetchUntyped(context.Background()) + + for _, row := range page.Data { + fmt.Println(row["page"]) + } +} + +func ExampleSQL() { + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("admin-token"), + }) + + rows, err := wavehouse.SQL[map[string]any]( + context.Background(), client, + "SELECT page, count() as views FROM clicks GROUP BY page LIMIT 5", + ) + if err != nil { + log.Fatal(err) + } + for _, row := range rows { + fmt.Println(row["page"], row["views"]) + } +} diff --git a/clients/go/go.mod b/clients/go/go.mod new file mode 100644 index 00000000..84e065de --- /dev/null +++ b/clients/go/go.mod @@ -0,0 +1,3 @@ +module github.com/Wave-RF/WaveHouse/clients/go + +go 1.26.5 diff --git a/clients/go/http.go b/clients/go/http.go new file mode 100644 index 00000000..52aae6b4 --- /dev/null +++ b/clients/go/http.go @@ -0,0 +1,213 @@ +package wavehouse + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "io" + "math" + "net/http" + "net/url" + "strconv" + "time" +) + +// httpContext carries per-client state needed by every request. +type httpContext struct { + baseURL string + auth func(ctx context.Context) (string, error) + maxRetries int + httpClient *http.Client +} + +// requestOptions describes a single HTTP request. +type requestOptions struct { + method string + path string + body any // JSON-serialized if non-nil + rawBody string // sent verbatim if non-empty (takes precedence over body) + contentType string // overrides Content-Type (default "application/json") + params url.Values +} + +// doRequest is the internal fetch wrapper with auth, retry, and backoff. +// It decodes the response body into dst (unless dst is nil). +func doRequest(hctx httpContext, ctx context.Context, opts requestOptions, dst any) error { + reqURL := buildURL(hctx.baseURL, opts.path, opts.params) + ct := opts.contentType + if ct == "" { + ct = "application/json" + } + + // Serialize body once so every retry sends identical bytes. + var bodyBytes []byte + if opts.rawBody != "" { + bodyBytes = []byte(opts.rawBody) + } else if opts.body != nil { + var err error + bodyBytes, err = json.Marshal(opts.body) + if err != nil { + return fmt.Errorf("wavehouse: marshal request body: %w", err) + } + } + + // Resolve auth once per request (not per attempt). + var authHeader string + if hctx.auth != nil { + token, err := hctx.auth(ctx) + if err != nil { + return fmt.Errorf("wavehouse: auth provider: %w", err) + } + if token != "" { + authHeader = "Bearer " + token + } + } + + var lastErr error + maxAttempts := hctx.maxRetries + 1 + + // Retries below are not restricted by HTTP method — this matches the TS + // SDK's http.ts, which retries POST the same as GET on network errors, + // 503/Retry-After, and other retryable 5xx. For /v1/ingest, at-least-once + // delivery on retry is a documented contract (see docs/api.md's + // "At-least-once on retry" note); dedup is the prescribed server-side + // safety net when duplicate suppression matters. The only other mutation + // path, /v1/admin/query, is gated by admin_role, so repeated execution on + // retry is assumed to be an accepted risk for admin-only raw SQL. + for attempt := range maxAttempts { + var bodyReader io.Reader + if bodyBytes != nil { + bodyReader = bytes.NewReader(bodyBytes) + } + + req, err := http.NewRequestWithContext(ctx, opts.method, reqURL, bodyReader) + if err != nil { + return fmt.Errorf("wavehouse: build request: %w", err) + } + req.Header.Set("Content-Type", ct) + req.Header.Set("Accept", "application/json") + if authHeader != "" { + req.Header.Set("Authorization", authHeader) + } + + res, err := hctx.httpClient.Do(req) + if err != nil { + // Context cancellation — return immediately, no retry. + if ctx.Err() != nil { + return &Error{ + Status: 0, + Code: "ABORTED", + Message: "Request aborted", + Retryable: false, + } + } + lastErr = networkError(err) + if attempt < maxAttempts-1 { + if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { + return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + } + } + continue + } + + if res.StatusCode >= 200 && res.StatusCode < 300 { + defer res.Body.Close() + if dst == nil { + _, _ = io.Copy(io.Discard, res.Body) + return nil + } + raw, readErr := io.ReadAll(res.Body) + if readErr != nil { + return networkError(readErr) + } + if len(raw) == 0 { + return nil + } + if err := json.Unmarshal(raw, dst); err != nil { + return &Error{ + Status: 0, + Code: "NETWORK_ERROR", + Message: fmt.Errorf("decode response: %w", err).Error(), + Retryable: false, + } + } + return nil + } + + apiErr := parseErrorResponse(res) + res.Body.Close() + + // 503 with Retry-After: wait the specified duration. + if res.StatusCode == http.StatusServiceUnavailable { + if ra := res.Header.Get("Retry-After"); ra != "" && attempt < maxAttempts-1 { + delay := 30 * time.Second + if secs, parseErr := strconv.Atoi(ra); parseErr == nil && secs > 0 { + delay = time.Duration(secs) * time.Second + } else if parsed, parseErr := http.ParseTime(ra); parseErr == nil { + if d := time.Until(parsed); d > 0 { + delay = d + } + } + if sleepErr := sleepWithContext(ctx, delay); sleepErr != nil { + return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + } + lastErr = apiErr + continue + } + } + + // Retryable server errors (5xx). + if apiErr.Retryable && attempt < maxAttempts-1 { + if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { + return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + } + lastErr = apiErr + continue + } + + return apiErr + } + + return lastErr +} + +func buildURL(base, path string, params url.Values) string { + u := base + path + if len(params) > 0 { + u += "?" + params.Encode() + } + return u +} + +func backoff(attempt int) time.Duration { + ms := 1000 * math.Pow(2, float64(attempt)) + if ms > 30000 { + ms = 30000 + } + return time.Duration(ms) * time.Millisecond +} + +func sleepWithContext(ctx context.Context, d time.Duration) error { + if ctx.Err() != nil { + return ctx.Err() + } + t := time.NewTimer(d) + defer t.Stop() + select { + case <-t.C: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} + +// errIs checks if err wraps a *Error with the given code. +func errIs(err error, code string) bool { + var e *Error + if errors.As(err, &e) { + return e.Code == code + } + return false +} diff --git a/clients/go/http_test.go b/clients/go/http_test.go new file mode 100644 index 00000000..7ce4c964 --- /dev/null +++ b/clients/go/http_test.go @@ -0,0 +1,218 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "sync/atomic" + "testing" + "time" +) + +func testCtx(handler http.Handler) httpContext { + srv := httptest.NewServer(handler) + return httpContext{ + baseURL: srv.URL, + maxRetries: 0, + httpClient: srv.Client(), + } +} + +func TestDoRequest_SuccessfulGET(t *testing.T) { + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + })) + + var result map[string]string + err := doRequest(hctx, context.Background(), requestOptions{ + method: "GET", + path: "/health", + }, &result) + if err != nil { + t.Fatal(err) + } + if result["status"] != "ok" { + t.Fatalf("want ok, got %v", result) + } +} + +func TestDoRequest_POSTWithBody(t *testing.T) { + var gotBody map[string]string + var gotCT string + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotCT = r.Header.Get("Content-Type") + json.NewDecoder(r.Body).Decode(&gotBody) + w.WriteHeader(200) + })) + + err := doRequest(hctx, context.Background(), requestOptions{ + method: "POST", + path: "/v1/ingest", + body: map[string]string{"page": "/home"}, + }, nil) + if err != nil { + t.Fatal(err) + } + if gotCT != "application/json" { + t.Fatalf("want application/json, got %s", gotCT) + } + if gotBody["page"] != "/home" { + t.Fatalf("want /home, got %v", gotBody) + } +} + +func TestDoRequest_RawBody(t *testing.T) { + var gotBody string + var gotCT string + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotCT = r.Header.Get("Content-Type") + raw := make([]byte, 1024) + n, _ := r.Body.Read(raw) + gotBody = string(raw[:n]) + json.NewEncoder(w).Encode(map[string]int{"total": 1}) + })) + + err := doRequest(hctx, context.Background(), requestOptions{ + method: "POST", + path: "/v1/ingest", + rawBody: `{"page":"/a"}`, + contentType: "application/x-ndjson", + }, nil) + if err != nil { + t.Fatal(err) + } + if gotCT != "application/x-ndjson" { + t.Fatalf("want ndjson content type, got %s", gotCT) + } + if gotBody != `{"page":"/a"}` { + t.Fatalf("want raw body, got %s", gotBody) + } +} + +func TestDoRequest_AuthInjection(t *testing.T) { + var gotAuth string + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotAuth = r.Header.Get("Authorization") + w.WriteHeader(200) + })) + hctx.auth = StaticToken("my-token") + + err := doRequest(hctx, context.Background(), requestOptions{ + method: "GET", + path: "/v1/schema", + }, nil) + if err != nil { + t.Fatal(err) + } + if gotAuth != "Bearer my-token" { + t.Fatalf("want 'Bearer my-token', got %s", gotAuth) + } +} + +func TestDoRequest_4xxNotRetried(t *testing.T) { + var count atomic.Int32 + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + count.Add(1) + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(404) + json.NewEncoder(w).Encode(map[string]string{"error": "not found"}) + })) + hctx.maxRetries = 2 + + err := doRequest(hctx, context.Background(), requestOptions{ + method: "GET", + path: "/v1/schema", + }, nil) + + if !errIs(err, "HTTP_404") { + t.Fatalf("want HTTP_404 error, got %v", err) + } + if count.Load() != 1 { + t.Fatalf("4xx should not retry, got %d attempts", count.Load()) + } +} + +func TestDoRequest_5xxRetried(t *testing.T) { + var count atomic.Int32 + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + n := count.Add(1) + if n < 3 { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(500) + json.NewEncoder(w).Encode(map[string]string{"error": "internal"}) + return + } + json.NewEncoder(w).Encode(map[string]string{"ok": "true"}) + })) + hctx.maxRetries = 2 + + var result map[string]string + err := doRequest(hctx, context.Background(), requestOptions{ + method: "GET", + path: "/health", + }, &result) + if err != nil { + t.Fatalf("want success after retries, got %v", err) + } + if count.Load() != 3 { + t.Fatalf("want 3 attempts, got %d", count.Load()) + } +} + +func TestDoRequest_AbortedOnCancel(t *testing.T) { + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + time.Sleep(5 * time.Second) + })) + + ctx, cancel := context.WithCancel(context.Background()) + cancel() // cancel immediately + + err := doRequest(hctx, ctx, requestOptions{ + method: "GET", + path: "/health", + }, nil) + + if !errIs(err, "ABORTED") { + t.Fatalf("want ABORTED, got %v", err) + } +} + +func TestDoRequest_EmptyResponse(t *testing.T) { + hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(200) + })) + + var result map[string]string + err := doRequest(hctx, context.Background(), requestOptions{ + method: "POST", + path: "/v1/schema/refresh", + }, &result) + if err != nil { + t.Fatal(err) + } + // Empty body = no decode, result stays zero value. + if result != nil { + t.Fatalf("want nil, got %v", result) + } +} + +func TestBackoff(t *testing.T) { + tests := []struct { + attempt int + want time.Duration + }{ + {0, 1 * time.Second}, + {1, 2 * time.Second}, + {2, 4 * time.Second}, + {3, 8 * time.Second}, + {10, 30 * time.Second}, // capped at 30s + } + for _, tt := range tests { + got := backoff(tt.attempt) + if got != tt.want { + t.Errorf("backoff(%d) = %v, want %v", tt.attempt, got, tt.want) + } + } +} diff --git a/clients/go/live_query.go b/clients/go/live_query.go new file mode 100644 index 00000000..fad4e80a --- /dev/null +++ b/clients/go/live_query.go @@ -0,0 +1,158 @@ +package wavehouse + +import ( + "context" + "sync" +) + +// LiveQueryHandle controls a live query that combines historical backfill +// with a real-time stream. +type LiveQueryHandle struct { + stream *StreamController + cancel context.CancelFunc + closeOnce sync.Once +} + +// newLiveQuery starts a live query: opens the stream immediately, fetches +// historical data, deduplicates buffered events, then goes live. +func newLiveQuery( + stream *StreamController, + fetchFn func(ctx context.Context) ([]map[string]any, error), + sub *StreamSubscriber, + filters []QueryFilter, +) *LiveQueryHandle { + ctx, cancel := context.WithCancel(context.Background()) + lq := &LiveQueryHandle{ + stream: stream, + cancel: cancel, + } + + var ( + mu sync.Mutex + buffer []StreamEvent + buffering = true + closed = false + ) + + // Step 1: Subscribe to live events and buffer them. + stream.Subscribe(&StreamSubscriber{ + Next: func(event StreamEvent) { + mu.Lock() + defer mu.Unlock() + if closed { + return + } + if buffering { + buffer = append(buffer, event) + } else if sub.Next != nil { + sub.Next(event) + } + }, + Status: func(s StreamStatus) { + if sub.Status != nil { + sub.Status(s) + } + }, + Error: func(err error) { + if sub.Error != nil { + sub.Error(err) + } + }, + }) + + // Step 2–5: Fetch historical and flush. + go func() { + rows, err := fetchFn(ctx) + if ctx.Err() != nil { + return + } + + // Step 3: Deliver initial snapshot. + if sub.Initial != nil { + sub.Initial(rows, err) + } + + if err != nil { + mu.Lock() + buffering = false + buffer = nil + mu.Unlock() + return + } + + // Step 4: Deduplicate buffered events. + var lastTimestamp string + if len(rows) > 0 { + lastRow := rows[len(rows)-1] + if ts, ok := lastRow["received_timestamp"].(string); ok { + lastTimestamp = ts + } + } + + // Step 5: Flush buffered events newer than the fetch. + // + // buffering stays true for the whole flush: events that arrive + // concurrently (after Subscribe's Next handler releases mu but + // before we're done here) must keep landing in buffer rather than + // being dispatched directly by the live path, or two goroutines + // could call sub.Next at once. We only flip buffering to false + // once a lock-protected check finds the buffer empty, which + // guarantees no event is ever handed to sub.Next by both paths + // and that delivery stays in arrival order. + for { + mu.Lock() + if closed { + mu.Unlock() + return + } + pending := buffer + buffer = nil + if len(pending) == 0 { + buffering = false + mu.Unlock() + break + } + mu.Unlock() + + for _, event := range pending { + mu.Lock() + c := closed + mu.Unlock() + if c { + return + } + // Use <= (not <) to filter events whose timestamp matches the last + // historical row — those rows were already delivered in the backfill + // response. If two distinct events share a timestamp and only one + // appeared in the backfill, the duplicate is lost; this matches the + // TS SDK's dedup behavior and is acceptable because received_timestamp + // has sub-millisecond precision in practice. + if lastTimestamp != "" && event.Timestamp <= lastTimestamp { + continue + } + if sub.Next != nil { + sub.Next(event) + } + } + } + }() + + // Cleanup on context cancel. + go func() { + <-ctx.Done() + mu.Lock() + closed = true + buffer = nil + mu.Unlock() + }() + + return lq +} + +// Close shuts down the live query and the underlying stream. +func (lq *LiveQueryHandle) Close() { + lq.closeOnce.Do(func() { + lq.cancel() + lq.stream.Close() + }) +} diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go new file mode 100644 index 00000000..84bc4b2f --- /dev/null +++ b/clients/go/namespaces_test.go @@ -0,0 +1,193 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" +) + +func nsClient(handler http.Handler) *Client { + srv := httptest.NewServer(handler) + return NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{MaxRetries: 0}, + }) +} + +func TestSysNamespace_Health(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/health" { + t.Errorf("want /v1/health, got %s", r.URL.Path) + } + w.WriteHeader(200) + })) + err := c.Sys.Health(context.Background()) + if err != nil { + t.Fatal(err) + } +} + +func TestSchemaNamespace_List(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/schema" { + t.Errorf("want /v1/schema, got %s", r.URL.Path) + } + json.NewEncoder(w).Encode([]TableSchema{ + {Name: "clicks", Columns: []Column{{Name: "page", Type: "String"}}}, + }) + })) + schemas, err := c.Schema.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if _, ok := schemas["clicks"]; !ok { + t.Fatal("want clicks in schemas") + } +} + +func TestSchemaNamespace_Refresh(t *testing.T) { + var gotMethod string + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotMethod = r.Method + w.WriteHeader(200) + })) + err := c.Schema.Refresh(context.Background()) + if err != nil { + t.Fatal(err) + } + if gotMethod != "POST" { + t.Fatalf("want POST, got %s", gotMethod) + } +} + +func TestPolicyNamespace_GetSetValidate(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch { + case r.Method == "GET": + json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) + case r.Method == "PUT": + w.WriteHeader(200) + case r.Method == "POST": + json.NewEncoder(w).Encode(ValidationResult{Valid: true}) + } + })) + + pol, err := c.Policy.Get(context.Background()) + if err != nil { + t.Fatal(err) + } + if pol.Tables == nil { + t.Fatal("want tables map") + } + + err = c.Policy.Set(context.Background(), pol) + if err != nil { + t.Fatal(err) + } + + v, err := c.Policy.Validate(context.Background(), pol) + if err != nil { + t.Fatal(err) + } + if !v.Valid { + t.Fatal("want valid=true") + } +} + +func TestDLQNamespace_List(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) + })) + stats, err := c.DLQ.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if stats.Total != 3 { + t.Fatalf("want total=3, got %d", stats.Total) + } +} + +func TestDLQNamespace_Table(t *testing.T) { + var gotParam string + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotParam = r.URL.Query().Get("table") + json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) + })) + _, err := c.DLQ.Table(context.Background(), "clicks") + if err != nil { + t.Fatal(err) + } + if gotParam != "clicks" { + t.Fatalf("want table=clicks, got %s", gotParam) + } +} + +func TestPipesNamespace_CRUD(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.Method { + case "GET": + if r.URL.Path == "/v1/admin/pipes" { + json.NewEncoder(w).Encode([]Pipe{{Name: "p1", SQL: "SELECT 1"}}) + } else { + json.NewEncoder(w).Encode(Pipe{Name: "p1", SQL: "SELECT 1"}) + } + case "PUT": + w.WriteHeader(200) + case "DELETE": + w.WriteHeader(200) + } + })) + + pipes, err := c.Pipes.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if len(pipes) != 1 || pipes[0].Name != "p1" { + t.Fatalf("want [p1], got %v", pipes) + } + + p, err := c.Pipes.Get(context.Background(), "p1") + if err != nil { + t.Fatal(err) + } + if p.Name != "p1" { + t.Fatalf("want p1, got %s", p.Name) + } + + err = c.Pipes.Set(context.Background(), "p1", PipeDef{SQL: "SELECT 1"}) + if err != nil { + t.Fatal(err) + } + + err = c.Pipes.Delete(context.Background(), "p1") + if err != nil { + t.Fatal(err) + } +} + +func TestPipeRef_Fetch(t *testing.T) { + var gotPath, gotMethod string + var gotBody map[string]any + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotPath = r.URL.Path + gotMethod = r.Method + json.NewDecoder(r.Body).Decode(&gotBody) + json.NewEncoder(w).Encode([]map[string]any{{"count": 42}}) + })) + rows, err := Fetch[map[string]any](context.Background(), c.Pipe("top_pages", map[string]any{"limit": 10})) + if err != nil { + t.Fatal(err) + } + if len(rows) != 1 { + t.Fatalf("want 1 row, got %d", len(rows)) + } + if gotPath != "/v1/pipes/top_pages" { + t.Fatalf("want /v1/pipes/top_pages, got %s", gotPath) + } + if gotMethod != "POST" { + t.Fatalf("want POST, got %s", gotMethod) + } +} diff --git a/clients/go/pipes.go b/clients/go/pipes.go new file mode 100644 index 00000000..f87f4d64 --- /dev/null +++ b/clients/go/pipes.go @@ -0,0 +1,99 @@ +package wavehouse + +import ( + "context" + "net/url" +) + +// PipesNamespace provides admin-only named-pipe management. +type PipesNamespace struct { + ctx httpContext +} + +// List returns all registered pipes. Admin-only. +func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { + var pipes []Pipe + if err := doRequest(p.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/admin/pipes", + }, &pipes); err != nil { + return nil, err + } + return pipes, nil +} + +// Get returns a single pipe definition by name. Admin-only. +func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { + var pipe Pipe + if err := doRequest(p.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/admin/pipes/" + url.PathEscape(name), + }, &pipe); err != nil { + return nil, err + } + return &pipe, nil +} + +// Set creates or updates a pipe. Admin-only. +func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) error { + return doRequest(p.ctx, ctx, requestOptions{ + method: "PUT", + path: "/v1/admin/pipes/" + url.PathEscape(name), + body: def, + }, nil) +} + +// Delete removes a pipe by name. Admin-only. +func (p *PipesNamespace) Delete(ctx context.Context, name string) error { + return doRequest(p.ctx, ctx, requestOptions{ + method: "DELETE", + path: "/v1/admin/pipes/" + url.PathEscape(name), + }, nil) +} + +// PipeDef is the definition body for creating/updating a pipe (Pipe minus name). +type PipeDef struct { + SQL string `json:"sql"` + Parameters []ParamDef `json:"parameters,omitempty"` + Description string `json:"description,omitempty"` + AllowedRoles []string `json:"allowed_roles,omitempty"` +} + +// PipeRef is a reference to a named query pipe. Use Fetch to execute it. +type PipeRef struct { + ctx httpContext + name string + params map[string]any + createStream func(table string, opts *StreamOptions) *StreamController +} + +// Fetch executes the pipe and returns the result rows decoded into []T. +func Fetch[Row any](ctx context.Context, p *PipeRef) ([]Row, error) { + body := p.params + if body == nil { + body = map[string]any{} + } + var rows []Row + if err := doRequest(p.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/pipes/" + url.PathEscape(p.name), + body: body, + }, &rows); err != nil { + return nil, err + } + return rows, nil +} + +// FetchUntyped executes the pipe and returns rows as []map[string]any. +func (p *PipeRef) FetchUntyped(ctx context.Context) ([]map[string]any, error) { + return Fetch[map[string]any](ctx, p) +} + +// Stream opens a live event stream from the pipe's underlying query. +// +// This streams by table name, using the pipe's own name as the table — it +// only works when the pipe name is also a valid table name. This matches +// the TS SDK's PipeRef.stream(), which has the same limitation. +func (p *PipeRef) Stream(opts *StreamOptions) *StreamController { + return p.createStream(p.name, opts) +} diff --git a/clients/go/policy.go b/clients/go/policy.go new file mode 100644 index 00000000..c7eaa2af --- /dev/null +++ b/clients/go/policy.go @@ -0,0 +1,42 @@ +package wavehouse + +import "context" + +// PolicyNamespace provides admin-only access-control policy management. +type PolicyNamespace struct { + ctx httpContext +} + +// Get returns the current access-control policy. Admin-only. +func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { + var pol Policy + if err := doRequest(p.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/admin/policy", + }, &pol); err != nil { + return nil, err + } + return &pol, nil +} + +// Set replaces the entire access-control policy. Admin-only. +func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { + return doRequest(p.ctx, ctx, requestOptions{ + method: "PUT", + path: "/v1/admin/policy", + body: pol, + }, nil) +} + +// Validate checks a policy without applying it (dry run). Admin-only. +func (p *PolicyNamespace) Validate(ctx context.Context, pol *Policy) (*ValidationResult, error) { + var result ValidationResult + if err := doRequest(p.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/admin/policy/validate", + body: pol, + }, &result); err != nil { + return nil, err + } + return &result, nil +} diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go new file mode 100644 index 00000000..9dda452d --- /dev/null +++ b/clients/go/query_builder.go @@ -0,0 +1,310 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/url" +) + +// DefaultLimit is applied when no explicit limit is set — deliberately tighter +// than the backend's DefaultMaxRows (10000) safety cap. +const DefaultLimit = 1000 + +// queryState is the immutable core of a QueryBuilder. +type queryState struct { + table string + columns []string + selectAll bool + aggregations []Aggregation + filters []QueryFilter + groupBy []string + orderBy []OrderClause + limit *int + timeRange *TimeRange + cacheTTL *int // ponytail: client-side only, not sent to server (#280) +} + +// QueryBuilder builds structured queries. Immutable — every chain method +// returns a new builder. Use Fetch or FetchUntyped to execute. +type QueryBuilder struct { + ctx httpContext + createStream func(table string, opts *StreamOptions) *StreamController + state queryState +} + +func (q *QueryBuilder) clone(mutate func(*queryState)) *QueryBuilder { + s := q.state + // Deep-copy slices so mutations don't alias. + s.columns = append([]string(nil), s.columns...) + s.aggregations = append([]Aggregation(nil), s.aggregations...) + s.filters = append([]QueryFilter(nil), s.filters...) + s.groupBy = append([]string(nil), s.groupBy...) + s.orderBy = append([]OrderClause(nil), s.orderBy...) + mutate(&s) + return &QueryBuilder{ctx: q.ctx, createStream: q.createStream, state: s} +} + +// Select appends columns to the projection. +func (q *QueryBuilder) Select(columns ...string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.columns = append(s.columns, columns...) + }) +} + +// SelectAll requests every column the caller's role may read. +func (q *QueryBuilder) SelectAll() *QueryBuilder { + return q.clone(func(s *queryState) { + s.selectAll = true + }) +} + +// Where adds a filter condition. +func (q *QueryBuilder) Where(column string, op FilterOp, value any) *QueryBuilder { + wireOp, ok := opMap[op] + if !ok { + wireOp = string(op) + } + return q.clone(func(s *queryState) { + s.filters = append(s.filters, QueryFilter{Column: column, Op: wireOp, Value: value}) + }) +} + +// Count adds a COUNT aggregation. +func (q *QueryBuilder) Count(column, alias string) *QueryBuilder { + if column == "" { + column = "*" + } + if alias == "" { + alias = "count" + } + return q.addAgg("count", column, alias) +} + +// Sum adds a SUM aggregation. +func (q *QueryBuilder) Sum(column, alias string) *QueryBuilder { + if alias == "" { + alias = "sum_" + column + } + return q.addAgg("sum", column, alias) +} + +// Avg adds an AVG aggregation. +func (q *QueryBuilder) Avg(column, alias string) *QueryBuilder { + if alias == "" { + alias = "avg_" + column + } + return q.addAgg("avg", column, alias) +} + +// Min adds a MIN aggregation. +func (q *QueryBuilder) Min(column, alias string) *QueryBuilder { + if alias == "" { + alias = "min_" + column + } + return q.addAgg("min", column, alias) +} + +// Max adds a MAX aggregation. +func (q *QueryBuilder) Max(column, alias string) *QueryBuilder { + if alias == "" { + alias = "max_" + column + } + return q.addAgg("max", column, alias) +} + +// CountDistinct adds a COUNT DISTINCT aggregation. +func (q *QueryBuilder) CountDistinct(column, alias string) *QueryBuilder { + if alias == "" { + alias = "count_distinct_" + column + } + return q.addAgg("countDistinct", column, alias) +} + +// Aggregate adds a custom aggregation function. +func (q *QueryBuilder) Aggregate(fn, column, alias string) *QueryBuilder { + return q.addAgg(fn, column, alias) +} + +// GroupBy appends columns to the GROUP BY clause. +func (q *QueryBuilder) GroupBy(columns ...string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.groupBy = append(s.groupBy, columns...) + }) +} + +// OrderBy appends an ORDER BY clause. dir defaults to "asc". +func (q *QueryBuilder) OrderBy(column, dir string) *QueryBuilder { + if dir == "" { + dir = "asc" + } + return q.clone(func(s *queryState) { + s.orderBy = append(s.orderBy, OrderClause{Column: column, Dir: dir}) + }) +} + +// Limit sets the maximum number of rows to return. +func (q *QueryBuilder) Limit(n int) *QueryBuilder { + return q.clone(func(s *queryState) { + s.limit = &n + }) +} + +// TimeRange filters by a time window. since and until accept RFC3339 timestamps +// or relative durations ("1h", "30m", "7d", "2w"). +func (q *QueryBuilder) TimeRange(column, since, until string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.timeRange = &TimeRange{Column: column, Since: since, Until: until} + }) +} + +// CacheTTL records a desired result-cache TTL. Currently client-side only — +// the server derives TTLs adaptively (#280). +func (q *QueryBuilder) CacheTTL(seconds int) *QueryBuilder { + return q.clone(func(s *queryState) { + s.cacheTTL = &seconds + }) +} + +// FetchTyped executes the query and decodes rows into []T. +func FetchTyped[Row any](ctx context.Context, q *QueryBuilder) (*Page[Row], error) { + limit := DefaultLimit + if q.state.limit != nil { + limit = *q.state.limit + } + ast := q.buildAST(limit) + + var rows []Row + if err := doRequest(q.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/query", + params: url.Values{"table": {q.state.table}}, + body: ast, + }, &rows); err != nil { + return nil, err + } + + hasMore := limit > 0 && len(rows) >= limit + page := &Page[Row]{Data: rows, HasMore: hasMore} + + // Attach Next whenever we have an order column to build a cursor from. + // This doesn't check that the order column is present in the row + // projection — a Select() that omits it means fetchNextTyped can't find + // a cursor value and will quietly return an empty page (matches the TS + // SDK's QueryBuilder.fetch()/_fetchNext(), which has the same limitation). + if hasMore && len(q.state.orderBy) > 0 { + page.Next = func(ctx context.Context) (*Page[Row], error) { + return fetchNextTyped[Row](ctx, q, rows, limit) + } + } + + return page, nil +} + +// FetchUntyped executes the query and returns rows as []map[string]any. +func (q *QueryBuilder) FetchUntyped(ctx context.Context) (*Page[map[string]any], error) { + return FetchTyped[map[string]any](ctx, q) +} + +// Stream opens a live SSE event stream for this query's table. +// Filters and column projections are applied client-side. +func (q *QueryBuilder) Stream(opts *StreamOptions) *StreamController { + raw := q.createStream(q.state.table, opts) + if len(q.state.filters) == 0 && len(q.state.columns) == 0 { + return raw + } + return newFilteredStreamController(raw, q.state.filters, q.state.columns) +} + +// LiveQuery starts a live query: fetches historical data, then streams live +// updates. The subscriber's Initial is called once, then Next fires for each +// live event. Returns a LiveQuery handle with a Close method. +func (q *QueryBuilder) LiveQuery(sub *StreamSubscriber, opts *StreamOptions) *LiveQueryHandle { + stream := q.Stream(opts) + fetchFn := func(ctx context.Context) ([]map[string]any, error) { + page, err := q.FetchUntyped(ctx) + if err != nil { + return nil, err + } + return page.Data, nil + } + return newLiveQuery(stream, fetchFn, sub, q.state.filters) +} + +func (q *QueryBuilder) addAgg(fn, column, alias string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.aggregations = append(s.aggregations, Aggregation{Fn: fn, Column: column, Alias: alias}) + }) +} + +func (q *QueryBuilder) buildAST(effectiveLimit int) *StructuredQuery { + ast := &StructuredQuery{} + hasColumns := len(q.state.columns) > 0 + hasAggs := len(q.state.aggregations) > 0 + + // Projection: explicit select_all, then explicit columns, else — for a bare + // query with no projection and no aggregations — default to select_all so + // from(t).fetch() returns rows. + if q.state.selectAll { + ast.SelectAll = true + } else if hasColumns { + ast.Columns = q.state.columns + } else if !hasAggs { + ast.SelectAll = true + } + + if hasAggs { + ast.Aggregations = q.state.aggregations + } + if len(q.state.filters) > 0 { + ast.Filters = q.state.filters + } + if len(q.state.groupBy) > 0 { + ast.GroupBy = q.state.groupBy + } + if len(q.state.orderBy) > 0 { + ast.OrderBy = q.state.orderBy + } + ast.Limit = &effectiveLimit + if q.state.timeRange != nil { + ast.TimeRange = q.state.timeRange + } + return ast +} + +func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Row, limit int) (*Page[Row], error) { + if len(q.state.orderBy) == 0 { + return &Page[Row]{}, nil + } + cursor := q.state.orderBy[0] + + // Extract the last row's value for the cursor column. + lastRow := any(prevRows[len(prevRows)-1]) + m, ok := lastRow.(map[string]any) + if !ok { + // ponytail: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. + raw, _ := json.Marshal(lastRow) + m = make(map[string]any) + _ = json.Unmarshal(raw, &m) + } + lastValue, exists := m[cursor.Column] + if !exists { + // Cursor column wasn't in the projection (e.g. Select() omitted it) — + // no cursor value to page from, so end pagination quietly rather than + // erroring. Matches the TS SDK's _fetchNext(). + return &Page[Row]{}, nil + } + + cursorOp := "gt" + if cursor.Dir == "desc" { + cursorOp = "lt" + } + + next := q.clone(func(s *queryState) { + s.filters = append(s.filters, QueryFilter{ + Column: cursor.Column, + Op: cursorOp, + Value: lastValue, + }) + }) + return FetchTyped[Row](ctx, next) +} diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go new file mode 100644 index 00000000..b2f560bb --- /dev/null +++ b/clients/go/query_builder_test.go @@ -0,0 +1,259 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" +) + +func queryTestCtx(handler http.Handler) (*Client, *httptest.Server) { + srv := httptest.NewServer(handler) + c := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{MaxRetries: 0}, + }) + return c, srv +} + +func captureQueryBody(t *testing.T, handler http.Handler) (*Client, func() map[string]any) { + t.Helper() + var body []byte + wrapper := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw := make([]byte, 32*1024) + n, _ := r.Body.Read(raw) + body = raw[:n] + handler.ServeHTTP(w, r) + }) + c, _ := queryTestCtx(wrapper) + return c, func() map[string]any { + var m map[string]any + json.Unmarshal(body, &m) + return m + } +} + +var emptyRows = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + json.NewEncoder(w).Encode([]map[string]any{{"page": "/home"}}) +}) + +func TestQueryBuilder_Immutability(t *testing.T) { + c, _ := queryTestCtx(emptyRows) + b1 := c.From("clicks").Select("page") + b2 := b1.Where("score", OpGt, 10) + if b1 == b2 { + t.Fatal("builder should be immutable — chain methods return new instances") + } +} + +func TestQueryBuilder_SelectColumns(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page", "button").FetchUntyped(context.Background()) + + body := getBody() + cols, ok := body["columns"].([]any) + if !ok || len(cols) != 2 { + t.Fatalf("want [page, button], got %v", body["columns"]) + } +} + +func TestQueryBuilder_SelectAll(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").SelectAll().FetchUntyped(context.Background()) + + body := getBody() + if body["select_all"] != true { + t.Fatalf("want select_all=true, got %v", body) + } +} + +func TestQueryBuilder_BareQueryDefaultsToSelectAll(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select().FetchUntyped(context.Background()) + + body := getBody() + if body["select_all"] != true { + t.Fatalf("bare query should default to select_all, got %v", body) + } +} + +func TestQueryBuilder_AggregationOnlyNoSelectAll(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select().Count("*", "n").FetchUntyped(context.Background()) + + body := getBody() + if body["select_all"] != nil { + t.Fatalf("aggregation-only query should not set select_all, got %v", body) + } +} + +func TestQueryBuilder_Where(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").Where("score", OpGt, 10).FetchUntyped(context.Background()) + + body := getBody() + filters, ok := body["filters"].([]any) + if !ok || len(filters) != 1 { + t.Fatalf("want 1 filter, got %v", body["filters"]) + } + f := filters[0].(map[string]any) + if f["column"] != "score" || f["op"] != "gt" { + t.Fatalf("want score/gt filter, got %v", f) + } +} + +func TestQueryBuilder_AllOperators(t *testing.T) { + ops := []struct { + sdk FilterOp + wire string + }{ + {OpEq, "eq"}, + {OpNeq, "neq"}, + {OpGt, "gt"}, + {OpGte, "gte"}, + {OpLt, "lt"}, + {OpLte, "lte"}, + {OpIn, "in"}, + {OpLike, "like"}, + {OpNotLike, "not_like"}, + } + for _, tt := range ops { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("x").Where("col", tt.sdk, "v").FetchUntyped(context.Background()) + body := getBody() + filters := body["filters"].([]any) + f := filters[0].(map[string]any) + if f["op"] != tt.wire { + t.Errorf("%s: want wire op %s, got %s", tt.sdk, tt.wire, f["op"]) + } + } +} + +func TestQueryBuilder_Aggregations(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select(). + Count("*", "total"). + Sum("score", ""). + Avg("score", ""). + Min("score", ""). + Max("score", ""). + CountDistinct("page", ""). + Aggregate("uniqExact", "user_id", "unique_users"). + FetchUntyped(context.Background()) + + body := getBody() + aggs, ok := body["aggregations"].([]any) + if !ok || len(aggs) != 7 { + t.Fatalf("want 7 aggregations, got %v", body["aggregations"]) + } +} + +func TestQueryBuilder_GroupBy(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").GroupBy("page").FetchUntyped(context.Background()) + + body := getBody() + gb, ok := body["group_by"].([]any) + if !ok || len(gb) != 1 || gb[0] != "page" { + t.Fatalf("want [page], got %v", body["group_by"]) + } +} + +func TestQueryBuilder_OrderBy(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").OrderBy("page", "desc").FetchUntyped(context.Background()) + + body := getBody() + ob := body["order_by"].([]any) + o := ob[0].(map[string]any) + if o["column"] != "page" || o["dir"] != "desc" { + t.Fatalf("want page/desc, got %v", o) + } +} + +func TestQueryBuilder_Limit(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").Limit(50).FetchUntyped(context.Background()) + + body := getBody() + if body["limit"] != float64(50) { + t.Fatalf("want 50, got %v", body["limit"]) + } +} + +func TestQueryBuilder_TimeRange(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page"). + TimeRange("received_timestamp", "1h", ""). + FetchUntyped(context.Background()) + + body := getBody() + tr := body["time_range"].(map[string]any) + if tr["column"] != "received_timestamp" || tr["since"] != "1h" { + t.Fatalf("want received_timestamp/1h, got %v", tr) + } +} + +func TestQueryBuilder_Pagination_HasMore(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) + + page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if !page.HasMore { + t.Fatal("want hasMore=true") + } + if page.Next == nil { + t.Fatal("want next function") + } +} + +func TestQueryBuilder_Pagination_NoOrderNoNext(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) + + page, err := c.From("clicks").Select("id").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if !page.HasMore { + t.Fatal("want hasMore=true") + } + if page.Next != nil { + t.Fatal("want nil next — no order column for cursor") + } +} + +func TestQueryBuilder_ComplexQuery(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks"). + Select("page"). + Where("score", OpGt, 10). + Count("*", "total"). + GroupBy("page"). + OrderBy("total", "desc"). + Limit(50). + TimeRange("received_timestamp", "1h", ""). + CacheTTL(60). + FetchUntyped(context.Background()) + + body := getBody() + if body["columns"].([]any)[0] != "page" { + t.Fatal("missing page column") + } + if body["limit"] != float64(50) { + t.Fatal("wrong limit") + } + if body["group_by"].([]any)[0] != "page" { + t.Fatal("wrong group_by") + } +} diff --git a/clients/go/schema.go b/clients/go/schema.go new file mode 100644 index 00000000..67addc5b --- /dev/null +++ b/clients/go/schema.go @@ -0,0 +1,33 @@ +package wavehouse + +import "context" + +// SchemaNamespace provides admin-only schema introspection. +type SchemaNamespace struct { + ctx httpContext +} + +// List returns all table schemas discovered from ClickHouse. Admin-only. +func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { + // The backend returns []TableSchema; transform to map[string]TableSchema. + var raw []TableSchema + if err := doRequest(s.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/schema", + }, &raw); err != nil { + return nil, err + } + schemas := make(Schemas, len(raw)) + for _, t := range raw { + schemas[t.Name] = t + } + return schemas, nil +} + +// Refresh forces a schema re-discovery from ClickHouse. Admin-only. +func (s *SchemaNamespace) Refresh(ctx context.Context) error { + return doRequest(s.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/schema/refresh", + }, nil) +} diff --git a/clients/go/stream.go b/clients/go/stream.go new file mode 100644 index 00000000..e30d6346 --- /dev/null +++ b/clients/go/stream.go @@ -0,0 +1,561 @@ +package wavehouse + +import ( + "bufio" + "context" + "encoding/json" + "fmt" + "log" + "net/http" + "net/url" + "reflect" + "regexp" + "strings" + "sync" + "time" +) + +// StreamController manages a live SSE event stream. Use Subscribe for +// callback-based consumption or Events for channel-based consumption. +type StreamController struct { + mu sync.Mutex + status StreamStatus + subscribers []*StreamSubscriber + eventCh chan StreamEvent // ponytail: single buffered channel for Go-native consumption + cancel context.CancelFunc + done chan struct{} + closed bool +} + +// newStreamController opens an SSE connection for the given table. +func newStreamController(hctx httpContext, table string, opts *StreamOptions) *StreamController { + ctx, cancel := context.WithCancel(context.Background()) + sc := &StreamController{ + status: StatusConnecting, + eventCh: make(chan StreamEvent, 256), + cancel: cancel, + done: make(chan struct{}), + } + go sc.run(ctx, hctx, table, opts) + return sc +} + +// Status returns the current connection status. +func (sc *StreamController) Status() StreamStatus { + sc.mu.Lock() + defer sc.mu.Unlock() + return sc.status +} + +// Subscribe registers callbacks for stream events. Returns an unsubscribe +// function. The subscriber's Status callback fires immediately with the +// current status. +func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { + sc.mu.Lock() + sc.subscribers = append(sc.subscribers, sub) + currentStatus := sc.status + sc.mu.Unlock() + + // Benign race: if setStatus fires between the unlock above and the + // callback below, the subscriber may see a stale status here. This is + // harmless because setStatus also invokes the subscriber's callback, + // so the subscriber will receive the up-to-date status immediately + // after. Matches the TS SDK's registration behavior. + if sub.Status != nil { + sub.Status(currentStatus) + } + + return func() { + sc.mu.Lock() + defer sc.mu.Unlock() + for i, s := range sc.subscribers { + if s == sub { + sc.subscribers = append(sc.subscribers[:i], sc.subscribers[i+1:]...) + break + } + } + } +} + +// Events returns a read-only channel that receives stream events. +// The channel is closed when the stream closes. +func (sc *StreamController) Events() <-chan StreamEvent { + return sc.eventCh +} + +// Connected blocks until the stream reaches "live" status or the context +// expires. Returns an error if the stream closes before connecting. +func (sc *StreamController) Connected(ctx context.Context) error { + sc.mu.Lock() + if sc.status == StatusLive { + sc.mu.Unlock() + return nil + } + if sc.status == StatusClosed || sc.closed { + sc.mu.Unlock() + return fmt.Errorf("stream is closed") + } + sc.mu.Unlock() + + // Poll — simple and correct. + // ponytail: condition variable if polling shows up in profiles. + ticker := time.NewTicker(50 * time.Millisecond) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-sc.done: + return fmt.Errorf("stream closed before connecting") + case <-ticker.C: + sc.mu.Lock() + s := sc.status + sc.mu.Unlock() + if s == StatusLive { + return nil + } + if s == StatusClosed { + return fmt.Errorf("stream closed before connecting") + } + } + } +} + +// Close shuts down the stream and releases resources. Non-blocking so it is +// safe to call from subscriber callbacks (which run on the stream goroutine). +func (sc *StreamController) Close() { + sc.mu.Lock() + if sc.closed { + sc.mu.Unlock() + return + } + sc.closed = true + sc.mu.Unlock() + + sc.cancel() + // Don't block on <-sc.done: callbacks execute on the stream goroutine, + // so waiting here would deadlock if Close is called from a callback. +} + +func (sc *StreamController) setStatus(s StreamStatus) { + sc.mu.Lock() + if s == sc.status { + sc.mu.Unlock() + return + } + sc.status = s + subs := append([]*StreamSubscriber(nil), sc.subscribers...) + sc.mu.Unlock() + + for _, sub := range subs { + if sub.Status != nil { + sub.Status(s) + } + } +} + +func (sc *StreamController) emitEvent(event StreamEvent) { + sc.mu.Lock() + subs := append([]*StreamSubscriber(nil), sc.subscribers...) + sc.mu.Unlock() + + for _, sub := range subs { + if sub.Next != nil { + sub.Next(event) + } + } + + // Non-blocking send to the channel. + select { + case sc.eventCh <- event: + default: + log.Printf("[wavehouse] stream event dropped: channel buffer full") + } +} + +func (sc *StreamController) emitError(err error) { + sc.mu.Lock() + subs := append([]*StreamSubscriber(nil), sc.subscribers...) + sc.mu.Unlock() + + for _, sub := range subs { + if sub.Error != nil { + sub.Error(err) + } + } +} + +// run is the SSE connection loop with reconnect/backoff. +func (sc *StreamController) run(ctx context.Context, hctx httpContext, table string, opts *StreamOptions) { + defer func() { + sc.setStatus(StatusClosed) + close(sc.eventCh) + close(sc.done) + }() + + since := "" + if opts != nil { + since = opts.Since + } + + attempt := 0 + for { + if ctx.Err() != nil { + return + } + + lastID, err := sc.connect(ctx, hctx, table, since) + // Persist the last event ID so the next reconnect resumes from it. + if lastID != "" { + since = lastID + } + if ctx.Err() != nil { + return + } + + if err != nil { + sc.emitError(&Error{ + Status: 0, + Code: "SSE_ERROR", + Message: err.Error(), + Retryable: true, + }) + } + + sc.setStatus(StatusReconnecting) + delay := backoff(attempt) + attempt++ + + select { + case <-ctx.Done(): + return + case <-time.After(delay): + } + } +} + +// connect opens a single SSE connection and reads events until it closes. +// Returns the last seen event ID (empty if none) and any error. +func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table, since string) (string, error) { + u, err := url.Parse(hctx.baseURL + "/v1/stream") + if err != nil { + return "", err + } + q := u.Query() + q.Set("table", table) + if since != "" { + q.Set("since", since) + } + + // Auth: Go SDK uses Authorization header (not ?token= like browser EventSource). + var authHeader string + if hctx.auth != nil { + token, err := hctx.auth(ctx) + if err != nil { + return "", fmt.Errorf("auth: %w", err) + } + if token != "" { + authHeader = "Bearer " + token + } + } + + u.RawQuery = q.Encode() + + req, err := http.NewRequestWithContext(ctx, "GET", u.String(), nil) + if err != nil { + return "", err + } + req.Header.Set("Accept", "text/event-stream") + req.Header.Set("Cache-Control", "no-cache") + if authHeader != "" { + req.Header.Set("Authorization", authHeader) + } + + resp, err := hctx.httpClient.Do(req) + if err != nil { + return "", err + } + defer resp.Body.Close() + + if resp.StatusCode != http.StatusOK { + return "", fmt.Errorf("SSE connect failed: HTTP %d", resp.StatusCode) + } + + sc.setStatus(StatusLive) + + // Parse SSE frames. + scanner := bufio.NewScanner(resp.Body) + scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) // 16 MiB max, matching server ingest cap + var eventID, dataLine string + lastID := since + + for scanner.Scan() { + if ctx.Err() != nil { + return lastID, nil + } + + line := scanner.Text() + + if line == "" { + // Empty line = end of event frame. + if dataLine != "" { + sc.handleSSEData(dataLine, eventID) + // Track last event ID for reconnect gap-fill. + if eventID != "" { + lastID = eventID + } + } + eventID = "" + dataLine = "" + continue + } + + if strings.HasPrefix(line, ":") { + // Comment (keepalive or connected). Skip. + continue + } + + if strings.HasPrefix(line, "id:") { + eventID = strings.TrimSpace(strings.TrimPrefix(line, "id:")) + } else if strings.HasPrefix(line, "data:") { + trimmed := strings.TrimSpace(strings.TrimPrefix(line, "data:")) + if dataLine == "" { + dataLine = trimmed + } else { + dataLine = dataLine + "\n" + trimmed + } + } + } + + return lastID, scanner.Err() +} + +// sseMessage matches the server's SSE event JSON shape. +type sseMessage struct { + TableName string `json:"table_name"` + ReceivedTimestamp string `json:"received_timestamp"` + Data map[string]any `json:"data"` +} + +func (sc *StreamController) handleSSEData(data, eventID string) { + var msg sseMessage + if err := json.Unmarshal([]byte(data), &msg); err != nil { + log.Printf("[wavehouse] SSE received malformed message: %s", data) + return + } + + event := StreamEvent{ + Table: msg.TableName, + Timestamp: msg.ReceivedTimestamp, + Data: msg.Data, + } + sc.emitEvent(event) +} + +// newFilteredStreamController wraps a StreamController with client-side +// filtering and column projection. +func newFilteredStreamController(inner *StreamController, filters []QueryFilter, columns []string) *StreamController { + ctx, cancel := context.WithCancel(context.Background()) + sc := &StreamController{ + status: inner.Status(), + eventCh: make(chan StreamEvent, 256), + cancel: cancel, + done: make(chan struct{}), + } + + go func() { + defer func() { + sc.setStatus(StatusClosed) + close(sc.eventCh) + close(sc.done) + }() + + inner.Subscribe(&StreamSubscriber{ + Next: func(event StreamEvent) { + if !matchesFilters(event.Data, filters) { + return + } + if len(columns) > 0 { + event.Data = projectColumns(event.Data, columns) + } + sc.emitEvent(event) + }, + Status: func(s StreamStatus) { + sc.setStatus(s) + }, + Error: func(err error) { + sc.emitError(err) + }, + }) + + select { + case <-ctx.Done(): + inner.Close() + case <-inner.done: + } + }() + + return sc +} + +// matchesFilters evaluates all filters against a data row (AND). +func matchesFilters(row map[string]any, filters []QueryFilter) bool { + for _, f := range filters { + val := row[f.Column] + if !evaluateFilter(val, f.Op, f.Value) { + return false + } + } + return true +} + +func evaluateFilter(actual any, op string, expected any) bool { + switch op { + case "eq": + return equalValues(actual, expected) + case "neq": + return !equalValues(actual, expected) + case "gt": + c, ok := compareOrdered(actual, expected) + return ok && c > 0 + case "gte": + c, ok := compareOrdered(actual, expected) + return ok && c >= 0 + case "lt": + c, ok := compareOrdered(actual, expected) + return ok && c < 0 + case "lte": + c, ok := compareOrdered(actual, expected) + return ok && c <= 0 + case "in": + return evaluateIn(actual, expected) + case "like": + aStr, aOK := actual.(string) + eStr, eOK := expected.(string) + if !aOK || !eOK { + return false + } + return matchLike(aStr, eStr) + case "not_like": + aStr, aOK := actual.(string) + eStr, eOK := expected.(string) + if !aOK || !eOK { + return false + } + return !matchLike(aStr, eStr) + default: + return false + } +} + +// equalValues compares two values for equality, normalizing numeric types +// (JSON decodes numbers as float64, but callers may pass int). +func equalValues(a, b any) bool { + if af, aOK := toFloat64(a); aOK { + if bf, bOK := toFloat64(b); bOK { + return af == bf + } + } + // fmt.Sprint is safe for all types (no panic on maps/slices). + return fmt.Sprint(a) == fmt.Sprint(b) +} + +// evaluateIn checks whether actual is contained in the expected slice. +// Handles both []any and typed slices (e.g., []string, []int). +func evaluateIn(actual, expected any) bool { + if arr, ok := expected.([]any); ok { + for _, v := range arr { + if equalValues(actual, v) { + return true + } + } + return false + } + // Handle typed slices via reflection. + rv := reflect.ValueOf(expected) + if rv.Kind() == reflect.Slice { + for i := range rv.Len() { + if equalValues(actual, rv.Index(i).Interface()) { + return true + } + } + } + return false +} + +var likeRegexCache sync.Map // pattern string → *regexp.Regexp + +// matchLike converts a SQL LIKE pattern to a regex and tests it +// (case-insensitive, matching the TS SDK). +func matchLike(actual, pattern string) bool { + if cached, ok := likeRegexCache.Load(pattern); ok { + return cached.(*regexp.Regexp).MatchString(actual) + } + escaped := regexp.QuoteMeta(pattern) + escaped = strings.ReplaceAll(escaped, "%", ".*") + escaped = strings.ReplaceAll(escaped, "_", ".") + re, err := regexp.Compile("(?i)^" + escaped + "$") + if err != nil { + return false + } + likeRegexCache.Store(pattern, re) + return re.MatchString(actual) +} + +// compareOrdered returns (-1, 0, or 1) and true for comparable ordered types, +// or (0, false) when the types cannot be compared. +func compareOrdered(actual, expected any) (int, bool) { + if a, aOK := toFloat64(actual); aOK { + if b, bOK := toFloat64(expected); bOK { + switch { + case a < b: + return -1, true + case a > b: + return 1, true + default: + return 0, true + } + } + } + if aStr, ok := actual.(string); ok { + if bStr, ok := expected.(string); ok { + switch { + case aStr < bStr: + return -1, true + case aStr > bStr: + return 1, true + default: + return 0, true + } + } + } + return 0, false +} + +func toFloat64(v any) (float64, bool) { + switch n := v.(type) { + case float64: + return n, true + case float32: + return float64(n), true + case int: + return float64(n), true + case int64: + return float64(n), true + case json.Number: + f, err := n.Float64() + return f, err == nil + default: + return 0, false + } +} + +func projectColumns(row map[string]any, columns []string) map[string]any { + result := make(map[string]any, len(columns)) + for _, col := range columns { + if v, ok := row[col]; ok { + result[col] = v + } + } + return result +} diff --git a/clients/go/sys.go b/clients/go/sys.go new file mode 100644 index 00000000..4d2af556 --- /dev/null +++ b/clients/go/sys.go @@ -0,0 +1,17 @@ +package wavehouse + +import "context" + +// SysNamespace provides system health checks. +type SysNamespace struct { + ctx httpContext +} + +// Health pings the server's public /v1/health endpoint. Returns nil when the +// server is reachable and past boot, or an error describing the failure. +func (s *SysNamespace) Health(ctx context.Context) error { + return doRequest(s.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/health", + }, nil) +} diff --git a/clients/go/table.go b/clients/go/table.go new file mode 100644 index 00000000..adc8d829 --- /dev/null +++ b/clients/go/table.go @@ -0,0 +1,205 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/url" + "reflect" + "strings" +) + +// TableRef is a reference to a table. Use it for queries, inserts, schema, and +// streams. NOT safe to use concurrently from multiple goroutines for mutations; +// reads (Fetch, Select, etc.) are safe. +type TableRef struct { + ctx httpContext + table string + createStream func(table string, opts *StreamOptions) *StreamController +} + +// Fetch is a SELECT * shortcut with a default limit of 1000. +func (t *TableRef) Fetch(ctx context.Context) (*Page[map[string]any], error) { + return t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx) +} + +// Select starts building a typed query with the given column projection. +func (t *TableRef) Select(columns ...string) *QueryBuilder { + return &QueryBuilder{ + ctx: t.ctx, + createStream: t.createStream, + state: queryState{ + table: t.table, + columns: columns, + }, + } +} + +// SelectAll starts a query that selects every column the caller's role may read. +func (t *TableRef) SelectAll() *QueryBuilder { + return t.Select().SelectAll() +} + +// Insert inserts one or more rows into this table. A single map or struct is +// sent as JSON; any slice — []map[string]any, a generated/user-defined row +// type such as []ClickRow, etc. — is serialized to NDJSON for batch ingest. +func (t *TableRef) Insert(ctx context.Context, data any) (*InsertResult, error) { + if rows, ok := data.([]map[string]any); ok { + return t.insertBatch(ctx, rows) + } + if rv, ok := sliceValue(data); ok { + return t.insertBatchReflect(ctx, rv) + } + return t.insertSingle(ctx, data) +} + +// sliceValue reports whether data is a slice type, returning its +// reflect.Value for iteration. []byte is excluded and treated as an opaque +// single value (matching encoding/json's special-cased handling of byte +// slices) rather than a batch of numbers. +func sliceValue(data any) (reflect.Value, bool) { + if data == nil { + return reflect.Value{}, false + } + if _, isBytes := data.([]byte); isBytes { + return reflect.Value{}, false + } + v := reflect.ValueOf(data) + if v.Kind() != reflect.Slice { + return reflect.Value{}, false + } + return v, true +} + +// InsertNDJSON inserts pre-formatted NDJSON (one record per line). +func (t *TableRef) InsertNDJSON(ctx context.Context, ndjson string) (*InsertResult, error) { + return t.sendNDJSON(ctx, ndjson) +} + +// Schema returns the table's column definitions from ClickHouse. Admin-only. +func (t *TableRef) Schema(ctx context.Context) (*TableSchema, error) { + var schema TableSchema + if err := doRequest(t.ctx, ctx, requestOptions{ + method: "GET", + path: "/v1/schema", + params: url.Values{"table": {t.table}}, + }, &schema); err != nil { + return nil, err + } + return &schema, nil +} + +// Stream opens a live SSE event stream for this table. +func (t *TableRef) Stream(opts *StreamOptions) *StreamController { + return t.createStream(t.table, opts) +} + +func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, error) { + var res struct { + OK *bool `json:"ok"` + Duplicate *bool `json:"duplicate"` + } + if err := doRequest(t.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/ingest", + params: url.Values{"table": {t.table}}, + body: data, + }, &res); err != nil { + return nil, err + } + ok := true + if res.OK != nil { + ok = *res.OK + } + result := &InsertResult{OK: ok} + if res.Duplicate != nil { + result.Duplicate = res.Duplicate + } + return result, nil +} + +func (t *TableRef) insertBatch(ctx context.Context, rows []map[string]any) (*InsertResult, error) { + if len(rows) == 0 { + zero := 0 + return &InsertResult{ + OK: true, + Total: &zero, + Succeeded: &zero, + Failed: &zero, + Duplicates: &zero, + }, nil + } + var sb strings.Builder + for i, row := range rows { + if i > 0 { + sb.WriteByte('\n') + } + raw, err := json.Marshal(row) + if err != nil { + return nil, err + } + sb.Write(raw) + } + return t.sendNDJSON(ctx, sb.String()) +} + +// insertBatchReflect is the fallback batch path for any slice type other +// than []map[string]any (the fast path in insertBatch above) — e.g. a +// generated or user-defined row type such as []ClickRow. Each element is +// marshaled to JSON individually and joined as NDJSON, exactly like +// insertBatch, so the server's per-record batch summary (failed, results, +// etc.) is preserved instead of being silently dropped by insertSingle. +func (t *TableRef) insertBatchReflect(ctx context.Context, rows reflect.Value) (*InsertResult, error) { + n := rows.Len() + if n == 0 { + zero := 0 + return &InsertResult{ + OK: true, + Total: &zero, + Succeeded: &zero, + Failed: &zero, + Duplicates: &zero, + }, nil + } + var sb strings.Builder + for i := 0; i < n; i++ { + if i > 0 { + sb.WriteByte('\n') + } + raw, err := json.Marshal(rows.Index(i).Interface()) + if err != nil { + return nil, err + } + sb.Write(raw) + } + return t.sendNDJSON(ctx, sb.String()) +} + +func (t *TableRef) sendNDJSON(ctx context.Context, ndjson string) (*InsertResult, error) { + var res struct { + Total int `json:"total"` + Succeeded int `json:"succeeded"` + Failed int `json:"failed"` + Duplicates int `json:"duplicates"` + Results []InsertRecordResult `json:"results"` + } + if err := doRequest(t.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/ingest", + params: url.Values{"table": {t.table}}, + rawBody: ndjson, + contentType: "application/x-ndjson", + }, &res); err != nil { + return nil, err + } + result := &InsertResult{ + OK: res.Failed == 0, + Total: &res.Total, + Succeeded: &res.Succeeded, + Failed: &res.Failed, + Duplicates: &res.Duplicates, + } + if len(res.Results) > 0 { + result.Results = res.Results + } + return result, nil +} diff --git a/clients/go/table_test.go b/clients/go/table_test.go new file mode 100644 index 00000000..39bb1efa --- /dev/null +++ b/clients/go/table_test.go @@ -0,0 +1,209 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "testing" +) + +func TestTableRef_InsertSingle(t *testing.T) { + var gotBody map[string]any + var gotPath string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotPath = r.URL.Path + json.NewDecoder(r.Body).Decode(&gotBody) + json.NewEncoder(w).Encode(map[string]any{"ok": true}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/home"}) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + if gotPath != "/v1/ingest" { + t.Fatalf("want /v1/ingest, got %s", gotPath) + } + if gotBody["page"] != "/home" { + t.Fatalf("want page=/home, got %v", gotBody) + } +} + +func TestTableRef_InsertBatch(t *testing.T) { + var gotCT string + var gotBody string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotCT = r.Header.Get("Content-Type") + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) + json.NewEncoder(w).Encode(map[string]any{ + "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, + }) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), []map[string]any{ + {"page": "/a"}, + {"page": "/b"}, + }) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + if gotCT != "application/x-ndjson" { + t.Fatalf("want ndjson content type, got %s", gotCT) + } + if gotBody != `{"page":"/a"}`+"\n"+`{"page":"/b"}` { + t.Fatalf("want NDJSON body, got %s", gotBody) + } +} + +// TestTableRef_InsertTypedSlice covers the P1 finding: a typed slice (e.g. a +// generated or user-defined row type such as []ClickRow) must take the batch +// NDJSON path — not fall through to insertSingle, which would send the slice +// as a single JSON body and silently ignore any per-record failures the +// server reports. +func TestTableRef_InsertTypedSlice(t *testing.T) { + type ClickRow struct { + Page string `json:"page"` + } + + var gotCT string + var gotBody string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotCT = r.Header.Get("Content-Type") + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) + json.NewEncoder(w).Encode(map[string]any{ + "total": 2, "succeeded": 1, "failed": 1, "duplicates": 0, + }) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), []ClickRow{ + {Page: "/a"}, + {Page: "/b"}, + }) + if err != nil { + t.Fatal(err) + } + if gotCT != "application/x-ndjson" { + t.Fatalf("want ndjson content type, got %s", gotCT) + } + if gotBody != `{"page":"/a"}`+"\n"+`{"page":"/b"}` { + t.Fatalf("want NDJSON body, got %s", gotBody) + } + if result.OK { + t.Fatal("want ok=false when a batch record fails") + } + if result.Failed == nil || *result.Failed != 1 { + t.Fatalf("want failed=1, got %v", result.Failed) + } + if result.Total == nil || *result.Total != 2 { + t.Fatalf("want total=2, got %v", result.Total) + } +} + +// TestTableRef_InsertByteSliceNotBatch ensures []byte keeps going through +// insertSingle rather than being (mis)treated as a slice of per-byte rows. +func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { + var gotPath string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotPath = r.URL.Path + json.NewEncoder(w).Encode(map[string]any{"ok": true}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + if gotPath != "/v1/ingest" { + t.Fatalf("want /v1/ingest, got %s", gotPath) + } +} + +func TestTableRef_InsertEmptyBatch(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + t.Fatal("should not make a request for empty batch") + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), []map[string]any{}) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + if result.Total == nil || *result.Total != 0 { + t.Fatal("want total=0") + } +} + +func TestTableRef_InsertNDJSON(t *testing.T) { + var gotBody string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) + json.NewEncoder(w).Encode(map[string]any{ + "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, + }) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + ndjson := `{"page":"/a"}` + "\n" + `{"page":"/b"}` + result, err := c.From("clicks").InsertNDJSON(context.Background(), ndjson) + if err != nil { + t.Fatal(err) + } + if result.Total == nil || *result.Total != 2 { + t.Fatalf("want total=2, got %v", result.Total) + } + if gotBody != ndjson { + t.Fatalf("want raw NDJSON, got %s", gotBody) + } +} + +func TestTableRef_Schema(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Query().Get("table") != "clicks" { + t.Errorf("want table=clicks") + } + json.NewEncoder(w).Encode(TableSchema{ + Name: "clicks", + Columns: []Column{ + {Name: "page", Type: "String"}, + }, + }) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + schema, err := c.From("clicks").Schema(context.Background()) + if err != nil { + t.Fatal(err) + } + if schema.Name != "clicks" { + t.Fatalf("want clicks, got %s", schema.Name) + } + if len(schema.Columns) != 1 || schema.Columns[0].Name != "page" { + t.Fatalf("unexpected columns: %v", schema.Columns) + } +} + +func TestTableRef_InsertDuplicate(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) + })) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/dup"}) + if err != nil { + t.Fatal(err) + } + if result.Duplicate == nil || !*result.Duplicate { + t.Fatal("want duplicate=true") + } +} diff --git a/clients/go/testdata/wire_cases.json b/clients/go/testdata/wire_cases.json new file mode 100644 index 00000000..4ec8613d --- /dev/null +++ b/clients/go/testdata/wire_cases.json @@ -0,0 +1,547 @@ +[ + { + "name": "bare query defaults to select_all", + "endpoint": "query", + "table": "clicks", + "operations": [], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "select_all": true, + "limit": 1000 + } + }, + { + "name": "select explicit columns", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page", "button"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page", "button"], + "limit": 1000 + } + }, + { + "name": "selectAll sends select_all flag", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "selectAll" } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "select_all": true, + "limit": 1000 + } + }, + { + "name": "where with eq operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "=", "/home"] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "eq", "value": "/home" }], + "limit": 1000 + } + }, + { + "name": "where with neq operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "!=", "/home"] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "neq", "value": "/home" }], + "limit": 1000 + } + }, + { + "name": "where with gt operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">", 10] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "gt", "value": 10 }], + "limit": 1000 + } + }, + { + "name": "where with gte operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">=", 10] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "gte", "value": 10 }], + "limit": 1000 + } + }, + { + "name": "where with lt operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", "<", 5] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "lt", "value": 5 }], + "limit": 1000 + } + }, + { + "name": "where with lte operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", "<=", 5] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "lte", "value": 5 }], + "limit": 1000 + } + }, + { + "name": "where with in operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "in", ["/home", "/about"]] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "in", "value": ["/home", "/about"] }], + "limit": 1000 + } + }, + { + "name": "where with like operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "like", "/home%"] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "like", "value": "/home%" }], + "limit": 1000 + } + }, + { + "name": "count aggregation", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "count", "args": ["*", "total"] } + ], + "expected_body": { + "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], + "limit": 1000 + } + }, + { + "name": "sum aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "sum", "args": ["score", ""] } + ], + "expected_body": { + "aggregations": [{ "fn": "sum", "column": "score", "alias": "sum_score" }], + "limit": 1000 + } + }, + { + "name": "avg aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "avg", "args": ["score", ""] } + ], + "expected_body": { + "aggregations": [{ "fn": "avg", "column": "score", "alias": "avg_score" }], + "limit": 1000 + } + }, + { + "name": "min aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "min", "args": ["score", ""] } + ], + "expected_body": { + "aggregations": [{ "fn": "min", "column": "score", "alias": "min_score" }], + "limit": 1000 + } + }, + { + "name": "max aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "max", "args": ["score", ""] } + ], + "expected_body": { + "aggregations": [{ "fn": "max", "column": "score", "alias": "max_score" }], + "limit": 1000 + } + }, + { + "name": "countDistinct aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "countDistinct", "args": ["page", ""] } + ], + "expected_body": { + "aggregations": [{ "fn": "countDistinct", "column": "page", "alias": "count_distinct_page" }], + "limit": 1000 + } + }, + { + "name": "custom aggregate function", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "aggregate", "args": ["uniqExact", "user_id", "unique_users"] } + ], + "expected_body": { + "aggregations": [{ "fn": "uniqExact", "column": "user_id", "alias": "unique_users" }], + "limit": 1000 + } + }, + { + "name": "groupBy", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "groupBy", "args": ["page"] } + ], + "expected_body": { + "columns": ["page"], + "group_by": ["page"], + "limit": 1000 + } + }, + { + "name": "orderBy ascending (default)", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "orderBy", "args": ["page", "asc"] } + ], + "expected_body": { + "columns": ["page"], + "order_by": [{ "column": "page", "dir": "asc" }], + "limit": 1000 + } + }, + { + "name": "orderBy descending", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "orderBy", "args": ["score", "desc"] } + ], + "expected_body": { + "columns": ["page"], + "order_by": [{ "column": "score", "dir": "desc" }], + "limit": 1000 + } + }, + { + "name": "explicit limit", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "limit", "args": [50] } + ], + "expected_body": { + "columns": ["page"], + "limit": 50 + } + }, + { + "name": "timeRange with since only", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } + ], + "expected_body": { + "columns": ["page"], + "time_range": { "column": "received_timestamp", "since": "1h" }, + "limit": 1000 + } + }, + { + "name": "timeRange with since and until", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "timeRange", "args": ["ts", "2026-01-01", "2026-02-01"] } + ], + "expected_body": { + "columns": ["page"], + "time_range": { "column": "ts", "since": "2026-01-01", "until": "2026-02-01" }, + "limit": 1000 + } + }, + { + "name": "multiple where clauses (AND)", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">", 10] }, + { "method": "where", "args": ["page", "=", "/home"] } + ], + "expected_body": { + "columns": ["page"], + "filters": [ + { "column": "score", "op": "gt", "value": 10 }, + { "column": "page", "op": "eq", "value": "/home" } + ], + "limit": 1000 + } + }, + { + "name": "complex query combining everything", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">", 10] }, + { "method": "count", "args": ["*", "total"] }, + { "method": "groupBy", "args": ["page"] }, + { "method": "orderBy", "args": ["total", "desc"] }, + { "method": "limit", "args": [50] }, + { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } + ], + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "gt", "value": 10 }], + "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], + "group_by": ["page"], + "order_by": [{ "column": "total", "dir": "desc" }], + "limit": 50, + "time_range": { "column": "received_timestamp", "since": "1h" } + } + }, + { + "name": "insert single row path", + "endpoint": "ingest", + "table": "clicks", + "operations": [ + { "method": "insert", "args": [{ "page": "/home", "button": "cta" }] } + ], + "expected_path": "/v1/ingest?table=clicks", + "expected_method": "POST", + "expected_content_type": "application/json", + "expected_body": { "page": "/home", "button": "cta" } + }, + { + "name": "insert batch as NDJSON", + "endpoint": "ingest_batch", + "table": "clicks", + "operations": [ + { + "method": "insert", + "args": [[{ "page": "/a" }, { "page": "/b" }]] + } + ], + "expected_path": "/v1/ingest?table=clicks", + "expected_method": "POST", + "expected_content_type": "application/x-ndjson", + "expected_raw_body": "{\"page\":\"/a\"}\n{\"page\":\"/b\"}" + }, + { + "name": "pipe execution", + "endpoint": "pipe", + "pipe_name": "top_pages", + "pipe_params": { "limit": 10 }, + "expected_path": "/v1/pipes/top_pages", + "expected_method": "POST", + "expected_body": { "limit": 10 } + }, + { + "name": "pipe execution with no params sends empty object", + "endpoint": "pipe", + "pipe_name": "simple", + "pipe_params": null, + "expected_path": "/v1/pipes/simple", + "expected_method": "POST", + "expected_body": {} + }, + { + "name": "raw SQL", + "endpoint": "sql", + "sql": "SELECT count() FROM clicks", + "expected_path": "/v1/admin/query", + "expected_method": "POST", + "expected_body": { "sql": "SELECT count() FROM clicks" } + }, + { + "name": "health check", + "endpoint": "health", + "expected_path": "/v1/health", + "expected_method": "GET" + }, + { + "name": "schema list", + "endpoint": "schema_list", + "expected_path": "/v1/schema", + "expected_method": "GET" + }, + { + "name": "schema refresh", + "endpoint": "schema_refresh", + "expected_path": "/v1/schema/refresh", + "expected_method": "POST" + }, + { + "name": "policy get", + "endpoint": "policy_get", + "expected_path": "/v1/admin/policy", + "expected_method": "GET" + }, + { + "name": "table with special characters URL-encodes correctly", + "endpoint": "query", + "table": "my table", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "limit", "args": [10] } + ], + "expected_path": "/v1/query?table=my+table", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "limit": 10 + } + }, + { + "name": "DLQ list", + "endpoint": "dlq_list", + "expected_path": "/v1/dlq/stats", + "expected_method": "GET" + }, + { + "name": "DLQ table filter", + "endpoint": "dlq_table", + "table": "events", + "expected_path": "/v1/dlq/stats?table=events", + "expected_method": "GET" + }, + { + "name": "policy set", + "endpoint": "policy_set", + "policy_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + }, + "expected_path": "/v1/admin/policy", + "expected_method": "PUT", + "expected_content_type": "application/json", + "expected_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + } + }, + { + "name": "policy validate", + "endpoint": "policy_validate", + "policy_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + }, + "expected_path": "/v1/admin/policy/validate", + "expected_method": "POST", + "expected_content_type": "application/json", + "expected_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + } + }, + { + "name": "pipes list", + "endpoint": "pipes_list", + "expected_path": "/v1/admin/pipes", + "expected_method": "GET" + }, + { + "name": "pipes get", + "endpoint": "pipes_get", + "pipe_name": "my_pipe", + "expected_path": "/v1/admin/pipes/my_pipe", + "expected_method": "GET" + }, + { + "name": "pipes set", + "endpoint": "pipes_set", + "pipe_name": "my_pipe", + "pipe_def": { + "sql": "SELECT page, count() AS views FROM events GROUP BY page", + "parameters": [ + { "name": "limit", "type": "Int32", "default": 100 } + ], + "description": "Top pages by view count" + }, + "expected_path": "/v1/admin/pipes/my_pipe", + "expected_method": "PUT", + "expected_content_type": "application/json", + "expected_body": { + "sql": "SELECT page, count() AS views FROM events GROUP BY page", + "parameters": [ + { "name": "limit", "type": "Int32", "default": 100 } + ], + "description": "Top pages by view count" + } + }, + { + "name": "pipes delete", + "endpoint": "pipes_delete", + "pipe_name": "my_pipe", + "expected_path": "/v1/admin/pipes/my_pipe", + "expected_method": "DELETE" + } +] diff --git a/clients/go/types.go b/clients/go/types.go new file mode 100644 index 00000000..b59333fd --- /dev/null +++ b/clients/go/types.go @@ -0,0 +1,245 @@ +package wavehouse + +import "context" + +// ── Structured query AST (matches backend wire format) ──────────────────── + +// StructuredQuery is the wire format for POST /v1/query. +type StructuredQuery struct { + // Columns to project. A literal "*" is a column named "*", not a wildcard. + // Omitting columns (with no aggregations and no select_all) selects nothing. + Columns []string `json:"columns,omitempty"` + // SelectAll requests every column the caller's role may read. + // Mutually exclusive with a non-empty Columns list. + SelectAll bool `json:"select_all,omitempty"` + // Aggregations (count, sum, avg, etc.). + Aggregations []Aggregation `json:"aggregations,omitempty"` + // Filters (WHERE conditions, ANDed). + Filters []QueryFilter `json:"filters,omitempty"` + // GroupBy columns. + GroupBy []string `json:"group_by,omitempty"` + // OrderBy clauses. + OrderBy []OrderClause `json:"order_by,omitempty"` + // Limit caps the result set. + Limit *int `json:"limit,omitempty"` + // TimeRange filters by a time window. + TimeRange *TimeRange `json:"time_range,omitempty"` +} + +// Aggregation describes a single aggregation (e.g. count, sum). +type Aggregation struct { + Fn string `json:"fn"` + Column string `json:"column"` + Alias string `json:"alias"` +} + +// QueryFilter describes a single WHERE condition. +type QueryFilter struct { + Column string `json:"column"` + Op string `json:"op"` + Value any `json:"value"` +} + +// OrderClause describes a single ORDER BY clause. +type OrderClause struct { + Column string `json:"column"` + Dir string `json:"dir"` // "asc" or "desc" +} + +// TimeRange filters by a time window on a column. +type TimeRange struct { + Column string `json:"column"` + Since string `json:"since"` + Until string `json:"until,omitempty"` +} + +// FilterOp is an SDK-facing filter operator. +type FilterOp string + +const ( + OpEq FilterOp = "=" + OpNeq FilterOp = "!=" + OpGt FilterOp = ">" + OpGte FilterOp = ">=" + OpLt FilterOp = "<" + OpLte FilterOp = "<=" + OpIn FilterOp = "in" + OpLike FilterOp = "like" + OpNotLike FilterOp = "not_like" +) + +// opMap translates SDK operators to backend wire tokens. +var opMap = map[FilterOp]string{ + OpEq: "eq", + OpNeq: "neq", + OpGt: "gt", + OpGte: "gte", + OpLt: "lt", + OpLte: "lte", + OpIn: "in", + OpLike: "like", + OpNotLike: "not_like", +} + +// ── Schema types ────────────────────────────────────────────────────────── + +// Column describes a single column in a table schema. +type Column struct { + Name string `json:"name"` + Type string `json:"type"` + IsNullable bool `json:"is_nullable"` + HasDefault bool `json:"has_default"` +} + +// TableSchema describes a table's schema. +type TableSchema struct { + Name string `json:"name"` + Columns []Column `json:"columns"` +} + +// Schemas maps table names to their schemas. +type Schemas map[string]TableSchema + +// ── Insert result ───────────────────────────────────────────────────────── + +// InsertRecordResult is a per-record outcome from a batch insert. +type InsertRecordResult struct { + Index int `json:"index"` + OK *bool `json:"ok,omitempty"` + Duplicate *bool `json:"duplicate,omitempty"` + Error string `json:"error,omitempty"` +} + +// InsertResult is the outcome of an insert operation. +type InsertResult struct { + OK bool `json:"ok"` + Duplicate *bool `json:"duplicate,omitempty"` + Total *int `json:"total,omitempty"` + Succeeded *int `json:"succeeded,omitempty"` + Failed *int `json:"failed,omitempty"` + Duplicates *int `json:"duplicates,omitempty"` + Results []InsertRecordResult `json:"results,omitempty"` +} + +// ── DLQ types ───────────────────────────────────────────────────────────── + +// DLQStats describes dead-letter-queue statistics. +type DLQStats struct { + Tables map[string]int `json:"tables"` + Total int `json:"total"` +} + +// ── Pipe types ──────────────────────────────────────────────────────────── + +// Pipe describes a named query pipe definition. +type Pipe struct { + Name string `json:"name"` + SQL string `json:"sql"` + Parameters []ParamDef `json:"parameters,omitempty"` + Description string `json:"description,omitempty"` + AllowedRoles []string `json:"allowed_roles,omitempty"` +} + +// ParamDef describes a pipe parameter. +type ParamDef struct { + Name string `json:"name"` + Type string `json:"type"` + Required bool `json:"required,omitempty"` + Default any `json:"default,omitempty"` +} + +// ── Policy types ────────────────────────────────────────────────────────── + +// Policy describes the server's access-control policy. +type Policy struct { + DefaultRole string `json:"default_role,omitempty"` + // AdminRole is the role granted full access and the allowlist bypass. + // Empty means the server's default ("admin") applies. + AdminRole string `json:"admin_role,omitempty"` + Tables map[string]TablePolicy `json:"tables"` +} + +// TablePolicy describes per-table access control. +type TablePolicy struct { + Select map[string]RolePermissions `json:"select,omitempty"` + Insert map[string]RolePermissions `json:"insert,omitempty"` +} + +// RolePermissions describes a role's access to a table. +type RolePermissions struct { + AllowColumns []string `json:"allow_columns,omitempty"` + DenyColumns []string `json:"deny_columns,omitempty"` + Filter map[string]PolicyFilter `json:"filter,omitempty"` + Check map[string]PolicyFilter `json:"check,omitempty"` + AllowedAggregations []string `json:"allowed_aggregations,omitempty"` + DeniedAggregations []string `json:"denied_aggregations,omitempty"` + MaxRows *int `json:"max_rows,omitempty"` + MaxExecutionTime any `json:"max_execution_time,omitempty"` + MaxRowsToRead *int `json:"max_rows_to_read,omitempty"` + MaxMemoryUsage any `json:"max_memory_usage,omitempty"` +} + +// PolicyFilter describes a policy filter predicate. Fields are pointers so an +// intentional empty-string comparison (e.g. Eq pointing at "") round-trips +// distinctly from an absent operator, matching the server's semantics. +type PolicyFilter struct { + Eq *string `json:"_eq"` + Neq *string `json:"_neq"` + Gt *string `json:"_gt"` + Lt *string `json:"_lt"` + In *string `json:"_in"` +} + +// ValidationResult is the response from policy validation. +type ValidationResult struct { + Valid bool `json:"valid"` +} + +// ── Streaming types ─────────────────────────────────────────────────────── + +// StreamStatus represents the connection state of a stream. +type StreamStatus string + +const ( + StatusConnecting StreamStatus = "connecting" + StatusLive StreamStatus = "live" + StatusReconnecting StreamStatus = "reconnecting" + StatusClosed StreamStatus = "closed" +) + +// StreamEvent is a single event from an SSE stream. +type StreamEvent struct { + Table string `json:"table"` + Timestamp string `json:"timestamp"` + Data map[string]any `json:"data"` +} + +// StreamSubscriber receives events from a stream. +type StreamSubscriber struct { + // Initial is called once with historical backfill data (live queries only). + Initial func(rows []map[string]any, err error) + // Next is called for each live event. + Next func(event StreamEvent) + // Status is called when the connection status changes. + Status func(status StreamStatus) + // Error is called on stream errors. + Error func(err error) +} + +// StreamOptions configures a stream. +type StreamOptions struct { + // Since is an RFC3339 timestamp for gap-fill replay. + Since string +} + +// ── Fetch/page types ────────────────────────────────────────────────────── + +// Page wraps a result set with pagination metadata. +type Page[T any] struct { + // Data is the result rows. + Data []T + // HasMore is true if more rows may be available. + HasMore bool + // Next fetches the next page. Nil when no cursor is available. + Next func(ctx context.Context) (*Page[T], error) +} diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go new file mode 100644 index 00000000..6bcb8e5f --- /dev/null +++ b/clients/go/wavehouse.go @@ -0,0 +1,142 @@ +// Package wavehouse is the official Go SDK for WaveHouse — a schema-aware +// real-time API gateway for ClickHouse. Zero third-party runtime dependencies. +// +// Create a client with [NewClient], then use [Client.From] for table +// operations, [Client.Pipe] for named queries, or the admin namespaces +// ([Client.Schema], [Client.Policy], etc.) for management. +// +// client := wavehouse.NewClient(wavehouse.Config{ +// BaseURL: "http://localhost:8080", +// }) +// rows, err := client.From("clicks").SelectAll().Fetch(ctx) +package wavehouse + +import ( + "context" + "net/http" +) + +// Config configures a [Client]. +type Config struct { + // BaseURL of the WaveHouse server (e.g. "http://localhost:8080"). + BaseURL string + + // Auth provides a bearer token for authenticated requests. Called before + // each request; return "" to skip the Authorization header. Nil means + // unauthenticated access (the server falls back to default_role). + Auth func(ctx context.Context) (string, error) + + // Options tunes transport behavior. + Options *ClientOptions + + // HTTPClient overrides the default http.Client. Useful for custom TLS, + // proxies, or test transports. + HTTPClient *http.Client +} + +// ClientOptions tunes transport behavior. +type ClientOptions struct { + // MaxRetries is the maximum number of retry attempts for retryable errors. + // Total attempts = MaxRetries + 1. Default: 2. + MaxRetries int +} + +// StaticToken returns an Auth function that always returns the same token. +// Convenience for cases where the token doesn't rotate. +func StaticToken(token string) func(context.Context) (string, error) { + return func(context.Context) (string, error) { return token, nil } +} + +// Client is the WaveHouse SDK entry point. +type Client struct { + ctx httpContext + + // Schema provides admin-only schema introspection. + Schema *SchemaNamespace + // Policy provides admin-only access-control policy management. + Policy *PolicyNamespace + // DLQ provides admin-only dead-letter-queue statistics. + DLQ *DLQNamespace + // Sys provides system health checks. + Sys *SysNamespace + // Pipes provides admin-only named-pipe management. + Pipes *PipesNamespace +} + +// NewClient creates a new WaveHouse client. +func NewClient(cfg Config) *Client { + maxRetries := 2 + if cfg.Options != nil && cfg.Options.MaxRetries >= 0 { + maxRetries = cfg.Options.MaxRetries + } + + hc := cfg.HTTPClient + if hc == nil { + hc = http.DefaultClient + } + + c := &Client{ + ctx: httpContext{ + baseURL: trimTrailingSlashes(cfg.BaseURL), + auth: cfg.Auth, + maxRetries: maxRetries, + httpClient: hc, + }, + } + + c.Schema = &SchemaNamespace{ctx: c.ctx} + c.Policy = &PolicyNamespace{ctx: c.ctx} + c.DLQ = &DLQNamespace{ctx: c.ctx, createStream: c.createStream} + c.Sys = &SysNamespace{ctx: c.ctx} + c.Pipes = &PipesNamespace{ctx: c.ctx} + + return c +} + +// From returns a reference to a table for queries, inserts, and streams. +func (c *Client) From(table string) *TableRef { + return &TableRef{ + ctx: c.ctx, + table: table, + createStream: c.createStream, + } +} + +// Pipe returns a reference to a named query pipe. Pass params for the pipe's +// template parameters. +func (c *Client) Pipe(name string, params map[string]any) *PipeRef { + return &PipeRef{ + ctx: c.ctx, + name: name, + params: params, + createStream: c.createStream, + } +} + +// SQL executes a raw SQL query against ClickHouse. Requires the admin role. +// The server proxies the SQL verbatim to ClickHouse's HTTP interface. Results +// are decoded into []T; use [map[string]any] for dynamic schemas. +func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { + var rows []Row + err := doRequest(c.ctx, ctx, requestOptions{ + method: "POST", + path: "/v1/admin/query", + body: map[string]string{"sql": query}, + }, &rows) + if err != nil { + return nil, err + } + return rows, nil +} + +// createStream opens an SSE stream for the given table. +func (c *Client) createStream(table string, opts *StreamOptions) *StreamController { + return newStreamController(c.ctx, table, opts) +} + +func trimTrailingSlashes(s string) string { + for len(s) > 0 && s[len(s)-1] == '/' { + s = s[:len(s)-1] + } + return s +} diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs new file mode 100644 index 00000000..8eafc796 --- /dev/null +++ b/tests/conformance/conformance_ts.mjs @@ -0,0 +1,279 @@ +#!/usr/bin/env node +/** + * Cross-language wire-format conformance test for the TypeScript SDK. + * + * Reads wire_cases.json (owned by the Go module, at clients/go/testdata/) + * and verifies the TS SDK produces identical HTTP requests (method, path, + * content-type, body) to the shared fixture. + * + * Run: node tests/conformance/conformance_ts.mjs + * Exit 0 = all pass, exit 1 = failures. + */ + +import { readFileSync } from "node:fs"; +import { createServer } from "node:http"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); + +// Import the built SDK. +const { createClient } = await import(join(__dirname, "../../clients/ts/dist/index.js")); + +const cases = JSON.parse(readFileSync(join(__dirname, "../../clients/go/testdata/wire_cases.json"), "utf-8")); + +let lastCapture = { method: "", path: "", contentType: "", body: "" }; + +function resetCapture() { + lastCapture = { method: "", path: "", contentType: "", body: "" }; +} + +// Start echo server. +const server = createServer((req, res) => { + const chunks = []; + req.on("data", (c) => chunks.push(c)); + req.on("end", () => { + lastCapture = { + method: req.method ?? "", + path: req.url ?? "", + contentType: req.headers["content-type"] ?? "", + body: Buffer.concat(chunks).toString("utf-8"), + }; + res.setHeader("Content-Type", "application/json"); + if (req.url?.startsWith("/v1/dlq")) { + res.end(JSON.stringify({ tables: {}, total: 0 })); + } else if (req.url?.startsWith("/v1/schema") && req.method === "GET") { + res.end(JSON.stringify({})); + } else if (req.url === "/v1/admin/policy/validate" && req.method === "POST") { + res.end(JSON.stringify({ valid: true })); + } else if (req.url?.startsWith("/v1/admin/policy") && req.method === "GET") { + res.end(JSON.stringify({ tables: {} })); + } else if (req.url?.startsWith("/v1/admin/pipes/") && req.method === "GET") { + res.end(JSON.stringify({ name: "test", sql: "SELECT 1" })); + } else if (req.url === "/v1/admin/pipes" && req.method === "GET") { + res.end(JSON.stringify([])); + } else if (req.url?.startsWith("/v1/ingest")) { + if (lastCapture.contentType === "application/x-ndjson") { + res.end(JSON.stringify({ total: 0, succeeded: 0, failed: 0, duplicates: 0 })); + } else { + res.end(JSON.stringify({ ok: true })); + } + } else if (req.url === "/v1/health") { + res.end(""); + } else { + res.end(JSON.stringify([])); + } + }); +}); + +await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); +const { port } = server.address(); +const baseURL = `http://127.0.0.1:${port}`; + +function applyQueryOps(wh, table, operations) { + let q = wh.from(table).select(); + for (const op of operations) { + switch (op.method) { + case "select": + q = wh.from(table).select(...op.args); + break; + case "selectAll": + q = q.selectAll(); + break; + case "where": + q = q.where(op.args[0], op.args[1], op.args[2]); + break; + case "count": + q = q.count(op.args[0] || "*", op.args[1] || "count"); + break; + case "sum": + q = q.sum(op.args[0], op.args[1] || undefined); + break; + case "avg": + q = q.avg(op.args[0], op.args[1] || undefined); + break; + case "min": + q = q.min(op.args[0], op.args[1] || undefined); + break; + case "max": + q = q.max(op.args[0], op.args[1] || undefined); + break; + case "countDistinct": + q = q.countDistinct(op.args[0], op.args[1] || undefined); + break; + case "aggregate": + q = q.aggregate(op.args[0], op.args[1], op.args[2]); + break; + case "groupBy": + q = q.groupBy(...op.args); + break; + case "orderBy": + q = q.orderBy(op.args[0], op.args[1] || "asc"); + break; + case "limit": + q = q.limit(op.args[0]); + break; + case "timeRange": + q = q.timeRange(op.args[0], op.args[1], op.args[2] || undefined); + break; + case "cacheTTL": + q = q.cacheTTL(op.args[0]); + break; + } + } + return q; +} + +function normalizePath(p) { + return p.replace(/\+/g, "%20"); +} + +function deepEqual(a, b) { + return JSON.stringify(sortKeys(a)) === JSON.stringify(sortKeys(b)); +} + +function sortKeys(v) { + if (v === null || v === undefined) return v; + if (Array.isArray(v)) return v.map(sortKeys); + if (typeof v === "object") { + const sorted = {}; + for (const k of Object.keys(v).sort()) { + sorted[k] = sortKeys(v[k]); + } + return sorted; + } + return v; +} + +let passed = 0; +let failed = 0; +const failures = []; + +for (const tc of cases) { + resetCapture(); + const wh = createClient({ baseURL, options: { maxRetries: 0 } }); + + try { + switch (tc.endpoint) { + case "query": { + const q = applyQueryOps(wh, tc.table, tc.operations ?? []); + await q.fetch(); + break; + } + case "ingest": + if (tc.operations?.[0]?.method === "insert") { + await wh.from(tc.table).insert(tc.operations[0].args[0]); + } + break; + case "ingest_batch": + if (tc.operations?.[0]?.method === "insert") { + await wh.from(tc.table).insert(tc.operations[0].args[0]); + } + break; + case "pipe": + await wh.pipe(tc.pipe_name, tc.pipe_params ?? undefined).fetch(); + break; + case "sql": + await wh.sql(tc.sql); + break; + case "health": + await wh.sys.health(); + break; + case "schema_list": + await wh.schema.list(); + break; + case "schema_refresh": + await wh.schema.refresh(); + break; + case "policy_get": + await wh.policy.get(); + break; + case "policy_set": + await wh.policy.set(tc.policy_body); + break; + case "policy_validate": + await wh.policy.validate(tc.policy_body); + break; + case "dlq_list": + await wh.dlq.list(); + break; + case "dlq_table": + await wh.dlq.table(tc.table); + break; + case "pipes_list": + await wh.pipes.list(); + break; + case "pipes_get": + await wh.pipes.get(tc.pipe_name); + break; + case "pipes_set": + await wh.pipes.set(tc.pipe_name, tc.pipe_def); + break; + case "pipes_delete": + await wh.pipes.delete(tc.pipe_name); + break; + default: + passed++; + continue; + } + + const errs = []; + + if (tc.expected_method && lastCapture.method !== tc.expected_method) { + errs.push(`method: want ${tc.expected_method}, got ${lastCapture.method}`); + } + + if (tc.expected_path && normalizePath(lastCapture.path) !== normalizePath(tc.expected_path)) { + errs.push(`path: want ${tc.expected_path}, got ${lastCapture.path}`); + } + + if (tc.expected_content_type && lastCapture.contentType !== tc.expected_content_type) { + errs.push(`content-type: want ${tc.expected_content_type}, got ${lastCapture.contentType}`); + } + + if (tc.expected_raw_body !== undefined) { + if (lastCapture.body !== tc.expected_raw_body) { + errs.push(`raw body:\n want: ${tc.expected_raw_body}\n got: ${lastCapture.body}`); + } + } else if (tc.expected_body !== undefined && tc.expected_body !== null) { + let captured; + try { + captured = JSON.parse(lastCapture.body); + } catch { + errs.push(`body not valid JSON: ${lastCapture.body}`); + } + if (captured !== undefined && !deepEqual(captured, tc.expected_body)) { + errs.push( + `body mismatch:\n want: ${JSON.stringify(tc.expected_body)}\n got: ${JSON.stringify(captured)}`, + ); + } + } + + if (errs.length > 0) { + failed++; + failures.push({ name: tc.name, errors: errs }); + } else { + passed++; + } + } catch (err) { + failed++; + failures.push({ name: tc.name, errors: [`exception: ${err.message}`] }); + } +} + +server.close(); + +console.log(`\nWire-format conformance (TS SDK): ${passed} passed, ${failed} failed, ${cases.length} total\n`); + +for (const f of failures) { + console.log(` ✗ ${f.name}`); + for (const e of f.errors) { + console.log(` ${e}`); + } +} + +if (failed > 0) { + process.exit(1); +} else { + console.log(" ✓ All cases passed\n"); +} From 3089c719c72fb5045d7bef77c62bd72b0353bfb1 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 15:13:33 -0400 Subject: [PATCH 02/40] docs(sdk): add Go SDK documentation pages Six Starlight pages covering installation, queries, streaming, pipes, admin operations, and API reference. Sidebar nav group added. Cross-link from SDK index page. Architecture page updated with Go SDK. Root README updated with Go SDK install. --- README.md | 1 + docs/src/config/sidebar.ts | 24 +- docs/src/content/docs/architecture.md | 1 + docs/src/content/docs/sdk/go/admin.md | 118 +++++++ docs/src/content/docs/sdk/go/index.md | 248 ++++++++++++++ docs/src/content/docs/sdk/go/pipes.md | 100 ++++++ docs/src/content/docs/sdk/go/queries.md | 390 ++++++++++++++++++++++ docs/src/content/docs/sdk/go/reference.md | 233 +++++++++++++ docs/src/content/docs/sdk/go/streaming.md | 254 ++++++++++++++ docs/src/content/docs/sdk/index.mdx | 21 ++ 10 files changed, 1386 insertions(+), 4 deletions(-) create mode 100644 docs/src/content/docs/sdk/go/admin.md create mode 100644 docs/src/content/docs/sdk/go/index.md create mode 100644 docs/src/content/docs/sdk/go/pipes.md create mode 100644 docs/src/content/docs/sdk/go/queries.md create mode 100644 docs/src/content/docs/sdk/go/reference.md create mode 100644 docs/src/content/docs/sdk/go/streaming.md diff --git a/README.md b/README.md index 966bc683..d9e658c4 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,7 @@ If you're building user-facing analytics, WaveHouse is like **Supabase for Click - **Real-time** — native SSE push, broadcast *before* the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients. - **Security** — Hasura-style per-table, per-role column + row policies with JWT claim templating, stored in NATS KV. - **Client** — `@wavehouse/sdk`: zero-dependency TypeScript client with query builder, live queries, streaming, and schema codegen. +- **Go SDK** — `go get github.com/Wave-RF/WaveHouse/clients/go`: full API-tree parity with the TS SDK — ingest, query, streaming, and policy management from Go. ## 📊 How it compares diff --git a/docs/src/config/sidebar.ts b/docs/src/config/sidebar.ts index b3a61a20..f114c692 100644 --- a/docs/src/config/sidebar.ts +++ b/docs/src/config/sidebar.ts @@ -25,10 +25,15 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ items: [ { label: "API Reference", slug: "api" }, { - // Topic-first SDK pages: when a second SDK language lands, these - // shared usage pages grow code tabs and each - // language gets its own setup/caveats page — the topic URLs never - // churn (decision in PR #313). + // Topic-first SDK pages: the multi-language plan on record (PR #313) + // was for a second language to grow on these + // shared pages instead of a parallel tree. The Go SDK launched as + // its own docs/src/content/docs/sdk/go/* tree instead — its API + // shape (context.Context, (T, error), generics on package-level + // funcs) diverges enough from the TS builder that shared prose read + // worse than dedicated pages. Revisit folding these into tabs if a + // third language lands and the duplication becomes a maintenance + // cost. label: "TypeScript SDK", items: [ { label: "Overview", slug: "sdk" }, @@ -39,6 +44,17 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ { label: "Reference & CLI", slug: "sdk/reference" }, ], }, + { + label: "Go SDK", + items: [ + { label: "Overview", slug: "sdk/go" }, + { label: "Queries", slug: "sdk/go/queries" }, + { label: "Streaming & Live Queries", slug: "sdk/go/streaming" }, + { label: "Pipes", slug: "sdk/go/pipes" }, + { label: "Admin & System", slug: "sdk/go/admin" }, + { label: "Reference & CLI", slug: "sdk/go/reference" }, + ], + }, ], }, { diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 4c6a8958..8b8b6ff2 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -276,4 +276,5 @@ Client GET /v1/stream | Embedded KV | Pebble | Optional deduplication | | Config | cleanenv | YAML + env var config loading | | Release | GoReleaser | Cross-platform binary builds | +| Client SDKs | TypeScript, Go | Typed clients with the same feature set (ingest, query, pipes, streaming, admin) | | Containers | Docker (distroless) | Minimal production images | diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md new file mode 100644 index 00000000..a0941433 --- /dev/null +++ b/docs/src/content/docs/sdk/go/admin.md @@ -0,0 +1,118 @@ +--- +title: "Go SDK Admin & System" +description: "Schema introspection, access-control policy, DLQ stats, and health checks in the WaveHouse Go SDK." +--- + +Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. +Everything here except `client.Sys.Health` requires the admin role +(`policy.admin_role`) — see [Access Control](/access-control) for how roles +resolve. Compare with the TypeScript SDK's [Admin & System](/sdk/admin) +page. + +## Schema — `client.Schema` + +Introspect ClickHouse table schemas. + +```go +// List all table schemas. +schemas, err := wh.Schema.List(ctx) +// schemas is wavehouse.Schemas — map[string]TableSchema, keyed by table name + +// Force refresh from ClickHouse. +err = wh.Schema.Refresh(ctx) +``` + +Individual table schema is also available via `wh.From("clicks").Schema(ctx)`. + +> `wh.Schema.List`, `wh.Schema.Refresh`, and `wh.From(t).Schema` hit +> `/v1/schema*`, which are **admin-only** endpoints. Against any non-dev +> policy (anything but `default_role: admin`), construct the client with an +> admin-role token or these calls return a `*wavehouse.Error` with +> `Status: 403`. + +--- + +## Policy — `client.Policy` + +Manage Hasura-style access control policies. Requires the admin role +(`policy.admin_role`). + +```go +// Get current policy. +policy, err := wh.Policy.Get(ctx) + +// Update policy. +tenantFilter := "{{ jwt.app_metadata.tenant_id }}" +err = wh.Policy.Set(ctx, &wavehouse.Policy{ + DefaultRole: "viewer", + Tables: map[string]wavehouse.TablePolicy{ + "clicks": { + Select: map[string]wavehouse.RolePermissions{ + "viewer": { + AllowColumns: []string{"page", "button", "received_timestamp"}, + Filter: map[string]wavehouse.PolicyFilter{ + "tenant_id": {Eq: &tenantFilter}, + }, + }, + "admin": {AllowColumns: []string{"*"}}, + }, + }, + }, +}) + +// Validate without applying (dry run). +result, err := wh.Policy.Validate(ctx, policyDraft) +// result.Valid == true, or err wraps the validation failure details +``` + +`PolicyFilter`'s fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, not +`string` — an intentional empty-string comparison round-trips distinctly +from an absent operator. Take the address of a local variable (as above) or +write a small helper if you find yourself doing this often: + +```go +func strPtr(s string) *string { return &s } +``` + +--- + +## DLQ — `client.DLQ` + +Dead Letter Queue operations. Requires the admin role (`policy.admin_role`). + +```go +// Get DLQ statistics. +stats, err := wh.DLQ.List(ctx) +// stats.Tables: map[string]int{"clicks": 3, "users": 0} +// stats.Total: 3 + +// Stats for a specific table. +stats, err = wh.DLQ.Table(ctx, "clicks") +``` + +`wh.DLQ.Stream(opts)` exists in the API but is **not yet functional**: +there is no server-side DLQ stream today (the SSE bridge only carries +`ingest.>` subjects), so it connects and receives no events — live DLQ +streaming is tracked in +[#197](https://github.com/Wave-RF/WaveHouse/issues/197). + +--- + +## System — `client.Sys` + +Content-free server-online check. + +```go +// Health hits the public, content-free /v1/health route — 200 → nil error, +// any other status (including 503) → a non-nil *wavehouse.Error. +// Use it to check a server is reachable before sending data. +if err := wh.Sys.Health(ctx); err != nil { + // server is unreachable or not yet past boot + log.Println(err) +} +``` + +> Readiness (`/readyz`) is intentionally **not** exposed through the SDK — +> it runs a ClickHouse query per call and is a load-balancer / reverse-proxy +> concern, not the client's. Probe `/readyz` directly from your +> orchestrator if you need it. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md new file mode 100644 index 00000000..db515ef2 --- /dev/null +++ b/docs/src/content/docs/sdk/go/index.md @@ -0,0 +1,248 @@ +--- +title: "Go SDK" +description: "Zero-dependency Go client SDK — query builder, real-time streaming, codegen." +--- + +`github.com/Wave-RF/WaveHouse/clients/go` — zero third-party runtime +dependency Go client for WaveHouse (stdlib only). + +:::tip[Looking for the TypeScript SDK?] +This page and the rest of `/sdk/go/*` cover the Go client. The +JavaScript/TypeScript client (`@wavehouse/sdk`) has its own docs starting at +[SDK Overview](/sdk) — the two SDKs speak the same wire format, so anything +you learn about WaveHouse's query builder, streaming, or admin endpoints on +either page mostly carries over. +::: + +## Installation + +```bash +go get github.com/Wave-RF/WaveHouse/clients/go +``` + +Requires Go 1.26.5 or later (the minimum pinned in the module's `go.mod`). + +## Import + +```go +import wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +``` + +The package name is `wavehouse`; aliasing the import isn't required, but +keeps call sites short (`wavehouse.NewClient(...)`, `wavehouse.OpEq`, ...) — +every example on these pages assumes it. + +## Quick Start + +```go +package main + +import ( + "context" + "fmt" + "log" + + wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +) + +func main() { + ctx := context.Background() + + // Create a client. Auth is optional — omit it for public/unauthenticated + // access (the server falls back to policy.default_role). + wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), + }) + + // Health check. + if err := wh.Sys.Health(ctx); err != nil { + log.Fatal(err) + } + + // Insert a row. + if _, err := wh.From("clicks").Insert(ctx, map[string]any{ + "page": "/home", "button": "signup", + }); err != nil { + log.Fatal(err) + } + + // Query with the fluent builder. + page, err := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(ctx) + if err != nil { + log.Fatal(err) + } + for _, row := range page.Data { + fmt.Println(row["page"], row["button"]) + } + + // Stream. + stream := wh.From("clicks").Stream(nil) + defer stream.Close() + unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ + Next: func(e wavehouse.StreamEvent) { fmt.Println(e.Data) }, + Status: func(s wavehouse.StreamStatus) { fmt.Println("Stream:", s) }, + }) + defer unsub() +} +``` + +## Creating a Client + +```go +import wavehouse "github.com/Wave-RF/WaveHouse/clients/go" + +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "https://wavehouse.example.com", + Auth: func(ctx context.Context) (string, error) { + return myAuthProvider.GetToken(ctx) + }, + Options: &wavehouse.ClientOptions{ + MaxRetries: 2, + }, +}) +``` + +### `Config` + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `BaseURL` | `string` | — | WaveHouse server URL (required) | +| `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider, called before each request. `nil` means unauthenticated access | +| `Options` | `*ClientOptions` | `nil` | Transport tuning (see below) | +| `HTTPClient` | `*http.Client` | `http.DefaultClient` | Override for custom TLS, proxies, or test transports | + +### `ClientOptions` + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, network failures) | + +:::caution[`Options` opts you out of the default, not just in] +The default of 2 retries only applies when `Config.Options` is `nil`. If you +set `Options` to configure anything else in the future, an unset +`MaxRetries` field is Go's int zero value — `0` — which is a **valid, +explicit** "no retries" setting, not "use the default." Today `MaxRetries` +is the struct's only field, so this mostly matters if you pass +`&wavehouse.ClientOptions{}` and expect retry-by-default: you won't get it. +::: + +For a static token that never rotates, use `wavehouse.StaticToken(token)` +instead of writing the closure yourself: + +```go +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), +}) +``` + +:::note[How the token is transmitted] +Unlike a browser's `EventSource`, Go's `net/http` client can set arbitrary +headers on any request — so the Go SDK sends `Authorization: Bearer ` +on **every** request, including SSE streams. There's no `?token=` query +parameter fallback to worry about (that's a TypeScript-SDK-in-the-browser +concern only; see its [equivalent note](/sdk#creating-a-client)). +::: + +## Typed Rows (Generics) + +Pass a row type as a type parameter to get results decoded straight into +your struct, instead of `map[string]any`: + +```go +type ClickRow struct { + Page string `json:"page"` + Button string `json:"button"` + DurationMS int `json:"duration_ms"` +} + +page, err := wavehouse.FetchTyped[ClickRow](ctx, + wh.From("clicks").Select("page", "button", "duration_ms").Limit(100), +) +// page.Data is []ClickRow +``` + +Generate row structs from a running server with the +[codegen CLI](/sdk/go/reference#codegen-cli). + +`FetchTyped` is a package-level generic function, not a method — Go doesn't +support generic methods, so this (and `Fetch[Row]` for pipes, and +`SQL[Row]` for raw SQL) are top-level functions that take the client or +builder as an argument. Untyped equivalents (`.FetchUntyped(ctx)`, decoding +into `map[string]any`) are ordinary methods, since they need no type +parameter. + +## Error Handling + +Every SDK operation returns `(T, error)` — the idiomatic Go shape, and the +direct equivalent of the TypeScript SDK's +[`Result`](/sdk#result-type) discriminated union. Errors are always +`*wavehouse.Error`; unwrap with `errors.As`: + +```go +page, err := wh.From("clicks").Fetch(ctx) +if err != nil { + var whErr *wavehouse.Error + if errors.As(err, &whErr) { + fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } + return err +} +``` + +```go +type Error struct { + Status int // HTTP status (0 for network/abort errors) + Code string // e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED" + Message string // Human-readable error message + Details map[string]any // Parsed response body, if available + Retryable bool // Whether the SDK would retry this error +} +``` + +`wavehouse.IsRetryable(err)` is a shortcut for `errors.As` + `.Retryable`. +The full error-code table lives in +[Reference → Error Handling](/sdk/go/reference#error-handling). + +## Differences from the TypeScript SDK + +The two SDKs share a wire format and mirror each other's feature set closely +(a shared `testdata/wire_cases.json` conformance fixture in the repo asserts +both produce identical HTTP requests for equivalent builder calls), but the +languages pull the API shape in different directions: + +- **No `Result` union.** Go returns `(T, error)`; nothing is wrapped in + an `{ok, data, error}` object, and there's no `error: null` sentinel to + check — a non-nil `error` is the only signal. +- **`context.Context` instead of `AbortSignal`.** Every non-streaming call + takes a `ctx context.Context` as its first argument; cancel it (timeout or + `cancel()`) instead of building an `AbortController`. See + [Reference → Context Cancellation](/sdk/go/reference#context-cancellation). +- **Streams are closed explicitly, not via `ctx`.** `TableRef.Stream` / + `QueryBuilder.Stream` don't take a `context.Context` — the returned + `*StreamController` manages its own background goroutine and connection, + torn down by calling `.Close()` (deferred `stream.Close()` is the usual + pattern). See [Streaming](/sdk/go/streaming). +- **Generics live on package-level functions, not methods** (`FetchTyped[Row]`, + `Fetch[Row]`, `SQL[Row]`), because Go doesn't support type parameters on + methods. +- **No implicit "await."** A `QueryBuilder` isn't `PromiseLike` — call + `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` + explicitly; there's no bare `await builder` shortcut. +- **`Insert` accepts typed row slices, not just maps.** Passing + `[]ClickRow{...}` (any slice type, detected via reflection) batches as + NDJSON exactly like `[]map[string]any` — see + [Queries → Insert](/sdk/go/queries#insertctx-data). + +## Explore the Go SDK + +- [Queries](/sdk/go/queries) — Tables, the chainable query builder, pagination, and raw SQL. +- [Streaming & Live Queries](/sdk/go/streaming) — Real-time SSE streams, client-side filtering, and backfill-then-live queries. +- [Pipes](/sdk/go/pipes) — Execute and manage named query pipes. +- [Admin & System](/sdk/go/admin) — Schema introspection, access-control policy, DLQ stats, and health checks. +- [Reference & CLI](/sdk/go/reference) — Error codes, context cancellation, the full API tree, and the codegen CLI. diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md new file mode 100644 index 00000000..44786601 --- /dev/null +++ b/docs/src/content/docs/sdk/go/pipes.md @@ -0,0 +1,100 @@ +--- +title: "Go SDK Pipes" +description: "Execute and manage named query pipes with the WaveHouse Go SDK." +--- + +Named pipes are server-defined, parameterized queries — the +[Named Pipes guide](/pipes) covers defining them. The SDK executes pipes for +any allowed role and manages their definitions under the admin role. +Compare with the TypeScript SDK's [Pipes](/sdk/pipes) page. + +## Named Pipes — `client.Pipe(name, params)` + +Execute a pre-defined named query pipe. Returns a `*PipeRef`; unlike the +TypeScript SDK's `PipeRef` (which is `PromiseLike`), you always call +`.FetchUntyped(ctx)` or the package-level `wavehouse.Fetch[Row]` explicitly. + +```go +rows, err := wavehouse.Fetch[map[string]any](ctx, + wh.Pipe("top_pages", map[string]any{"start_date": "2026-01-01", "limit": 50}), +) +``` + +### `wavehouse.Fetch[Row](ctx, pipeRef)` + +Execute and decode results into `[]Row`. Package-level generic function +(Go has no generic methods) — the same pattern as `FetchTyped` for queries +and `SQL` for raw SQL. + +```go +type TopPage struct { + Page string `json:"page"` + Views int `json:"views"` +} + +rows, err := wavehouse.Fetch[TopPage](ctx, wh.Pipe("top_pages", map[string]any{"limit": 50})) +``` + +### `.FetchUntyped(ctx)` + +Execute and decode results into `[]map[string]any`. The ordinary +(non-generic) method form of `Fetch`. + +```go +rows, err := wh.Pipe("top_pages", nil).FetchUntyped(ctx) +``` + +Pass `nil` for `params` when the pipe takes none, or the pipe requires only +parameters with server-side defaults. + +### `.Stream(opts)` + +Open a live stream from the pipe's underlying query. See +[Streaming](/sdk/go/streaming). + +This streams by table name, using the pipe's own name as the table — it +only works when the pipe name is also a valid table name. This matches the +TypeScript SDK's `PipeRef.stream()`, which has the same limitation. + +```go +stream := wh.Pipe("top_pages", nil).Stream(nil) +``` + +--- + +## Pipes Admin — `client.Pipes` + +Manage named query pipes. Requires the admin role (`policy.admin_role`). + +```go +// List all pipes. +pipes, err := wh.Pipes.List(ctx) + +// Get a single pipe definition. +pipe, err := wh.Pipes.Get(ctx, "top_pages") + +// Create or update. +err = wh.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ + SQL: "SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}", + Parameters: []wavehouse.ParamDef{ + {Name: "limit", Type: "number", Required: false, Default: 100}, + }, + Description: "Top pages by view count", + AllowedRoles: []string{"viewer", "admin"}, +}) + +// Delete. +err = wh.Pipes.Delete(ctx, "old_pipe") +``` + +`PipeDef` is `Pipe` minus the `Name` field — the name is already in the +`Set`/`Get`/`Delete` call's path argument: + +```go +type PipeDef struct { + SQL string + Parameters []ParamDef + Description string + AllowedRoles []string +} +``` diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md new file mode 100644 index 00000000..2b56fd5b --- /dev/null +++ b/docs/src/content/docs/sdk/go/queries.md @@ -0,0 +1,390 @@ +--- +title: "Go SDK Queries" +description: "Tables, the chainable query builder, pagination, and raw SQL in the WaveHouse Go SDK." +--- + +Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: +table references, the chainable query builder, cursor pagination, and the +admin-only raw-SQL escape hatch. Every call takes a `context.Context` as its +first argument and returns `(T, error)` — see +[Error Handling](/sdk/go#error-handling). Compare with the TypeScript SDK's +[Queries](/sdk/queries) page, which covers the same surface with a +`Result`-returning, `PromiseLike` builder. + +## Tables — `client.From(table)` + +`From` returns a `*TableRef` — a reference to a table. It performs no +request by itself, so it's safe to store in a variable or pass around. + +```go +clicks := wh.From("clicks") +``` + +### `.Fetch(ctx)` + +Shortcut for "select every column", with a default limit of 1000 +(`wavehouse.DefaultLimit`). Internally it's +`t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)` — unlike the +TypeScript SDK's `.fetch(opts?)`, there's no options struct to override the +limit or attach anything per-call; chain `.SelectAll().Limit(n)` yourself +(see [Query Builder](#query-builder)) if you need a different limit. + +When an access-control policy restricts your role's columns, the server +returns only the columns your role is allowed to read — `.Fetch()` is never +a way around `deny_columns`/`allow_columns` (see +[Access control](/access-control#column-permissions)). + +```go +page, err := clicks.Fetch(ctx) +if err != nil { + log.Fatal(err) +} +for _, row := range page.Data { + fmt.Println(row["page"]) +} +``` + +To paginate, use the query builder with an explicit `.OrderBy()` instead — +see [Pagination](#pagination). + +### `.Insert(ctx, data)` + +Insert one row or many. What you pass determines the wire format: + +- A single **map or struct** (anything that isn't a slice, and isn't + `[]byte`) is sent as JSON: `POST /v1/ingest?table={table}`. +- **Any slice** — `[]map[string]any`, a generated/user-defined row type like + `[]ClickRow`, etc. — is serialized to NDJSON (one record per line, via + reflection for non-`[]map[string]any` slices) and sent as a single + `application/x-ndjson` request, so a bad record doesn't fail or hide the + rest of the batch. Per-record outcomes come back in the result. + +```go +// Single row → InsertResult{OK: true} (or Duplicate: &true when dedup skips it) +res, err := clicks.Insert(ctx, map[string]any{"page": "/home", "button": "cta"}) + +// Many rows (map slice) → one NDJSON request, per-record summary +res, err = clicks.Insert(ctx, []map[string]any{ + {"page": "/home", "button": "cta"}, + {"page": "/about", "button": "nav"}, +}) +// res.OK, res.Total, res.Succeeded, res.Failed, res.Duplicates, res.Results + +// Many rows (typed slice) — same NDJSON path, via reflection +type ClickRow struct { + Page string `json:"page"` + Button string `json:"button"` +} +res, err = clicks.Insert(ctx, []ClickRow{ + {Page: "/home", Button: "cta"}, + {Page: "/about", Button: "nav"}, +}) +``` + +For a batch insert, `res.OK` is `true` only when every record succeeded +(`*res.Failed == 0`). Inspect `res.Failed` and `res.Results` (each +`InsertRecordResult{Index, OK, Duplicate, Error}`, 1-based `Index`) for +partial failures — the returned `error` is reserved for whole-request +failures (network, `404` unknown table, `403` forbidden, `503` +backpressure). An empty slice is a no-op and sends no request. + +> The server itself is format-agnostic: `POST /v1/ingest` also accepts a raw +> JSON array or a single object directly (the `Content-Type` is only a +> hint), so non-SDK clients can send whichever shape is convenient. See the +> [API reference](/api#post-v1ingesttabletable--ingest-data). + +### `.InsertNDJSON(ctx, ndjson)` + +Insert pre-formatted NDJSON you already have, as a plain `string` — a file +you've read, or a string you built yourself — without first parsing it into +Go values. Returns the same per-record summary as a slice `Insert`. + +```go +// From a literal string. +res, err := clicks.InsertNDJSON(ctx, `{"page":"/a"}`+"\n"+`{"page":"/b"}`) + +// From a file on disk. +raw, err := os.ReadFile("events.ndjson") +if err != nil { + log.Fatal(err) +} +res, err = clicks.InsertNDJSON(ctx, string(raw)) +``` + +### `.Schema(ctx)` + +Fetch the table's column definitions from ClickHouse. Admin-only. + +```go +schema, err := clicks.Schema(ctx) +// schema.Name == "clicks" +// schema.Columns: []Column{{Name: "page", Type: "String", IsNullable: false, HasDefault: false}, ...} +``` + +### `.Select(...columns)` + +Start a query builder chain. See [Query Builder](#query-builder). + +```go +page, err := clicks.Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(ctx) +``` + +### `.SelectAll()` + +Start a query that selects **every column your role is allowed to read** — +the explicit form of what `.Fetch()` does. Mutually exclusive with +`.Select(...)` and with aggregations (`.Count()`, `.Sum()`, etc.); the +server expands it to your allowed columns (never a raw `SELECT *`) and never +bypasses `deny_columns`/`allow_columns`. See +[Access control → Column permissions](/access-control#column-permissions). + +```go +page, err := clicks.SelectAll().Where("country", wavehouse.OpEq, "US").Limit(10).FetchUntyped(ctx) +``` + +### `.Stream(opts)` + +Open a real-time event subscription. See [Streaming](/sdk/go/streaming). + +```go +stream := clicks.Stream(&wavehouse.StreamOptions{Since: "2026-01-01T00:00:00Z"}) +``` + +--- + +## Query Builder + +Returned by `tableRef.Select()`. Immutable — every chain method returns a +new `*QueryBuilder`, so intermediate values can be reused safely. Unlike the +TypeScript SDK's `PromiseLike` builder, a Go `*QueryBuilder` doesn't +auto-execute — call `.FetchUntyped(ctx)` or the package-level +`wavehouse.FetchTyped[Row](ctx, builder)` explicitly: + +```go +page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) +``` + +### Chain Methods + +All methods return a new `*QueryBuilder` — the original is unchanged. + +#### `.Select(...columns)` + +Append columns to the SELECT clause. A literal `"*"` is the column *named* +`*`, not a wildcard — use `.SelectAll()` for all columns. + +```go +q := clicks.Select("page").Select("button") // SELECT page, button +``` + +#### `.SelectAll()` + +Select every column your role may read (the all-columns wildcard, expanded +server-side to your allowed columns). Mutually exclusive with `.Select(...)` +and with aggregations (`.Count()`, `.Sum()`, etc.). + +```go +q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") +``` + +#### `.Where(column, op, value)` + +Add a filter condition, using the `FilterOp` constants: + +```go +clicks.Select("page"). + Where("score", wavehouse.OpGt, 10). + Where("page", wavehouse.OpLike, "/home%") +``` + +| `FilterOp` constant | Backend wire token | Description | +|----------------------|---------------------|--------------| +| `wavehouse.OpEq` | `eq` | Equal | +| `wavehouse.OpNeq` | `neq` | Not equal | +| `wavehouse.OpGt` | `gt` | Greater than | +| `wavehouse.OpGte` | `gte` | Greater than or equal | +| `wavehouse.OpLt` | `lt` | Less than | +| `wavehouse.OpLte` | `lte` | Less than or equal | +| `wavehouse.OpIn` | `in` | Value in array — accepts a Go slice of any element type (`[]string`, `[]int`, `[]any`, ...) | +| `wavehouse.OpLike` | `like` | SQL LIKE pattern | +| `wavehouse.OpNotLike` | — | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | + +#### Aggregations + +```go +clicks.Select("page"). + Count("*", "total"). // COUNT(*) + Sum("score", "total_score"). // SUM(score) + Avg("score", "avg_score"). // AVG(score) + Min("score", "min_score"). // MIN(score) + Max("score", "max_score"). // MAX(score) + CountDistinct("page", "unique_pages"). + Aggregate("uniqExact", "user_id", "unique_users") // custom fn +``` + +Each aggregation method signature: `(column, alias string) *QueryBuilder`. +`Count` defaults to `column="*"` when `column` is `""`, and `alias="count"` +when `alias` is `""`; the other aggregations default `alias` to +`"_"` when left empty. + +#### `.GroupBy(...columns)` + +```go +clicks.Select("page").Count("", "").GroupBy("page") +``` + +#### `.OrderBy(column, dir)` + +```go +clicks.Select("page").Count("", "total").OrderBy("total", "desc") +``` + +`dir` defaults to `"asc"` when passed as `""`. + +#### `.Limit(n)` + +```go +clicks.Select().Limit(100) +``` + +If no limit is specified, `wavehouse.DefaultLimit` (1000) is applied +automatically to prevent unbounded result sets. The server also enforces the +configured maximum (`query.default_max_rows`, default 10,000 rows). + +#### `.TimeRange(column, since, until)` + +Filter by a time window. `since` and `until` accept RFC3339 timestamps or +relative durations (`"1h"`, `"30m"`, `"7d"`, `"2w"` — day and week suffixes +expand to hours, so `"7d"` is `"168h"`). Pass `""` for `until` to leave it +open-ended. + +```go +clicks.Select("page").TimeRange("received_timestamp", "1h", "") +clicks.Select("page").TimeRange( + "received_timestamp", "2026-01-01T00:00:00Z", "2026-02-01T00:00:00Z", +) +``` + +#### `.CacheTTL(seconds)` + +Records a desired result-cache TTL on the builder. **Currently client-side +state only** — the value is never sent to the server, which derives each +result's cache TTL adaptively from query execution time. Wiring it through +the wire format is tracked in +[#280](https://github.com/Wave-RF/WaveHouse/issues/280). + +```go +clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side — see #280 +``` + +### `wavehouse.FetchTyped[Row](ctx, q)` + +Execute the query and decode rows into `[]Row`. Package-level generic +function (Go has no generic methods) — takes the builder as its argument. + +```go +type PageCount struct { + Page string `json:"page"` + Count int `json:"total"` +} + +page, err := wavehouse.FetchTyped[PageCount](ctx, + clicks.Select("page").Count("*", "total").GroupBy("page"), +) +// page.Data is []PageCount +``` + +### `.FetchUntyped(ctx)` + +Execute the query and decode rows into `[]map[string]any`. The ordinary +(non-generic) method form of `FetchTyped`. + +```go +page, err := clicks.Select("page").Limit(50).FetchUntyped(ctx) + +if page.HasMore && page.Next != nil { + page2, err := page.Next(ctx) // cursor-based pagination +} +``` + +### `.Stream(opts)` + +Open a live stream from the builder's table, applying `.Where()`/`.Select()` +filters and column projection client-side. See +[Streaming](/sdk/go/streaming). + +### Pagination + +`Page[T]`: + +```go +type Page[T any] struct { + Data []T + HasMore bool + Next func(ctx context.Context) (*Page[T], error) // nil when no cursor is available +} +``` + +When `Limit` is set and the result contains at least that many rows, +`HasMore` is `true`. Cursor-based pagination's `Next` walks the **first** +`.OrderBy()` column — it adds a filter on that column using the last row's +value — so `Next` is only attached when the query has an explicit +`.OrderBy()`. With no order column the result still reports `HasMore` +honestly, but `Next` is `nil` (there is no deterministic cursor to build) — +add an `.OrderBy()` to paginate. If the order column was left out of an +explicit `.Select(...)` projection, `Next` quietly returns an empty page +instead of erroring (there is no cursor value to read). + +```go +page, err := clicks.Select(). + OrderBy("received_timestamp", "desc"). + Limit(100). + FetchUntyped(ctx) +if err != nil { + log.Fatal(err) +} + +allRows := append([]map[string]any(nil), page.Data...) +for page.HasMore && page.Next != nil { + page, err = page.Next(ctx) + if err != nil { + log.Fatal(err) + } + allRows = append(allRows, page.Data...) +} +``` + +--- + +## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` + +Execute a raw SQL query. `/v1/admin/query` is admin-only: the caller's JWT +must resolve to the policy admin role (`admin_role`, `"admin"` by default). +A request with no token, or an invalid/expired one, falls back to the +`default_role` and is rejected. Package-level generic function — use +`map[string]any` for a dynamic/unknown schema. + +```go +rows, err := wavehouse.SQL[map[string]any](ctx, wh, + "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") + +// Or decode into a struct that matches the projected columns/aliases: +type PageTotal struct { + Page string `json:"page"` + Total int `json:"total"` +} +rows, err := wavehouse.SQL[PageTotal](ctx, wh, + "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") +``` + +:::note[No parameter binding through the SDK] +Positional `?` substitution is not supported, and the SDK has no way to +forward ClickHouse-style named params (the `WHERE id = {id:UInt32}` + +`param_id=42` query-string combo) — the proxy doesn't forward arbitrary +query-string params and `SQL[Row]` doesn't expose a hook to add them. Inline +literals into the SQL, or — for safe binding from user-supplied input — use +the structured query builder (`wh.From(table)...`). +::: diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md new file mode 100644 index 00000000..7e6e90cb --- /dev/null +++ b/docs/src/content/docs/sdk/go/reference.md @@ -0,0 +1,233 @@ +--- +title: "Go SDK Reference & CLI" +description: "Error codes, context cancellation, the full API tree, and the codegen CLI for the WaveHouse Go SDK." +--- + +Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: +cancellation, the error model behind every SDK call's `(T, error)` return, +the complete API tree at a glance, and the `wavehouse-codegen` tool that +ships with the module. Compare with the TypeScript SDK's +[Reference & CLI](/sdk/reference) page. + +## Context Cancellation + +Every non-streaming operation takes a `context.Context` as its first +argument — Go's equivalent of the TypeScript SDK's `AbortSignal` support. +Cancel it with a timeout or an explicit `cancel()`: + +```go +ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) +defer cancel() + +page, err := wh.From("clicks").Fetch(ctx) +var whErr *wavehouse.Error +if errors.As(err, &whErr) && whErr.Code == "ABORTED" { + fmt.Println("Request timed out") +} +``` + +Context cancellation returns immediately (no retry) with +`&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. + +Streams work differently: `.Stream(opts)` doesn't take a `context.Context` +at all — the returned `*StreamController` owns its own internal context and +background goroutine, torn down explicitly via `.Close()`. See +[Streaming](/sdk/go/streaming#streamoptions). + +--- + +## Error Handling + +The SDK never panics on API or network failures — every operation returns +`(T, error)`, and errors are always `*wavehouse.Error` (unwrap with +`errors.As`). This is the direct Go equivalent of the TypeScript SDK's "the +SDK never throws" guarantee. + +| Status | Code | Retryable | Description | +|--------|------|-----------|--------------| +| 400 | `HTTP_400` | No | Bad request (validation, missing fields) | +| 401 | `HTTP_401` | No | Missing or invalid JWT | +| 403 | `HTTP_403` | No | Insufficient permissions | +| 404 | `HTTP_404` | No | Table or pipe not found | +| 500 | `HTTP_500` | Yes | Server error (retried per `ClientOptions.MaxRetries`) | +| 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`) | +| 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | +| 0 | `ABORTED` | No | Request canceled via `context.Context` | + +```go +page, err := wh.From("clicks").Fetch(ctx) +if err != nil { + var whErr *wavehouse.Error + if errors.As(err, &whErr) { + fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } + return err +} +``` + +`wavehouse.IsRetryable(err)` is a shortcut for the `errors.As` + `.Retryable` +check above. + +Retries apply uniformly to every HTTP method the SDK issues (not just GET) — +matching the TypeScript SDK's `http.ts` behavior. For `/v1/ingest`, +at-least-once delivery on retry is a documented contract (see the API +docs' ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data) +note); dedup is the prescribed server-side safety net when duplicate +suppression matters. `/v1/admin/query` (raw SQL) is gated by `admin_role`, +so repeated execution on retry is an accepted risk for admin-only usage. + +--- + +## Full API Tree + +```text +NewClient(Config) → *Client +├── .From(table) → *TableRef +│ ├── .Fetch(ctx) → (*Page[map[string]any], error) +│ ├── .Select(...cols) → *QueryBuilder +│ │ ├── .Select() .SelectAll() .Where() .Count() .Sum() .Avg() .Min() .Max() +│ │ │ .CountDistinct() .Aggregate() .GroupBy() .OrderBy() +│ │ │ .Limit() .TimeRange() .CacheTTL() +│ │ ├── FetchTyped[Row](ctx, q) → (*Page[Row], error) // package-level generic func +│ │ ├── .FetchUntyped(ctx) → (*Page[map[string]any], error) +│ │ ├── .Stream(opts) → *StreamController +│ │ └── .LiveQuery(sub, opts) → *LiveQueryHandle +│ ├── .SelectAll() → *QueryBuilder +│ ├── .Insert(ctx, data) → (*InsertResult, error) +│ ├── .InsertNDJSON(ctx, ndjson) → (*InsertResult, error) +│ ├── .Schema(ctx) → (*TableSchema, error) +│ └── .Stream(opts) → *StreamController +├── .Pipe(name, params) → *PipeRef +│ ├── Fetch[Row](ctx, p) → ([]Row, error) // package-level generic func +│ ├── .FetchUntyped(ctx) → ([]map[string]any, error) +│ └── .Stream(opts) → *StreamController +├── .Pipes (admin) → *PipesNamespace +│ ├── .List(ctx) → ([]Pipe, error) +│ ├── .Get(ctx, name) → (*Pipe, error) +│ ├── .Set(ctx, name, PipeDef) → error +│ └── .Delete(ctx, name) → error +├── SQL[Row](ctx, client, query) → ([]Row, error) // package-level generic func, admin-only +├── .Schema (admin) → *SchemaNamespace +│ ├── .List(ctx) → (Schemas, error) +│ └── .Refresh(ctx) → error +├── .Policy (admin) → *PolicyNamespace +│ ├── .Get(ctx) → (*Policy, error) +│ ├── .Set(ctx, *Policy) → error +│ └── .Validate(ctx, *Policy) → (*ValidationResult, error) +├── .DLQ (admin) → *DLQNamespace +│ ├── .List(ctx) → (*DLQStats, error) +│ ├── .Table(ctx, name) → (*DLQStats, error) +│ └── .Stream(opts) → *StreamController // not yet functional server-side — #197 +└── .Sys → *SysNamespace + └── .Health(ctx) → error + +*StreamController +├── .Subscribe(*StreamSubscriber) → func() // unsubscribe +├── .Events() → <-chan StreamEvent // idiomatic Go alternative to an async iterator +├── .Close() +├── .Status() → StreamStatus +└── .Connected(ctx) → error // Go-only addition, blocks until live +``` + +## Codegen CLI + +Generate Go structs from a running WaveHouse instance. The module ships a +`wavehouse-codegen` command under `cmd/`: + +```bash +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ + --url http://localhost:8080 \ + --auth \ + --out ./db_types.go \ + --package myapp +``` + +Or, working inside this repo (`clients/go/`): + +```bash +go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go +``` + +Codegen reads `/v1/schema`, which is **admin-only**. Against a non-dev +server, pass an admin-role token with `--auth ` or the request is +denied with `403`. + +**Options:** + +| Flag | Description | Default | +|------|-------------|---------| +| `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | +| `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | +| `--auth`, `-a` | Bearer token (if auth required) | — | +| `--package`, `-p` | Go package name for the generated file | `main` | +| `--help`, `-h` | Show usage and exit | — | + +The output is run through `go/format` before being written — if a table or +column name would produce invalid Go source (rare, but possible with exotic +names), codegen fails loudly instead of writing broken code. + +**Example output:** + +```go +// Code generated by wavehouse-codegen. DO NOT EDIT. + +package myapp + +// ClicksRow represents a row in the "clicks" table. +type ClicksRow struct { + EventID string `json:"event_id"` + Page string `json:"page"` + UserID string `json:"user_id"` + DurationMS int `json:"duration_ms"` + ReceivedTimestamp string `json:"received_timestamp"` +} +``` + +Table and column names are converted to `PascalCase` for Go field/type names +(a leading digit gets an `X` prefix — e.g. a table named `2fa_events` +becomes `X2faEventsRow` — to stay a valid Go identifier). A column with +`has_default: true` in the schema gets `,omitempty` appended to its JSON +tag. + +**ClickHouse → Go type mapping:** + +| ClickHouse Type | Go Type | +|------------------|---------| +| `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | +| `Bool` | `bool` | +| `UInt8` / `UInt16` / `UInt32` / `UInt64` | `uint8` / `uint16` / `uint32` / `uint64` | +| `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | +| `Float32` | `float32` | +| `Float64` | `float64` | +| `Decimal*`, `UInt128`/`UInt256`, `Int128`/`Int256` | `string` (big numbers are strings in JSON) | +| `Nullable(T)` | `*T` | +| `LowCardinality(T)` | same as `T` | +| `Array(T)` | `[]T` | +| `Map(K, V)` | `map[K]V` (falls back to `map[string]any` if `K`/`V` can't be split) | +| anything unrecognized | `any` | + +This differs from the TypeScript SDK's mapping in one notable way: Go's +codegen preserves ClickHouse's integer **widths** (`UInt32` → `uint32`, not +a generic `number`), since Go — unlike TypeScript — has native fixed-width +integer types. + +## Testing + +The Go SDK ships with unit tests colocated in `clients/go/` (its own Go +module — `clients/go/go.mod` — separate from the root `WaveHouse` module), +plus a wire-format **conformance suite** +(`clients/go/conformance_test.go` + `clients/go/testdata/wire_cases.json`) +that replays a shared fixture of builder calls and asserts the Go SDK +produces the exact same HTTP method, path, content type, and body as the +TypeScript SDK for each one — keeping the two clients honest about the wire +format they both speak. + +```bash +cd clients/go +go test ./... +``` + +Unlike the TypeScript SDK, the Go SDK isn't (yet) wired into the repo's +`make test-e2e` harness — see the TypeScript SDK's +[E2E Testing](/sdk/reference#e2e-testing) section for that suite's +architecture, which the Go client doesn't currently participate in. diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md new file mode 100644 index 00000000..55b31243 --- /dev/null +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -0,0 +1,254 @@ +--- +title: "Go SDK Streaming & Live Queries" +description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in the WaveHouse Go SDK." +--- + +Real-time consumption with `github.com/Wave-RF/WaveHouse/clients/go`: SSE +event streams from tables, builders, and pipes, plus live queries that +backfill history before going live. Builders and table refs come from +[Queries](/sdk/go/queries). Compare with the TypeScript SDK's +[Streaming & Live Queries](/sdk/streaming) page — the two implement the same +protocol and mostly the same client-side filtering, but connection lifecycle +differs: Go streams are goroutine-backed and closed explicitly, not tied to +a `context.Context` or a browser's `EventSource`. + +## Streaming + +Streams use SSE (Server-Sent Events), parsed by hand over `net/http` (no +third-party SSE library — the SDK has zero runtime dependencies). + +### `*StreamController` + +Returned by `.Stream(opts)` on `*TableRef`, `*QueryBuilder`, `*PipeRef`, and +`*DLQNamespace` (the DLQ variant is not yet functional server-side — +[#197](https://github.com/Wave-RF/WaveHouse/issues/197)). Calling `.Stream` +returns immediately; the connection opens in a background goroutine. + +```go +stream := wh.From("clicks").Stream(&wavehouse.StreamOptions{ + Since: "2026-01-01T00:00:00Z", +}) +defer stream.Close() +``` + +### `.Subscribe(sub) → func()` + +Callback-based consumption. Returns an unsubscribe function. The +subscriber's `Status` callback fires immediately with the current status. + +```go +unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ + Next: func(e wavehouse.StreamEvent) { + // e: {Table: "clicks", Timestamp: "2026-...", Data: map[string]any{"page": "/", ...}} + fmt.Println("New event:", e.Data) + }, + Status: func(s wavehouse.StreamStatus) { + // s: StatusConnecting | StatusLive | StatusReconnecting | StatusClosed + updateIndicator(s) + }, + Error: func(err error) { + fmt.Println("Stream error:", err) + }, +}) + +// Cleanup — removes this subscriber; the connection stays open for any +// others and must still be closed with stream.Close() when you're done +// with the stream itself. +defer unsub() +``` + +### Channel-based consumption — `.Events()` + +The idiomatic Go alternative to the TypeScript SDK's async iterator: a +read-only channel, closed automatically when the stream shuts down. + +```go +stream := wh.From("clicks").Stream(nil) +defer stream.Close() + +for event := range stream.Events() { + fmt.Println(event.Table, event.Data) + if shouldStop { + break + } +} +``` + +:::caution[`break` does not close the stream] +Unlike the TypeScript SDK's async iterator — where breaking out of a +`for await` loop auto-closes the underlying connection — breaking a Go +`for range stream.Events()` loop only stops consuming from the channel; the +background goroutine and its HTTP connection keep running. Always pair a +stream with `defer stream.Close()` (or an explicit `stream.Close()` on every +exit path) regardless of which consumption style you use. +::: + +The channel is buffered (256 events); a slow consumer that never drains it +causes the SDK to **drop** new events for that channel rather than block the +stream's read loop (`.Subscribe` callbacks still fire per event +regardless of channel backpressure). + +### `.Close()` + +Explicitly close the stream and release its resources. Non-blocking — safe +to call from inside a subscriber callback (which runs on the stream's own +goroutine); it signals the goroutine to stop without waiting for it to +finish. + +```go +stream.Close() +``` + +### `.Status()` + +Returns the current `StreamStatus`. A method (not a field), since Go has no +JS-style reactive property access. + +```go +status := stream.Status() +``` + +### `.Connected(ctx)` + +**Go-only addition** — not present in the TypeScript SDK. Blocks until the +stream reaches `StatusLive` or `ctx` is canceled; returns an error if the +stream closes before connecting. Useful when you need to know a stream is +live before doing something else (e.g. before starting a producer in a +test). + +```go +ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) +defer cancel() +if err := stream.Connected(ctx); err != nil { + log.Fatal(err) +} +``` + +### `StreamOptions` + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `Since` | `string` | RFC3339 timestamp for gap-fill replay | + +There's no `Signal`/context field here — a stream isn't canceled by passing +a `context.Context` into `.Stream()`; call `.Close()` instead (see above). + +### `StreamEvent` + +```go +type StreamEvent struct { + Table string // table name (e.g. "clicks") + Timestamp string // received_timestamp (RFC3339Nano) + Data map[string]any // row data +} +``` + +### Transport Behavior + +| Transport | Reconnect | Protocol | +| --------- | --------- | -------- | +| SSE | Automatic, with exponential backoff (capped at 30s) and gap-fill replay via the last-seen event ID | HTTP/2 recommended | + +Auth is sent as an `Authorization: Bearer` header on every stream +(re)connection — see +[the note in the Getting Started guide](/sdk/go#creating-a-client). The +TypeScript SDK's "more than 5 concurrent connections" warning is a +browser-specific `EventSource` limit and doesn't apply here. + +### Client-Side Stream Filtering + +When a `*QueryBuilder` with `.Where()` filters or `.Select()` columns calls +`.Stream()`, the returned stream applies those filters client-side: + +```go +stream := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Stream(nil) + +// Only events where page == "/home" are emitted, with only page + button fields +``` + +Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, +`not_like` — the same `FilterOp` set `.Where()` takes everywhere. `like` / +`not_like` match SQL LIKE semantics (`%` → any run of characters, `_` → any +single character), case-insensitively. `in` accepts any Go slice type on +the right-hand side (`[]string`, `[]int`, `[]any`, ...), not just `[]any`. + +--- + +## Live Queries + +Live queries combine a historical backfill (`.FetchUntyped`) with a +real-time stream, providing a seamless initial-load + live-updates +experience. Only available on `*QueryBuilder` (there's no `TableRef.LiveQuery` +shortcut, matching the TypeScript SDK). + +```go +lq := wh.From("clicks"). + SelectAll(). + Where("page", wavehouse.OpEq, "/home"). + OrderBy("received_timestamp", "desc"). + Limit(100). + LiveQuery(&wavehouse.StreamSubscriber{ + Initial: func(rows []map[string]any, err error) { + // Called once with the historical backfill. + setRows(rows) + }, + Next: func(e wavehouse.StreamEvent) { + // Called for each live event after backfill. + addRow(e.Data) + }, + Error: func(err error) { + log.Println(err) + }, + }, nil) + +// Cleanup +defer lq.Close() +``` + +### `StreamSubscriber` + +```go +type StreamSubscriber struct { + // Initial is called once with historical backfill data (live queries only). + Initial func(rows []map[string]any, err error) + // Next is called for each live event. + Next func(event StreamEvent) + // Status is called when the connection status changes. + Status func(status StreamStatus) + // Error is called on stream errors. + Error func(err error) +} +``` + +:::note[`Initial` is always untyped] +Unlike the TypeScript SDK's `initial: (result: Result) => void`, the Go +SDK's `LiveQuery` doesn't accept a type parameter — `Initial` always +receives `[]map[string]any` plus a plain `error`, even if you'd otherwise +use `wavehouse.FetchTyped[Row]` for the same query outside a live query. +Decode into your own type inside the callback if you need one. +::: + +### How it works + +1. Subscribes to the stream **immediately** and buffers incoming events. +2. Runs the `.FetchUntyped(ctx)` query for historical data, calls + `sub.Initial(rows, err)` with the result. +3. Deduplicates buffered events by comparing timestamps against the latest + historical row's `received_timestamp`. +4. Flushes remaining buffered events (re-checking for anything that arrived + mid-flush) and switches to live mode. + +This "stream-first" approach ensures no events are lost between the fetch +and stream start. + +### `.Close()` + +Shuts down the live query and its underlying stream. Safe to call more than +once (idempotent via `sync.Once`). + +```go +lq.Close() +``` diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 01c5fb84..aa4ca864 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -7,6 +7,15 @@ import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components `@wavehouse/sdk` — Zero-dependency TypeScript client for WaveHouse. +:::tip[Writing Go instead?] +WaveHouse also ships an official Go SDK +(`github.com/Wave-RF/WaveHouse/clients/go`) — zero third-party dependencies, +`context.Context`-first, generics for typed rows. See the +[Go SDK docs](/sdk/go). The two clients speak the same wire format, so +everything below about tables, the query builder, streaming, and admin +endpoints carries over conceptually — only the language idioms differ. +::: + ## Installation @@ -411,3 +420,15 @@ The full error-code table lives in href="/sdk/reference" /> + +## Go SDK + +Prefer Go? The same server, the same wire format, an idiomatic Go client: + + + + From 61b436f0de0ae0f2f2e60618df61afa6fccb500d Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 15:13:40 -0400 Subject: [PATCH 03/40] ci(sdk): wire Go SDK into CI and update AGENTS.md - Add test-go-sdk to CI unit job - AGENTS.md: Go SDK file structure + feature parity table - lint-go-sdk already wired via verify-parallel in Makefile --- .github/workflows/ci.yml | 4 ++-- AGENTS.md | 30 +++++++++++++++++++----------- 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 591820e8..8646bf53 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -204,8 +204,8 @@ jobs: uses: ./.github/actions/setup-env with: go-cache-suffix: "-unit" - - name: Run Go unit tests + SDK vitest tests - run: make test-unit test-ts COV_DEFER=1 + - name: Run Go unit tests + SDK vitest + Go SDK tests + run: make test-unit test-ts test-go-sdk COV_DEFER=1 - name: Upload coverage fragment uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: diff --git a/AGENTS.md b/AGENTS.md index 1e3ec198..4ff5f47c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,7 +60,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 11. **Hasura-style access control: fail-closed (security)** — `policy.IsAdmin` (role == `admin_role`, **exact case-sensitive**, default `"admin"`) is the single admin check, shared by `Evaluate`/`ResolveRole`/`Validate`/the `/v1/admin` gate/`RoleAllowed`. Empty/absent role matches nothing (no `"*"` wildcard); `Validate` rejects empty role keys; a `nil` policy (deleted) denies **everyone incl. admin** via a role — a total lockout for token-based callers, so bootstrap from the policy file, never an implicit admin grant (**exception:** the operator key's `auth.IsOperator` bit passes the `/v1/admin` gate even under a `nil` policy — a deliberate break-glass restore over HTTP, see #7). `default_role` is the one sanctioned roleless exception (`ResolveRole` maps empty → it pre-eval); `default_role == admin_role` is permitted but dev-only and loudly warned (`policy.DefaultRoleGrantsAdmin`). Preserve when touching `internal/policy` (policy twin of #13; see #159). Detail: architecture.md § `policy/`. 12. **Structured queries: column authz fail-closed (security)** — `POST /v1/query?table={table}`: typed AST validated against schema, permission-enforced, timestamp-bucketed for cache, `DefaultMaxRows` (10,000) cap. Every column reference — projection, aggregation args, `filters`, `group_by`, `order_by`, `time_range` — is authorized inside `query.Build` (the single chokepoint that enumerates them all), so no clause can skip the role's `allow_columns`/`deny_columns` check (#223). A `select_all` read by a *column-restricted* role expands to its allowed columns via `policy.AllowedProjection`, never a bare `SELECT *`; *unrestricted*/admin roles keep `SELECT *` (`policy.RestrictsColumns` decides). Omitting `columns` selects nothing (`ErrEmptyProjection` → `200 []`); `["*"]` is the literal column `*` (schema-gated, not a wildcard); a table-granted role with no readable columns fails closed (`ErrNoReadableColumns` → `403`). Structured and live-stream (`stream.filterColumns`) reads share the one per-column decision `policy.IsColumnAllowed`, so column visibility can't drift. Preserve when touching `internal/query` or the structured-query handler. Detail: architecture.md § `query/`. 13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). Detail: architecture.md § `pipes/`. -14. **TypeScript SDK** — `@wavehouse/sdk`: zero-dep client, typed query builder, real-time SSE, live queries (incrementable/decomposable/poll aggregation), codegen CLI. The canonical client (see §SDK Sync). +14. **Client SDKs** — TypeScript (`@wavehouse/sdk` in `clients/ts/`) and Go (`wavehouse-go` in `clients/go/`) are both canonical, officially supported clients with full API-tree parity. Zero third-party runtime dependencies in both. Each ships a typed query builder, real-time SSE streaming, live queries, and a codegen CLI. See §SDK Sync. 15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. The OTLP endpoint/TLS/custom-CA/mTLS/headers are delegated to the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` env vars — `InitProvider` passes **no** endpoint/header options. Known gap, intentionally not patched in WaveHouse app code: the pinned gRPC logs exporter (`otlploggrpc` v0.19/v0.20) ignores the env TLS-cert vars, so a custom/private CA and mutual TLS apply to traces/metrics but **not** the logs signal (public-CA/system-roots TLS and plaintext still work for logs) — upstream bug open-telemetry/opentelemetry-go#6661. A malformed `OTEL_EXPORTER_OTLP_HEADERS` is logged and skipped by the SDK (fail-soft), not fatal. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`. 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops. @@ -331,22 +331,22 @@ Diagrams render inside the Starlight content column (~46–58rem wide) as build- ## SDK Sync -The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) is the canonical client and ships from this repo. When backend changes alter the public API surface, the SDK needs corresponding updates. The `pre-commit` git hook flags likely misses informationally; consult this table when deciding what to update. +The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) and Go SDK (`wavehouse-go` in `clients/go/`) are both canonical, officially supported clients. Both ship from this repo with full API-tree parity. When backend changes alter the public API surface, both SDKs need corresponding updates. The `pre-commit` git hook flags likely misses informationally; consult this table when deciding what to update. | Backend change | SDK considerations | | -------------- | ------------------ | -| New user-facing API endpoint | Add a typed client method (in `clients/ts/src/client.ts` or the relevant subsystem file: `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`, etc.); update the matching SDK doc page under `docs/src/content/docs/sdk/` (`queries`, `streaming`, `pipes`, `admin`, or `reference` by topic — plus the API tree in `reference.md`) | -| Change to JWT auth / role extraction | Update auth handling in `clients/ts/src/http.ts` and types in `clients/ts/src/client.ts` | -| Change to `EventMessage` / ingest event format | Update payload types in `clients/ts/src/` (some are codegen-regenerated — re-run the SDK codegen CLI) | -| New / changed structured query AST | Update `clients/ts/src/query-builder.ts` types + builder methods | -| Change to live-query aggregation classification | Update live-query helpers in `clients/ts/src/stream/` | -| Named pipes API change | Update `clients/ts/src/pipes.ts` | -| Policy / access-control change | Update `clients/ts/src/policy.ts` | -| ClickHouse schema-driven type changes | Re-run the SDK codegen CLI; commit regenerated types | +| New user-facing API endpoint | Add a typed client method in **both** SDKs (TS: `clients/ts/src/` — `client.ts`, `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`; Go: `clients/go/` — corresponding file). Update doc pages under `docs/src/content/docs/sdk/` for both `ts/` and `go/`. Add a wire case to `clients/go/testdata/wire_cases.json` with dispatch in both conformance runners. | +| Change to JWT auth / role extraction | TS: `clients/ts/src/http.ts` + `client.ts`. Go: `clients/go/http.go` + `wavehouse.go`. | +| Change to `EventMessage` / ingest event format | Update payload types in both SDKs (some are codegen-regenerated — re-run both codegen CLIs). | +| New / changed structured query AST | TS: `clients/ts/src/query-builder.ts`. Go: `clients/go/query_builder.go` + `types.go`. | +| Change to live-query aggregation classification | TS: `clients/ts/src/stream/`. Go: `clients/go/live_query.go`. | +| Named pipes API change | TS: `clients/ts/src/pipes.ts`. Go: `clients/go/pipes.go`. | +| Policy / access-control change | TS: `clients/ts/src/policy.ts`. Go: `clients/go/policy.go`. | +| ClickHouse schema-driven type changes | Re-run both SDK codegen CLIs; commit regenerated types. | Internal-only backend changes (middleware refactors, observability internals, dedup implementation, sweeper logic, NATS plumbing) generally don't need SDK updates. Use judgement — table above is the source of truth; nothing automated nudges you. -**The decision test**: would a `@wavehouse/sdk` user's *code* need to change to take advantage of (or be compatible with) this change? If yes, SDK update needed. If no (purely internal optimization), no. +**The decision test**: would a user's *code* need to change to take advantage of (or be compatible with) this change? If yes, both SDKs need updates. If no (purely internal optimization), no. ## Common Tasks @@ -387,6 +387,14 @@ Internal-only backend changes (middleware refactors, observability internals, de ```text cmd/ → Binary entry points (thin — just wiring) +clients/ts/ → TypeScript SDK (@wavehouse/sdk) +clients/go/ → Go SDK (wavehouse-go) + wavehouse.go, http.go, errors.go, types.go → Client core (constructor, transport, errors, shared types) + query_builder.go, table.go → Structured query builder + per-table typed client + stream.go, live_query.go → SSE streaming + live queries + pipes.go, policy.go, schema.go, dlq.go, sys.go → Subsystem clients (pipes, policy, schema, DLQ, health) + cmd/wavehouse-codegen/main.go → Codegen CLI + testdata/wire_cases.json → Wire-format conformance fixtures internal/api/ → HTTP layer (handlers, router, middleware, schema/DLQ/policy/pipes endpoints) internal/auth/ → JWT/JWKS authentication middleware (HMAC or JWKS, role extraction from claims) internal/cache/ → Caching (interface + L1/L2/tiered implementations) From 28eb1cf3a7845349738361d2c8f58ed359a5e950 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 16:06:33 -0400 Subject: [PATCH 04/40] chore: trim verbose comments across SDK, Makefile, and docs --- Makefile | 20 +++----------------- README.md | 3 +-- clients/go/http.go | 10 ++-------- clients/go/live_query.go | 21 +++++---------------- clients/go/stream.go | 7 ++----- docs/src/config/sidebar.ts | 12 +++--------- 6 files changed, 16 insertions(+), 57 deletions(-) diff --git a/Makefile b/Makefile index 3174e5c6..0c614912 100644 --- a/Makefile +++ b/Makefile @@ -428,13 +428,7 @@ endif tidy: ## Verify go.mod/go.sum are tidy (run `make fix` to apply) $(call run,go.mod tidy,go mod tidy -diff,run make fix to tidy go.mod and go.sum) -# verify-go-sdk: static checks for clients/go/ — a nested Go module (its own -# go.mod), so it's invisible to `go list ./...` from the root and every leaf -# above (GO_DIRS, lint-go, vulncheck, tidy) silently skips it. Scoped and run -# explicitly here instead. `go vet` needs its own module context (cd -# clients/go), but gofumpt is a pure syntax formatter with no module -# resolution of its own, so the repo-pinned $(GOFUMPT) binary can format it -# directly by path from the root — no second tool pin needed. +# verify-go-sdk: nested module at clients/go/ — invisible to root go list. .PHONY: verify-go-sdk verify-go-sdk: ## Static checks for the Go SDK (clients/go, a nested module) — go vet + gofumpt $(call run,go vet (Go SDK),cd clients/go && go vet ./...,) @@ -732,21 +726,13 @@ test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against sui # test-go-sdk: unit tests for clients/go/ — a nested Go module (its own # go.mod), so it's outside test-unit's ./internal/... ./cmd/... scope and -# needs its own leaf; a root `go test ./...` wouldn't reach it either. Zero -# third-party runtime or test deps (stdlib only, no go.sum), so no -# go-mod-download prereq. Not yet wired into the Go/TS coverage gate — see -# verify-go-sdk above for the same "nested module, own leaf" reasoning. +# test-go-sdk: nested module — needs its own target. .PHONY: test-go-sdk test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" @cd clients/go && go test ./... -# test-go-sdk-e2e: runs the Go SDK's E2E tests against a live WaveHouse -# instance. Requires a running server (e.g. `make dev` in the main repo). -# Env vars: -# WAVEHOUSE_URL base URL of the server (default: http://localhost:8080) -# WAVEHOUSE_AUTH bearer token for auth (optional; omit for default_role) -# The tests skip gracefully when the server is unreachable. +# test-go-sdk-e2e: E2E against live server. WAVEHOUSE_URL + WAVEHOUSE_AUTH env vars. .PHONY: test-go-sdk-e2e test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVEHOUSE_URL, WAVEHOUSE_AUTH) @printf "$(CYAN)==> Running Go SDK E2E tests...$(RESET)\n" diff --git a/README.md b/README.md index d9e658c4..dc63707d 100644 --- a/README.md +++ b/README.md @@ -60,8 +60,7 @@ If you're building user-facing analytics, WaveHouse is like **Supabase for Click - **Query** — in-process Ristretto cache + `singleflight` coalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). - **Real-time** — native SSE push, broadcast *before* the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients. - **Security** — Hasura-style per-table, per-role column + row policies with JWT claim templating, stored in NATS KV. -- **Client** — `@wavehouse/sdk`: zero-dependency TypeScript client with query builder, live queries, streaming, and schema codegen. -- **Go SDK** — `go get github.com/Wave-RF/WaveHouse/clients/go`: full API-tree parity with the TS SDK — ingest, query, streaming, and policy management from Go. +- **Client SDKs** — TypeScript (`@wavehouse/sdk`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`): zero-dependency clients with query builder, live queries, streaming, and schema codegen. ## 📊 How it compares diff --git a/clients/go/http.go b/clients/go/http.go index 52aae6b4..9646a0a2 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -68,14 +68,8 @@ func doRequest(hctx httpContext, ctx context.Context, opts requestOptions, dst a var lastErr error maxAttempts := hctx.maxRetries + 1 - // Retries below are not restricted by HTTP method — this matches the TS - // SDK's http.ts, which retries POST the same as GET on network errors, - // 503/Retry-After, and other retryable 5xx. For /v1/ingest, at-least-once - // delivery on retry is a documented contract (see docs/api.md's - // "At-least-once on retry" note); dedup is the prescribed server-side - // safety net when duplicate suppression matters. The only other mutation - // path, /v1/admin/query, is gated by admin_role, so repeated execution on - // retry is assumed to be an accepted risk for admin-only raw SQL. + // Retries all methods including POST. For /v1/ingest, at-least-once delivery + // is the documented contract; dedup is the server-side safety net. for attempt := range maxAttempts { var bodyReader io.Reader if bodyBytes != nil { diff --git a/clients/go/live_query.go b/clients/go/live_query.go index fad4e80a..97cc6c58 100644 --- a/clients/go/live_query.go +++ b/clients/go/live_query.go @@ -89,16 +89,9 @@ func newLiveQuery( } } - // Step 5: Flush buffered events newer than the fetch. - // - // buffering stays true for the whole flush: events that arrive - // concurrently (after Subscribe's Next handler releases mu but - // before we're done here) must keep landing in buffer rather than - // being dispatched directly by the live path, or two goroutines - // could call sub.Next at once. We only flip buffering to false - // once a lock-protected check finds the buffer empty, which - // guarantees no event is ever handed to sub.Next by both paths - // and that delivery stays in arrival order. + // Step 5: Flush buffered events. buffering stays true until the + // buffer is provably empty under the lock — prevents concurrent + // sub.Next calls and preserves delivery order. for { mu.Lock() if closed { @@ -121,12 +114,8 @@ func newLiveQuery( if c { return } - // Use <= (not <) to filter events whose timestamp matches the last - // historical row — those rows were already delivered in the backfill - // response. If two distinct events share a timestamp and only one - // appeared in the backfill, the duplicate is lost; this matches the - // TS SDK's dedup behavior and is acceptable because received_timestamp - // has sub-millisecond precision in practice. + // <= dedupes events already delivered in the backfill. + // Sub-millisecond received_timestamp precision makes collisions rare. if lastTimestamp != "" && event.Timestamp <= lastTimestamp { continue } diff --git a/clients/go/stream.go b/clients/go/stream.go index e30d6346..770550d7 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -56,11 +56,8 @@ func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { currentStatus := sc.status sc.mu.Unlock() - // Benign race: if setStatus fires between the unlock above and the - // callback below, the subscriber may see a stale status here. This is - // harmless because setStatus also invokes the subscriber's callback, - // so the subscriber will receive the up-to-date status immediately - // after. Matches the TS SDK's registration behavior. + // Benign race: setStatus also calls the subscriber, so a stale + // status here is immediately followed by the correct one. if sub.Status != nil { sub.Status(currentStatus) } diff --git a/docs/src/config/sidebar.ts b/docs/src/config/sidebar.ts index f114c692..b4ec5da3 100644 --- a/docs/src/config/sidebar.ts +++ b/docs/src/config/sidebar.ts @@ -25,15 +25,9 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ items: [ { label: "API Reference", slug: "api" }, { - // Topic-first SDK pages: the multi-language plan on record (PR #313) - // was for a second language to grow on these - // shared pages instead of a parallel tree. The Go SDK launched as - // its own docs/src/content/docs/sdk/go/* tree instead — its API - // shape (context.Context, (T, error), generics on package-level - // funcs) diverges enough from the TS builder that shared prose read - // worse than dedicated pages. Revisit folding these into tabs if a - // third language lands and the duplication becomes a maintenance - // cost. + // Separate trees per SDK — API shapes diverge enough that shared + // prose reads worse than dedicated pages. Revisit with tabs if a + // third language lands. label: "TypeScript SDK", items: [ { label: "Overview", slug: "sdk" }, From b66413a5867ee2bd4bff23bb2016bcaa47f1d0dc Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 16:16:35 -0400 Subject: [PATCH 05/40] fix(sdk): resolve 18 golangci-lint findings in Go SDK - context.Context as first param in doRequest (revive) - Checked all json Encode/Decode/Unmarshal returns (errcheck) - Wrapped defer Body.Close with error discard (errcheck) - if-else chain to switch in buildAST (gocritic) - Tagged switch on r.Method in test (staticcheck) - Renamed built-in shadow cap to capt (revive) - Removed wasted msg assignment (wastedassign) - WriteFile 0o644 to 0o600 (gosec) - nolint:gosec for cancel called in Close (gosec) --- clients/go/client_test.go | 6 ++-- clients/go/cmd/wavehouse-codegen/main.go | 4 +-- clients/go/conformance_test.go | 44 ++++++++++++------------ clients/go/dlq.go | 4 +-- clients/go/errors.go | 2 +- clients/go/http.go | 6 ++-- clients/go/http_test.go | 28 +++++++-------- clients/go/live_query.go | 2 +- clients/go/namespaces_test.go | 26 +++++++------- clients/go/pipes.go | 10 +++--- clients/go/policy.go | 6 ++-- clients/go/query_builder.go | 9 ++--- clients/go/query_builder_test.go | 8 ++--- clients/go/schema.go | 4 +-- clients/go/stream.go | 2 +- clients/go/sys.go | 2 +- clients/go/table.go | 6 ++-- clients/go/table_test.go | 16 ++++----- clients/go/wavehouse.go | 2 +- 19 files changed, 94 insertions(+), 93 deletions(-) diff --git a/clients/go/client_test.go b/clients/go/client_test.go index 4f57507f..2b5239e1 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -60,7 +60,7 @@ func TestClient_From(t *testing.T) { if r.URL.Query().Get("table") != "events" { t.Errorf("want table=events, got %s", r.URL.Query().Get("table")) } - json.NewEncoder(w).Encode([]map[string]any{}) + _ = json.NewEncoder(w).Encode([]map[string]any{}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) _, _ = c.From("events").Fetch(context.Background()) @@ -72,11 +72,11 @@ func TestClient_SQL(t *testing.T) { t.Errorf("want /v1/admin/query, got %s", r.URL.Path) } var body map[string]string - json.NewDecoder(r.Body).Decode(&body) + _ = json.NewDecoder(r.Body).Decode(&body) if body["sql"] != "SELECT 1" { t.Errorf("want sql=SELECT 1, got %s", body["sql"]) } - json.NewEncoder(w).Encode([]map[string]any{{"x": 1}}) + _ = json.NewEncoder(w).Encode([]map[string]any{{"x": 1}}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) rows, err := SQL[map[string]any](context.Background(), c, "SELECT 1") diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 1df138b9..213710e8 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -89,7 +89,7 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc if err != nil { return nil, err } - defer resp.Body.Close() + defer func() { _ = resp.Body.Close() }() if resp.StatusCode != 200 { return nil, fmt.Errorf("schema fetch failed: HTTP %d", resp.StatusCode) } @@ -329,7 +329,7 @@ func main() { os.Exit(1) } - if err := os.WriteFile(args.out, formatted, 0o644); err != nil { + if err := os.WriteFile(args.out, formatted, 0o600); err != nil { fmt.Fprintf(os.Stderr, "Error writing %s: %v\n", args.out, err) os.Exit(1) } diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index 5f5edf74..8809457a 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -64,31 +64,31 @@ func TestConformance_WireFormat(t *testing.T) { for _, tc := range cases { t.Run(tc.Name, func(t *testing.T) { - var cap captured + var capt captured srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - cap.method = r.Method - cap.path = r.URL.RequestURI() - cap.contentType = r.Header.Get("Content-Type") + capt.method = r.Method + capt.path = r.URL.RequestURI() + capt.contentType = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) - cap.body = string(raw) + capt.body = string(raw) // Return valid JSON so the SDK doesn't error on decode. w.Header().Set("Content-Type", "application/json") switch { case strings.HasPrefix(r.URL.Path, "/v1/dlq"): - json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{}, Total: 0}) + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{}, Total: 0}) case strings.HasPrefix(r.URL.Path, "/v1/schema") && r.Method == "GET": - json.NewEncoder(w).Encode([]TableSchema{}) + _ = json.NewEncoder(w).Encode([]TableSchema{}) case r.URL.Path == "/v1/admin/policy/validate" && r.Method == "POST": - json.NewEncoder(w).Encode(ValidationResult{Valid: true}) + _ = json.NewEncoder(w).Encode(ValidationResult{Valid: true}) case strings.HasPrefix(r.URL.Path, "/v1/admin/policy") && r.Method == "GET": - json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) + _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) case strings.HasPrefix(r.URL.Path, "/v1/admin/pipes/") && r.Method == "GET": - json.NewEncoder(w).Encode(Pipe{Name: "test", SQL: "SELECT 1"}) + _ = json.NewEncoder(w).Encode(Pipe{Name: "test", SQL: "SELECT 1"}) case r.URL.Path == "/v1/admin/pipes" && r.Method == "GET": - json.NewEncoder(w).Encode([]Pipe{}) + _ = json.NewEncoder(w).Encode([]Pipe{}) default: - json.NewEncoder(w).Encode([]map[string]any{}) + _ = json.NewEncoder(w).Encode([]map[string]any{}) } })) defer srv.Close() @@ -186,29 +186,29 @@ func TestConformance_WireFormat(t *testing.T) { } // Verify method. - if tc.ExpectedMethod != "" && cap.method != tc.ExpectedMethod { - t.Errorf("method: want %s, got %s", tc.ExpectedMethod, cap.method) + if tc.ExpectedMethod != "" && capt.method != tc.ExpectedMethod { + t.Errorf("method: want %s, got %s", tc.ExpectedMethod, capt.method) } // Verify path. if tc.ExpectedPath != "" { // Normalize: the SDK may use different encoding (+ vs %20). wantPath := normalizePath(tc.ExpectedPath) - gotPath := normalizePath(cap.path) + gotPath := normalizePath(capt.path) if wantPath != gotPath { - t.Errorf("path: want %s, got %s", tc.ExpectedPath, cap.path) + t.Errorf("path: want %s, got %s", tc.ExpectedPath, capt.path) } } // Verify content type. - if tc.ExpectedContentType != "" && cap.contentType != tc.ExpectedContentType { - t.Errorf("content-type: want %s, got %s", tc.ExpectedContentType, cap.contentType) + if tc.ExpectedContentType != "" && capt.contentType != tc.ExpectedContentType { + t.Errorf("content-type: want %s, got %s", tc.ExpectedContentType, capt.contentType) } // Verify raw body (for NDJSON). if tc.ExpectedRawBody != nil { - if cap.body != *tc.ExpectedRawBody { - t.Errorf("raw body:\n want: %s\n got: %s", *tc.ExpectedRawBody, cap.body) + if capt.body != *tc.ExpectedRawBody { + t.Errorf("raw body:\n want: %s\n got: %s", *tc.ExpectedRawBody, capt.body) } return } @@ -219,8 +219,8 @@ func TestConformance_WireFormat(t *testing.T) { if err := json.Unmarshal(tc.ExpectedBody, &want); err != nil { t.Fatalf("parse expected_body: %v", err) } - if err := json.Unmarshal([]byte(cap.body), &got); err != nil { - t.Fatalf("parse captured body: %v (body: %s)", err, cap.body) + if err := json.Unmarshal([]byte(capt.body), &got); err != nil { + t.Fatalf("parse captured body: %v (body: %s)", err, capt.body) } if !deepEqualJSON(want, got) { wantJSON, _ := json.MarshalIndent(want, "", " ") diff --git a/clients/go/dlq.go b/clients/go/dlq.go index 01248b2c..b523ab33 100644 --- a/clients/go/dlq.go +++ b/clients/go/dlq.go @@ -14,7 +14,7 @@ type DLQNamespace struct { // List returns DLQ statistics (message counts per table). Admin-only. func (d *DLQNamespace) List(ctx context.Context) (*DLQStats, error) { var stats DLQStats - if err := doRequest(d.ctx, ctx, requestOptions{ + if err := doRequest(ctx, d.ctx, requestOptions{ method: "GET", path: "/v1/dlq/stats", }, &stats); err != nil { @@ -26,7 +26,7 @@ func (d *DLQNamespace) List(ctx context.Context) (*DLQStats, error) { // Table returns DLQ stats filtered by table name. Admin-only. func (d *DLQNamespace) Table(ctx context.Context, name string) (*DLQStats, error) { var stats DLQStats - if err := doRequest(d.ctx, ctx, requestOptions{ + if err := doRequest(ctx, d.ctx, requestOptions{ method: "GET", path: "/v1/dlq/stats", params: url.Values{"table": {name}}, diff --git a/clients/go/errors.go b/clients/go/errors.go index 1a161a1b..f1fb48ac 100644 --- a/clients/go/errors.go +++ b/clients/go/errors.go @@ -47,7 +47,7 @@ func parseErrorResponse(res *http.Response) *Error { _ = json.Unmarshal(raw, &body) } - msg := "" + var msg string if s, ok := body["error"].(string); ok { msg = s } else if s, ok := body["message"].(string); ok { diff --git a/clients/go/http.go b/clients/go/http.go index 9646a0a2..c7dc1c11 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -34,7 +34,7 @@ type requestOptions struct { // doRequest is the internal fetch wrapper with auth, retry, and backoff. // It decodes the response body into dst (unless dst is nil). -func doRequest(hctx httpContext, ctx context.Context, opts requestOptions, dst any) error { +func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst any) error { reqURL := buildURL(hctx.baseURL, opts.path, opts.params) ct := opts.contentType if ct == "" { @@ -107,7 +107,7 @@ func doRequest(hctx httpContext, ctx context.Context, opts requestOptions, dst a } if res.StatusCode >= 200 && res.StatusCode < 300 { - defer res.Body.Close() + defer func() { _ = res.Body.Close() }() if dst == nil { _, _ = io.Copy(io.Discard, res.Body) return nil @@ -131,7 +131,7 @@ func doRequest(hctx httpContext, ctx context.Context, opts requestOptions, dst a } apiErr := parseErrorResponse(res) - res.Body.Close() + _ = res.Body.Close() // 503 with Retry-After: wait the specified duration. if res.StatusCode == http.StatusServiceUnavailable { diff --git a/clients/go/http_test.go b/clients/go/http_test.go index 7ce4c964..5a095c62 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -22,11 +22,11 @@ func testCtx(handler http.Handler) httpContext { func TestDoRequest_SuccessfulGET(t *testing.T) { hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") - json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) })) var result map[string]string - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", path: "/health", }, &result) @@ -43,11 +43,11 @@ func TestDoRequest_POSTWithBody(t *testing.T) { var gotCT string hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotCT = r.Header.Get("Content-Type") - json.NewDecoder(r.Body).Decode(&gotBody) + _ = json.NewDecoder(r.Body).Decode(&gotBody) w.WriteHeader(200) })) - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "POST", path: "/v1/ingest", body: map[string]string{"page": "/home"}, @@ -71,10 +71,10 @@ func TestDoRequest_RawBody(t *testing.T) { raw := make([]byte, 1024) n, _ := r.Body.Read(raw) gotBody = string(raw[:n]) - json.NewEncoder(w).Encode(map[string]int{"total": 1}) + _ = json.NewEncoder(w).Encode(map[string]int{"total": 1}) })) - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "POST", path: "/v1/ingest", rawBody: `{"page":"/a"}`, @@ -99,7 +99,7 @@ func TestDoRequest_AuthInjection(t *testing.T) { })) hctx.auth = StaticToken("my-token") - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", path: "/v1/schema", }, nil) @@ -117,11 +117,11 @@ func TestDoRequest_4xxNotRetried(t *testing.T) { count.Add(1) w.Header().Set("Content-Type", "application/json") w.WriteHeader(404) - json.NewEncoder(w).Encode(map[string]string{"error": "not found"}) + _ = json.NewEncoder(w).Encode(map[string]string{"error": "not found"}) })) hctx.maxRetries = 2 - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", path: "/v1/schema", }, nil) @@ -141,15 +141,15 @@ func TestDoRequest_5xxRetried(t *testing.T) { if n < 3 { w.Header().Set("Content-Type", "application/json") w.WriteHeader(500) - json.NewEncoder(w).Encode(map[string]string{"error": "internal"}) + _ = json.NewEncoder(w).Encode(map[string]string{"error": "internal"}) return } - json.NewEncoder(w).Encode(map[string]string{"ok": "true"}) + _ = json.NewEncoder(w).Encode(map[string]string{"ok": "true"}) })) hctx.maxRetries = 2 var result map[string]string - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", path: "/health", }, &result) @@ -169,7 +169,7 @@ func TestDoRequest_AbortedOnCancel(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) cancel() // cancel immediately - err := doRequest(hctx, ctx, requestOptions{ + err := doRequest(ctx, hctx, requestOptions{ method: "GET", path: "/health", }, nil) @@ -185,7 +185,7 @@ func TestDoRequest_EmptyResponse(t *testing.T) { })) var result map[string]string - err := doRequest(hctx, context.Background(), requestOptions{ + err := doRequest(context.Background(), hctx, requestOptions{ method: "POST", path: "/v1/schema/refresh", }, &result) diff --git a/clients/go/live_query.go b/clients/go/live_query.go index 97cc6c58..d6c665fe 100644 --- a/clients/go/live_query.go +++ b/clients/go/live_query.go @@ -21,7 +21,7 @@ func newLiveQuery( sub *StreamSubscriber, filters []QueryFilter, ) *LiveQueryHandle { - ctx, cancel := context.WithCancel(context.Background()) + ctx, cancel := context.WithCancel(context.Background()) //nolint:gosec // cancel is called in Close() lq := &LiveQueryHandle{ stream: stream, cancel: cancel, diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index 84bc4b2f..18ff1b39 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -35,7 +35,7 @@ func TestSchemaNamespace_List(t *testing.T) { if r.URL.Path != "/v1/schema" { t.Errorf("want /v1/schema, got %s", r.URL.Path) } - json.NewEncoder(w).Encode([]TableSchema{ + _ = json.NewEncoder(w).Encode([]TableSchema{ {Name: "clicks", Columns: []Column{{Name: "page", Type: "String"}}}, }) })) @@ -65,13 +65,13 @@ func TestSchemaNamespace_Refresh(t *testing.T) { func TestPolicyNamespace_GetSetValidate(t *testing.T) { c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - switch { - case r.Method == "GET": - json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) - case r.Method == "PUT": + switch r.Method { + case "GET": + _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) + case "PUT": w.WriteHeader(200) - case r.Method == "POST": - json.NewEncoder(w).Encode(ValidationResult{Valid: true}) + case "POST": + _ = json.NewEncoder(w).Encode(ValidationResult{Valid: true}) } })) @@ -99,7 +99,7 @@ func TestPolicyNamespace_GetSetValidate(t *testing.T) { func TestDLQNamespace_List(t *testing.T) { c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) })) stats, err := c.DLQ.List(context.Background()) if err != nil { @@ -114,7 +114,7 @@ func TestDLQNamespace_Table(t *testing.T) { var gotParam string c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotParam = r.URL.Query().Get("table") - json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) })) _, err := c.DLQ.Table(context.Background(), "clicks") if err != nil { @@ -130,9 +130,9 @@ func TestPipesNamespace_CRUD(t *testing.T) { switch r.Method { case "GET": if r.URL.Path == "/v1/admin/pipes" { - json.NewEncoder(w).Encode([]Pipe{{Name: "p1", SQL: "SELECT 1"}}) + _ = json.NewEncoder(w).Encode([]Pipe{{Name: "p1", SQL: "SELECT 1"}}) } else { - json.NewEncoder(w).Encode(Pipe{Name: "p1", SQL: "SELECT 1"}) + _ = json.NewEncoder(w).Encode(Pipe{Name: "p1", SQL: "SELECT 1"}) } case "PUT": w.WriteHeader(200) @@ -174,8 +174,8 @@ func TestPipeRef_Fetch(t *testing.T) { c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotPath = r.URL.Path gotMethod = r.Method - json.NewDecoder(r.Body).Decode(&gotBody) - json.NewEncoder(w).Encode([]map[string]any{{"count": 42}}) + _ = json.NewDecoder(r.Body).Decode(&gotBody) + _ = json.NewEncoder(w).Encode([]map[string]any{{"count": 42}}) })) rows, err := Fetch[map[string]any](context.Background(), c.Pipe("top_pages", map[string]any{"limit": 10})) if err != nil { diff --git a/clients/go/pipes.go b/clients/go/pipes.go index f87f4d64..767078a6 100644 --- a/clients/go/pipes.go +++ b/clients/go/pipes.go @@ -13,7 +13,7 @@ type PipesNamespace struct { // List returns all registered pipes. Admin-only. func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { var pipes []Pipe - if err := doRequest(p.ctx, ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", path: "/v1/admin/pipes", }, &pipes); err != nil { @@ -25,7 +25,7 @@ func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { // Get returns a single pipe definition by name. Admin-only. func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { var pipe Pipe - if err := doRequest(p.ctx, ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", path: "/v1/admin/pipes/" + url.PathEscape(name), }, &pipe); err != nil { @@ -36,7 +36,7 @@ func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { // Set creates or updates a pipe. Admin-only. func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) error { - return doRequest(p.ctx, ctx, requestOptions{ + return doRequest(ctx, p.ctx, requestOptions{ method: "PUT", path: "/v1/admin/pipes/" + url.PathEscape(name), body: def, @@ -45,7 +45,7 @@ func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) erro // Delete removes a pipe by name. Admin-only. func (p *PipesNamespace) Delete(ctx context.Context, name string) error { - return doRequest(p.ctx, ctx, requestOptions{ + return doRequest(ctx, p.ctx, requestOptions{ method: "DELETE", path: "/v1/admin/pipes/" + url.PathEscape(name), }, nil) @@ -74,7 +74,7 @@ func Fetch[Row any](ctx context.Context, p *PipeRef) ([]Row, error) { body = map[string]any{} } var rows []Row - if err := doRequest(p.ctx, ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "POST", path: "/v1/pipes/" + url.PathEscape(p.name), body: body, diff --git a/clients/go/policy.go b/clients/go/policy.go index c7eaa2af..c8a2306b 100644 --- a/clients/go/policy.go +++ b/clients/go/policy.go @@ -10,7 +10,7 @@ type PolicyNamespace struct { // Get returns the current access-control policy. Admin-only. func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { var pol Policy - if err := doRequest(p.ctx, ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", path: "/v1/admin/policy", }, &pol); err != nil { @@ -21,7 +21,7 @@ func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { // Set replaces the entire access-control policy. Admin-only. func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { - return doRequest(p.ctx, ctx, requestOptions{ + return doRequest(ctx, p.ctx, requestOptions{ method: "PUT", path: "/v1/admin/policy", body: pol, @@ -31,7 +31,7 @@ func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { // Validate checks a policy without applying it (dry run). Admin-only. func (p *PolicyNamespace) Validate(ctx context.Context, pol *Policy) (*ValidationResult, error) { var result ValidationResult - if err := doRequest(p.ctx, ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "POST", path: "/v1/admin/policy/validate", body: pol, diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 9dda452d..6e9ab68b 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -174,7 +174,7 @@ func FetchTyped[Row any](ctx context.Context, q *QueryBuilder) (*Page[Row], erro ast := q.buildAST(limit) var rows []Row - if err := doRequest(q.ctx, ctx, requestOptions{ + if err := doRequest(ctx, q.ctx, requestOptions{ method: "POST", path: "/v1/query", params: url.Values{"table": {q.state.table}}, @@ -244,11 +244,12 @@ func (q *QueryBuilder) buildAST(effectiveLimit int) *StructuredQuery { // Projection: explicit select_all, then explicit columns, else — for a bare // query with no projection and no aggregations — default to select_all so // from(t).fetch() returns rows. - if q.state.selectAll { + switch { + case q.state.selectAll: ast.SelectAll = true - } else if hasColumns { + case hasColumns: ast.Columns = q.state.columns - } else if !hasAggs { + case !hasAggs: ast.SelectAll = true } diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index b2f560bb..84261a05 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -30,14 +30,14 @@ func captureQueryBody(t *testing.T, handler http.Handler) (*Client, func() map[s c, _ := queryTestCtx(wrapper) return c, func() map[string]any { var m map[string]any - json.Unmarshal(body, &m) + _ = json.Unmarshal(body, &m) return m } } var emptyRows = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") - json.NewEncoder(w).Encode([]map[string]any{{"page": "/home"}}) + _ = json.NewEncoder(w).Encode([]map[string]any{{"page": "/home"}}) }) func TestQueryBuilder_Immutability(t *testing.T) { @@ -199,7 +199,7 @@ func TestQueryBuilder_TimeRange(t *testing.T) { func TestQueryBuilder_Pagination_HasMore(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) + _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) @@ -217,7 +217,7 @@ func TestQueryBuilder_Pagination_HasMore(t *testing.T) { func TestQueryBuilder_Pagination_NoOrderNoNext(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) + _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) diff --git a/clients/go/schema.go b/clients/go/schema.go index 67addc5b..3c7f4307 100644 --- a/clients/go/schema.go +++ b/clients/go/schema.go @@ -11,7 +11,7 @@ type SchemaNamespace struct { func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { // The backend returns []TableSchema; transform to map[string]TableSchema. var raw []TableSchema - if err := doRequest(s.ctx, ctx, requestOptions{ + if err := doRequest(ctx, s.ctx, requestOptions{ method: "GET", path: "/v1/schema", }, &raw); err != nil { @@ -26,7 +26,7 @@ func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { // Refresh forces a schema re-discovery from ClickHouse. Admin-only. func (s *SchemaNamespace) Refresh(ctx context.Context) error { - return doRequest(s.ctx, ctx, requestOptions{ + return doRequest(ctx, s.ctx, requestOptions{ method: "POST", path: "/v1/schema/refresh", }, nil) diff --git a/clients/go/stream.go b/clients/go/stream.go index 770550d7..84075005 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -272,7 +272,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if err != nil { return "", err } - defer resp.Body.Close() + defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { return "", fmt.Errorf("SSE connect failed: HTTP %d", resp.StatusCode) diff --git a/clients/go/sys.go b/clients/go/sys.go index 4d2af556..e52c252c 100644 --- a/clients/go/sys.go +++ b/clients/go/sys.go @@ -10,7 +10,7 @@ type SysNamespace struct { // Health pings the server's public /v1/health endpoint. Returns nil when the // server is reachable and past boot, or an error describing the failure. func (s *SysNamespace) Health(ctx context.Context) error { - return doRequest(s.ctx, ctx, requestOptions{ + return doRequest(ctx, s.ctx, requestOptions{ method: "GET", path: "/v1/health", }, nil) diff --git a/clients/go/table.go b/clients/go/table.go index adc8d829..3f32d030 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -78,7 +78,7 @@ func (t *TableRef) InsertNDJSON(ctx context.Context, ndjson string) (*InsertResu // Schema returns the table's column definitions from ClickHouse. Admin-only. func (t *TableRef) Schema(ctx context.Context) (*TableSchema, error) { var schema TableSchema - if err := doRequest(t.ctx, ctx, requestOptions{ + if err := doRequest(ctx, t.ctx, requestOptions{ method: "GET", path: "/v1/schema", params: url.Values{"table": {t.table}}, @@ -98,7 +98,7 @@ func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, e OK *bool `json:"ok"` Duplicate *bool `json:"duplicate"` } - if err := doRequest(t.ctx, ctx, requestOptions{ + if err := doRequest(ctx, t.ctx, requestOptions{ method: "POST", path: "/v1/ingest", params: url.Values{"table": {t.table}}, @@ -182,7 +182,7 @@ func (t *TableRef) sendNDJSON(ctx context.Context, ndjson string) (*InsertResult Duplicates int `json:"duplicates"` Results []InsertRecordResult `json:"results"` } - if err := doRequest(t.ctx, ctx, requestOptions{ + if err := doRequest(ctx, t.ctx, requestOptions{ method: "POST", path: "/v1/ingest", params: url.Values{"table": {t.table}}, diff --git a/clients/go/table_test.go b/clients/go/table_test.go index 39bb1efa..707e13ed 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -14,8 +14,8 @@ func TestTableRef_InsertSingle(t *testing.T) { var gotPath string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotPath = r.URL.Path - json.NewDecoder(r.Body).Decode(&gotBody) - json.NewEncoder(w).Encode(map[string]any{"ok": true}) + _ = json.NewDecoder(r.Body).Decode(&gotBody) + _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/home"}) @@ -40,7 +40,7 @@ func TestTableRef_InsertBatch(t *testing.T) { gotCT = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) gotBody = string(raw) - json.NewEncoder(w).Encode(map[string]any{ + _ = json.NewEncoder(w).Encode(map[string]any{ "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) @@ -79,7 +79,7 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { gotCT = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) gotBody = string(raw) - json.NewEncoder(w).Encode(map[string]any{ + _ = json.NewEncoder(w).Encode(map[string]any{ "total": 2, "succeeded": 1, "failed": 1, "duplicates": 0, }) })) @@ -114,7 +114,7 @@ func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { var gotPath string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotPath = r.URL.Path - json.NewEncoder(w).Encode(map[string]any{"ok": true}) + _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) @@ -151,7 +151,7 @@ func TestTableRef_InsertNDJSON(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { raw, _ := io.ReadAll(r.Body) gotBody = string(raw) - json.NewEncoder(w).Encode(map[string]any{ + _ = json.NewEncoder(w).Encode(map[string]any{ "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) @@ -174,7 +174,7 @@ func TestTableRef_Schema(t *testing.T) { if r.URL.Query().Get("table") != "clicks" { t.Errorf("want table=clicks") } - json.NewEncoder(w).Encode(TableSchema{ + _ = json.NewEncoder(w).Encode(TableSchema{ Name: "clicks", Columns: []Column{ {Name: "page", Type: "String"}, @@ -196,7 +196,7 @@ func TestTableRef_Schema(t *testing.T) { func TestTableRef_InsertDuplicate(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) + _ = json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) })) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/dup"}) diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index 6bcb8e5f..591e4b75 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -118,7 +118,7 @@ func (c *Client) Pipe(name string, params map[string]any) *PipeRef { // are decoded into []T; use [map[string]any] for dynamic schemas. func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { var rows []Row - err := doRequest(c.ctx, ctx, requestOptions{ + err := doRequest(ctx, c.ctx, requestOptions{ method: "POST", path: "/v1/admin/query", body: map[string]string{"sql": query}, From 38b23970b851fd68ee931919ac1d23ecc955ae1e Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 5 Aug 2026 17:07:41 -0400 Subject: [PATCH 06/40] refactor(sdk): shrink Go SDK diff by 114 lines, zero functionality loss Source: extract helpers (errAborted, aggDefault, emptyInsertResult, marshalNDJSON, dlq.stats, sortedKeys), collapse like/not_like, inline trimTrailingSlashes, one-line IsRetryable, map-based numeric type lookup. Tests: table-driven parseErrorResponse, loop namespace nil checks, merge DLQ List+Table subcases, skipIfUnauthorized helper. Docs: dedupe Quick Start + Error Handling in index.md (link to README and reference.md), drop codegen type table from README (link to docs). --- clients/go/README.md | 14 +-- clients/go/client_test.go | 18 +--- clients/go/cmd/wavehouse-codegen/main.go | 52 ++++------ clients/go/dlq.go | 15 ++- clients/go/e2e_test.go | 22 +++-- clients/go/errors.go | 5 +- clients/go/errors_test.go | 118 ++++++++++++----------- clients/go/http.go | 15 ++- clients/go/live_query.go | 12 +-- clients/go/namespaces_test.go | 52 +++++----- clients/go/query_builder.go | 32 +++--- clients/go/stream.go | 11 +-- clients/go/table.go | 64 ++++++------ clients/go/wavehouse.go | 6 +- docs/src/content/docs/sdk/go/index.md | 78 +++------------ 15 files changed, 200 insertions(+), 314 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index e3af7bd4..6a313cc4 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -200,19 +200,7 @@ go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ --package myapp ``` -The CLI reads `/v1/schema` (admin-only) and maps ClickHouse types to Go types: - -| ClickHouse | Go | -|---|---| -| `String`, `UUID`, `DateTime*`, `Date*`, `Enum*`, `IPv4/6` | `string` | -| `UInt8/16/32/64` | `uint8/16/32/64` | -| `Int8/16/32/64` | `int8/16/32/64` | -| `Float32/64` | `float32/64` | -| `Bool` | `bool` | -| `Nullable(T)` | `*T` | -| `Array(T)` | `[]T` | -| `Map(K,V)` | `map[K]V` | -| `UInt128/256`, `Int128/256`, `Decimal*` | `string` | +See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference/#codegen-cli). ## Error Handling diff --git a/clients/go/client_test.go b/clients/go/client_test.go index 2b5239e1..020124b1 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -37,20 +37,10 @@ func TestNewClient_CustomMaxRetries(t *testing.T) { func TestNewClient_HasNamespaces(t *testing.T) { c := NewClient(Config{BaseURL: "http://localhost:8080"}) - if c.Schema == nil { - t.Fatal("Schema namespace nil") - } - if c.Policy == nil { - t.Fatal("Policy namespace nil") - } - if c.DLQ == nil { - t.Fatal("DLQ namespace nil") - } - if c.Sys == nil { - t.Fatal("Sys namespace nil") - } - if c.Pipes == nil { - t.Fatal("Pipes namespace nil") + for name, ns := range map[string]any{"Sys": c.Sys, "Schema": c.Schema, "Policy": c.Policy, "Pipes": c.Pipes, "DLQ": c.DLQ} { + if ns == nil { + t.Fatalf("%s namespace is nil", name) + } } } diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 213710e8..5dd9752d 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -175,30 +175,15 @@ func chTypeToGo(chType string) string { case chType == "Bool", chType == "Boolean": return "bool" } - // Numeric — map widths honestly. + // Numeric — map lookup. + if mapped, ok := map[string]string{ + "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", + "Int8": "int8", "Int16": "int16", "Int32": "int32", "Int64": "int64", + "Float32": "float32", "Float64": "float64", "BFloat16": "float32", + }[chType]; ok { + return mapped + } switch { - case chType == "UInt8": - return "uint8" - case chType == "UInt16": - return "uint16" - case chType == "UInt32": - return "uint32" - case chType == "UInt64": - return "uint64" - case chType == "Int8": - return "int8" - case chType == "Int16": - return "int16" - case chType == "Int32": - return "int32" - case chType == "Int64": - return "int64" - case chType == "Float32": - return "float32" - case chType == "Float64": - return "float64" - case chType == "BFloat16": - return "float32" case strings.HasPrefix(chType, "Decimal"), strings.HasPrefix(chType, "UInt128"), strings.HasPrefix(chType, "UInt256"), @@ -268,15 +253,20 @@ func pascalCase(s string) string { return result } +func sortedKeys(m map[string]tableSchema) []string { + names := make([]string, 0, len(m)) + for name := range m { + names = append(names, name) + } + slices.Sort(names) + return names +} + func generate(schemas map[string]tableSchema, pkg string) string { var sb strings.Builder fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) - names := make([]string, 0, len(schemas)) - for name := range schemas { - names = append(names, name) - } - slices.Sort(names) + names := sortedKeys(schemas) for _, name := range names { schema := schemas[name] @@ -311,11 +301,7 @@ func main() { os.Exit(1) } - names := make([]string, 0, len(schemas)) - for name := range schemas { - names = append(names, name) - } - slices.Sort(names) + names := sortedKeys(schemas) fmt.Printf("Found %d table(s): %s\n", len(schemas), strings.Join(names, ", ")) output := generate(schemas, args.pkg) diff --git a/clients/go/dlq.go b/clients/go/dlq.go index b523ab33..e833a41a 100644 --- a/clients/go/dlq.go +++ b/clients/go/dlq.go @@ -13,23 +13,20 @@ type DLQNamespace struct { // List returns DLQ statistics (message counts per table). Admin-only. func (d *DLQNamespace) List(ctx context.Context) (*DLQStats, error) { - var stats DLQStats - if err := doRequest(ctx, d.ctx, requestOptions{ - method: "GET", - path: "/v1/dlq/stats", - }, &stats); err != nil { - return nil, err - } - return &stats, nil + return d.stats(ctx, nil) } // Table returns DLQ stats filtered by table name. Admin-only. func (d *DLQNamespace) Table(ctx context.Context, name string) (*DLQStats, error) { + return d.stats(ctx, url.Values{"table": {name}}) +} + +func (d *DLQNamespace) stats(ctx context.Context, params url.Values) (*DLQStats, error) { var stats DLQStats if err := doRequest(ctx, d.ctx, requestOptions{ method: "GET", path: "/v1/dlq/stats", - params: url.Values{"table": {name}}, + params: params, }, &stats); err != nil { return nil, err } diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go index ea8aa7bc..130e6db9 100644 --- a/clients/go/e2e_test.go +++ b/clients/go/e2e_test.go @@ -257,10 +257,7 @@ func TestE2E_SQLQuery(t *testing.T) { rows, err := SQL[map[string]any](ctx, c, "SELECT 1 AS n") if err != nil { - // SQL requires admin role — skip gracefully if forbidden. - if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { - t.Skipf("e2e: SQL query requires admin auth: %v", err) - } + skipIfUnauthorized(t, err, "SQL query") t.Fatalf("SQL query failed: %v", err) } if len(rows) != 1 { @@ -289,9 +286,7 @@ func TestE2E_PolicyGetSet(t *testing.T) { pol, err := c.Policy.Get(ctx) if err != nil { - if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { - t.Skipf("e2e: Policy.Get requires admin auth: %v", err) - } + skipIfUnauthorized(t, err, "Policy.Get") t.Fatalf("Policy.Get failed: %v", err) } @@ -322,9 +317,7 @@ func TestE2E_PipesCRUD(t *testing.T) { Description: "E2E test pipe — safe to delete", } if err := c.Pipes.Set(ctx, pipeName, def); err != nil { - if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { - t.Skipf("e2e: Pipes.Set requires admin auth: %v", err) - } + skipIfUnauthorized(t, err, "Pipes.Set") t.Fatalf("Pipes.Set (create) failed: %v", err) } @@ -424,6 +417,15 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { return row } +// skipIfUnauthorized skips the test when err indicates a 401 or 403, +// meaning the operation requires admin auth the current token lacks. +func skipIfUnauthorized(t *testing.T, err error, op string) { + t.Helper() + if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { + t.Skipf("%s requires admin auth, skipping", op) + } +} + // isHTTPStatus checks whether err is a wavehouse.Error with the given status. func isHTTPStatus(err error, status int) bool { if err == nil { diff --git a/clients/go/errors.go b/clients/go/errors.go index f1fb48ac..f2ea1b24 100644 --- a/clients/go/errors.go +++ b/clients/go/errors.go @@ -33,10 +33,7 @@ func (e *Error) Error() string { // IsRetryable reports whether err wraps a retryable [*Error]. func IsRetryable(err error) bool { var e *Error - if errors.As(err, &e) { - return e.Retryable - } - return false + return errors.As(err, &e) && e.Retryable } // parseErrorResponse creates an Error from an HTTP response. diff --git a/clients/go/errors_test.go b/clients/go/errors_test.go index 0c9dd05f..c5af80df 100644 --- a/clients/go/errors_test.go +++ b/clients/go/errors_test.go @@ -8,63 +8,69 @@ import ( "testing" ) -func TestParseErrorResponse_JSONError(t *testing.T) { - res := &http.Response{ - StatusCode: 404, - Body: io.NopCloser(strings.NewReader(`{"error":"unknown table: foo"}`)), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Status != 404 { - t.Fatalf("want status 404, got %d", e.Status) - } - if e.Code != "HTTP_404" { - t.Fatalf("want code HTTP_404, got %s", e.Code) - } - if e.Message != "unknown table: foo" { - t.Fatalf("want message 'unknown table: foo', got %s", e.Message) - } - if e.Retryable { - t.Fatal("4xx should not be retryable") - } -} - -func TestParseErrorResponse_MessageField(t *testing.T) { - res := &http.Response{ - StatusCode: 400, - Body: io.NopCloser(strings.NewReader(`{"message":"bad request"}`)), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Message != "bad request" { - t.Fatalf("want 'bad request', got %s", e.Message) - } -} - -func TestParseErrorResponse_FallsBackToStatusText(t *testing.T) { - res := &http.Response{ - StatusCode: 500, - Body: io.NopCloser(strings.NewReader(`{"code":123}`)), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Message != "Internal Server Error" { - t.Fatalf("want status text fallback, got %s", e.Message) - } -} - -func TestParseErrorResponse_NonJSONBody(t *testing.T) { - res := &http.Response{ - StatusCode: 502, - Body: io.NopCloser(strings.NewReader("plain text")), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Message != "Bad Gateway" { - t.Fatalf("want 'Bad Gateway', got %s", e.Message) +func TestParseErrorResponse(t *testing.T) { + tests := []struct { + name string + status int + body string + wantMsg string + wantCode string + wantRetry bool + nilDetails bool + }{ + { + name: "JSONError", + status: 404, + body: `{"error":"unknown table: foo"}`, + wantMsg: "unknown table: foo", + wantCode: "HTTP_404", + }, + { + name: "MessageField", + status: 400, + body: `{"message":"bad request"}`, + wantMsg: "bad request", + }, + { + name: "FallsBackToStatusText", + status: 500, + body: `{"code":123}`, + wantMsg: "Internal Server Error", + wantRetry: true, + }, + { + name: "NonJSONBody", + status: 502, + body: "plain text", + wantMsg: "Bad Gateway", + wantRetry: true, + nilDetails: true, + }, } - if e.Details != nil { - t.Fatal("details should be nil for non-JSON body") + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + res := &http.Response{ + StatusCode: tt.status, + Body: io.NopCloser(strings.NewReader(tt.body)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Status != tt.status { + t.Fatalf("want status %d, got %d", tt.status, e.Status) + } + if tt.wantCode != "" && e.Code != tt.wantCode { + t.Fatalf("want code %s, got %s", tt.wantCode, e.Code) + } + if e.Message != tt.wantMsg { + t.Fatalf("want message %q, got %q", tt.wantMsg, e.Message) + } + if e.Retryable != tt.wantRetry { + t.Fatalf("want retryable=%v, got %v", tt.wantRetry, e.Retryable) + } + if tt.nilDetails && e.Details != nil { + t.Fatal("details should be nil") + } + }) } } diff --git a/clients/go/http.go b/clients/go/http.go index c7dc1c11..33c3785c 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -14,6 +14,8 @@ import ( "time" ) +var errAborted = &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + // httpContext carries per-client state needed by every request. type httpContext struct { baseURL string @@ -90,17 +92,12 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a if err != nil { // Context cancellation — return immediately, no retry. if ctx.Err() != nil { - return &Error{ - Status: 0, - Code: "ABORTED", - Message: "Request aborted", - Retryable: false, - } + return errAborted } lastErr = networkError(err) if attempt < maxAttempts-1 { if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { - return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + return errAborted } } continue @@ -145,7 +142,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a } } if sleepErr := sleepWithContext(ctx, delay); sleepErr != nil { - return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + return errAborted } lastErr = apiErr continue @@ -155,7 +152,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a // Retryable server errors (5xx). if apiErr.Retryable && attempt < maxAttempts-1 { if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { - return &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + return errAborted } lastErr = apiErr continue diff --git a/clients/go/live_query.go b/clients/go/live_query.go index d6c665fe..1af2b770 100644 --- a/clients/go/live_query.go +++ b/clients/go/live_query.go @@ -48,16 +48,8 @@ func newLiveQuery( sub.Next(event) } }, - Status: func(s StreamStatus) { - if sub.Status != nil { - sub.Status(s) - } - }, - Error: func(err error) { - if sub.Error != nil { - sub.Error(err) - } - }, + Status: sub.Status, + Error: sub.Error, }) // Step 2–5: Fetch historical and flush. diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index 18ff1b39..2801c1ab 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -97,32 +97,34 @@ func TestPolicyNamespace_GetSetValidate(t *testing.T) { } } -func TestDLQNamespace_List(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { - _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) - })) - stats, err := c.DLQ.List(context.Background()) - if err != nil { - t.Fatal(err) - } - if stats.Total != 3 { - t.Fatalf("want total=3, got %d", stats.Total) - } -} +func TestDLQNamespace(t *testing.T) { + t.Run("List", func(t *testing.T) { + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) + })) + stats, err := c.DLQ.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if stats.Total != 3 { + t.Fatalf("want total=3, got %d", stats.Total) + } + }) -func TestDLQNamespace_Table(t *testing.T) { - var gotParam string - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - gotParam = r.URL.Query().Get("table") - _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) - })) - _, err := c.DLQ.Table(context.Background(), "clicks") - if err != nil { - t.Fatal(err) - } - if gotParam != "clicks" { - t.Fatalf("want table=clicks, got %s", gotParam) - } + t.Run("Table", func(t *testing.T) { + var gotParam string + c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotParam = r.URL.Query().Get("table") + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) + })) + _, err := c.DLQ.Table(context.Background(), "clicks") + if err != nil { + t.Fatal(err) + } + if gotParam != "clicks" { + t.Fatalf("want table=clicks, got %s", gotParam) + } + }) } func TestPipesNamespace_CRUD(t *testing.T) { diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 6e9ab68b..65679ad1 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -82,42 +82,27 @@ func (q *QueryBuilder) Count(column, alias string) *QueryBuilder { // Sum adds a SUM aggregation. func (q *QueryBuilder) Sum(column, alias string) *QueryBuilder { - if alias == "" { - alias = "sum_" + column - } - return q.addAgg("sum", column, alias) + return q.aggDefault("sum", "sum_", column, alias) } // Avg adds an AVG aggregation. func (q *QueryBuilder) Avg(column, alias string) *QueryBuilder { - if alias == "" { - alias = "avg_" + column - } - return q.addAgg("avg", column, alias) + return q.aggDefault("avg", "avg_", column, alias) } // Min adds a MIN aggregation. func (q *QueryBuilder) Min(column, alias string) *QueryBuilder { - if alias == "" { - alias = "min_" + column - } - return q.addAgg("min", column, alias) + return q.aggDefault("min", "min_", column, alias) } // Max adds a MAX aggregation. func (q *QueryBuilder) Max(column, alias string) *QueryBuilder { - if alias == "" { - alias = "max_" + column - } - return q.addAgg("max", column, alias) + return q.aggDefault("max", "max_", column, alias) } // CountDistinct adds a COUNT DISTINCT aggregation. func (q *QueryBuilder) CountDistinct(column, alias string) *QueryBuilder { - if alias == "" { - alias = "count_distinct_" + column - } - return q.addAgg("countDistinct", column, alias) + return q.aggDefault("countDistinct", "count_distinct_", column, alias) } // Aggregate adds a custom aggregation function. @@ -230,6 +215,13 @@ func (q *QueryBuilder) LiveQuery(sub *StreamSubscriber, opts *StreamOptions) *Li return newLiveQuery(stream, fetchFn, sub, q.state.filters) } +func (q *QueryBuilder) aggDefault(fn, prefix, column, alias string) *QueryBuilder { + if alias == "" { + alias = prefix + column + } + return q.addAgg(fn, column, alias) +} + func (q *QueryBuilder) addAgg(fn, column, alias string) *QueryBuilder { return q.clone(func(s *queryState) { s.aggregations = append(s.aggregations, Aggregation{Fn: fn, Column: column, Alias: alias}) diff --git a/clients/go/stream.go b/clients/go/stream.go index 84075005..2fabf1e2 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -426,20 +426,13 @@ func evaluateFilter(actual any, op string, expected any) bool { return ok && c <= 0 case "in": return evaluateIn(actual, expected) - case "like": + case "like", "not_like": aStr, aOK := actual.(string) eStr, eOK := expected.(string) if !aOK || !eOK { return false } - return matchLike(aStr, eStr) - case "not_like": - aStr, aOK := actual.(string) - eStr, eOK := expected.(string) - if !aOK || !eOK { - return false - } - return !matchLike(aStr, eStr) + return (op == "like") == matchLike(aStr, eStr) default: return false } diff --git a/clients/go/table.go b/clients/go/table.go index 3f32d030..d466d5b4 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -3,6 +3,7 @@ package wavehouse import ( "context" "encoding/json" + "fmt" "net/url" "reflect" "strings" @@ -117,29 +118,35 @@ func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, e return result, nil } -func (t *TableRef) insertBatch(ctx context.Context, rows []map[string]any) (*InsertResult, error) { - if len(rows) == 0 { - zero := 0 - return &InsertResult{ - OK: true, - Total: &zero, - Succeeded: &zero, - Failed: &zero, - Duplicates: &zero, - }, nil - } +func emptyInsertResult() *InsertResult { + z := 0 + return &InsertResult{OK: true, Total: &z, Succeeded: &z, Failed: &z, Duplicates: &z} +} + +func marshalNDJSON(n int, elem func(int) any) (string, error) { var sb strings.Builder - for i, row := range rows { + for i := range n { if i > 0 { sb.WriteByte('\n') } - raw, err := json.Marshal(row) + raw, err := json.Marshal(elem(i)) if err != nil { - return nil, err + return "", fmt.Errorf("wavehouse: marshal row %d: %w", i, err) } sb.Write(raw) } - return t.sendNDJSON(ctx, sb.String()) + return sb.String(), nil +} + +func (t *TableRef) insertBatch(ctx context.Context, rows []map[string]any) (*InsertResult, error) { + if len(rows) == 0 { + return emptyInsertResult(), nil + } + ndjson, err := marshalNDJSON(len(rows), func(i int) any { return rows[i] }) + if err != nil { + return nil, err + } + return t.sendNDJSON(ctx, ndjson) } // insertBatchReflect is the fallback batch path for any slice type other @@ -149,29 +156,14 @@ func (t *TableRef) insertBatch(ctx context.Context, rows []map[string]any) (*Ins // insertBatch, so the server's per-record batch summary (failed, results, // etc.) is preserved instead of being silently dropped by insertSingle. func (t *TableRef) insertBatchReflect(ctx context.Context, rows reflect.Value) (*InsertResult, error) { - n := rows.Len() - if n == 0 { - zero := 0 - return &InsertResult{ - OK: true, - Total: &zero, - Succeeded: &zero, - Failed: &zero, - Duplicates: &zero, - }, nil + if rows.Len() == 0 { + return emptyInsertResult(), nil } - var sb strings.Builder - for i := 0; i < n; i++ { - if i > 0 { - sb.WriteByte('\n') - } - raw, err := json.Marshal(rows.Index(i).Interface()) - if err != nil { - return nil, err - } - sb.Write(raw) + ndjson, err := marshalNDJSON(rows.Len(), func(i int) any { return rows.Index(i).Interface() }) + if err != nil { + return nil, err } - return t.sendNDJSON(ctx, sb.String()) + return t.sendNDJSON(ctx, ndjson) } func (t *TableRef) sendNDJSON(ctx context.Context, ndjson string) (*InsertResult, error) { diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index 591e4b75..3e645eaf 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -14,6 +14,7 @@ package wavehouse import ( "context" "net/http" + "strings" ) // Config configures a [Client]. @@ -135,8 +136,5 @@ func (c *Client) createStream(table string, opts *StreamOptions) *StreamControll } func trimTrailingSlashes(s string) string { - for len(s) > 0 && s[len(s)-1] == '/' { - s = s[:len(s)-1] - } - return s + return strings.TrimRight(s, "/") } diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index db515ef2..d95a83d5 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -35,62 +35,31 @@ every example on these pages assumes it. ## Quick Start ```go -package main - import ( "context" "fmt" - "log" wavehouse "github.com/Wave-RF/WaveHouse/clients/go" ) -func main() { - ctx := context.Background() - - // Create a client. Auth is optional — omit it for public/unauthenticated - // access (the server falls back to policy.default_role). - wh := wavehouse.NewClient(wavehouse.Config{ - BaseURL: "http://localhost:8080", - Auth: wavehouse.StaticToken("your-jwt"), - }) - - // Health check. - if err := wh.Sys.Health(ctx); err != nil { - log.Fatal(err) - } - - // Insert a row. - if _, err := wh.From("clicks").Insert(ctx, map[string]any{ - "page": "/home", "button": "signup", - }); err != nil { - log.Fatal(err) - } - - // Query with the fluent builder. - page, err := wh.From("clicks"). - Select("page", "button"). - Where("page", wavehouse.OpEq, "/home"). - Limit(10). - FetchUntyped(ctx) - if err != nil { - log.Fatal(err) - } - for _, row := range page.Data { - fmt.Println(row["page"], row["button"]) - } +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), +}) - // Stream. - stream := wh.From("clicks").Stream(nil) - defer stream.Close() - unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ - Next: func(e wavehouse.StreamEvent) { fmt.Println(e.Data) }, - Status: func(s wavehouse.StreamStatus) { fmt.Println("Stream:", s) }, - }) - defer unsub() +page, err := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(context.Background()) +if err != nil { /* handle */ } +for _, row := range page.Data { + fmt.Println(row["page"], row["button"]) } ``` +See the [README](https://github.com/Wave-RF/WaveHouse/blob/main/clients/go/README.md) for more quick-start examples. + ## Creating a Client ```go @@ -179,10 +148,7 @@ parameter. ## Error Handling -Every SDK operation returns `(T, error)` — the idiomatic Go shape, and the -direct equivalent of the TypeScript SDK's -[`Result`](/sdk#result-type) discriminated union. Errors are always -`*wavehouse.Error`; unwrap with `errors.As`: +All SDK operations return `(T, error)`. Errors are `*wavehouse.Error`; unwrap with `errors.As`: ```go page, err := wh.From("clicks").Fetch(ctx) @@ -195,19 +161,7 @@ if err != nil { } ``` -```go -type Error struct { - Status int // HTTP status (0 for network/abort errors) - Code string // e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED" - Message string // Human-readable error message - Details map[string]any // Parsed response body, if available - Retryable bool // Whether the SDK would retry this error -} -``` - -`wavehouse.IsRetryable(err)` is a shortcut for `errors.As` + `.Retryable`. -The full error-code table lives in -[Reference → Error Handling](/sdk/go/reference#error-handling). +See [Reference → Error Handling](/sdk/go/reference/#error-handling) for retry behavior and error codes. ## Differences from the TypeScript SDK From 5a480bf9edb0c7f57d9eadd057c6f5f13fc675d8 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 12:59:51 -0400 Subject: [PATCH 07/40] =?UTF-8?q?fix(sdk):=20address=20PR=20#434=20review?= =?UTF-8?q?=20feedback=20=E2=80=94=20CI=20blockers=20+=2051=20review=20fin?= =?UTF-8?q?dings?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI blockers: - Bump google.golang.org/grpc 1.82.1, golang.org/x/text 0.39.0, klauspost/compress 1.18.7 (GO-2026-6061/-5970/-5841) — vulncheck green - Drop forbidden trailing slash in sdk/go/index.md anchor link SDK behavior (CodeRabbit/Copilot review): - Retry: 429 now retryable, Retry-After honored for 429 and clamped to 30s, ±20% backoff jitter, dead 503 clause removed - NewClient uses a fresh http.Client instead of mutable http.DefaultClient - LiveQueryHandle: close state applied synchronously in Close(); dedup bound is the max backfilled timestamp compared as parsed time.Time - Stream reconnect backoff resets after a connection reaches live - Pagination replaces the cursor filter instead of stacking one per page - PolicyFilter operators marshal with omitempty (absent, never null) - Errors wrapped with operation context at every SDK boundary - codegen: unknown args rejected, 30s HTTP timeout, wrapped errors, field/type collision detection, WAVEHOUSE_AUTH env var for the token, Int64/UInt64 map to string (ClickHouse quotes 64-bit ints in JSON output) Tests: - httptest servers closed via t.Cleanup; handler captures synchronized; captureQueryBody uses io.ReadAll; multi-scenario tables use t.Run subtests - e2e: probe timeout, deterministic table pick returning its schema, polling instead of fixed sleeps, errors.As, unsupported-type skip - conformance: SDK errors logged, arg guards, normalizePath compares decoded query values (Go + TS), TS harness counts unhandled endpoints as skipped, stubs aligned to real server shapes, deterministic exit, cacheTTL fixture Docs/Makefile: policyDraft defined in admin example, (T, error) claim scoped to request-response ops, operator-key path documented for admin SQL, test-go-sdk-e2e documented, test-all includes test-go-sdk, --allow-parallel-runners on SDK lint, HTTPS caution for bearer tokens. --- CHANGELOG.md | 2 + Makefile | 5 +- clients/go/README.md | 2 +- clients/go/client_test.go | 15 +++ clients/go/cmd/wavehouse-codegen/main.go | 58 +++++++-- clients/go/conformance_test.go | 90 ++++++++++---- clients/go/e2e_test.go | 134 ++++++++++----------- clients/go/errors.go | 2 +- clients/go/errors_test.go | 30 ++--- clients/go/example_test.go | 10 +- clients/go/http.go | 14 ++- clients/go/http_test.go | 26 +++-- clients/go/live_query.go | 136 +++++++++++++--------- clients/go/namespaces_test.go | 36 ++++-- clients/go/pipes.go | 21 ++-- clients/go/policy.go | 16 ++- clients/go/query_builder.go | 10 +- clients/go/query_builder_test.go | 52 +++++---- clients/go/stream.go | 27 +++-- clients/go/sys.go | 12 +- clients/go/table.go | 6 +- clients/go/testdata/wire_cases.json | 15 +++ clients/go/types.go | 17 +-- clients/go/wavehouse.go | 9 +- docs/src/content/docs/sdk/go/admin.md | 5 +- docs/src/content/docs/sdk/go/index.md | 13 ++- docs/src/content/docs/sdk/go/queries.md | 10 +- docs/src/content/docs/sdk/go/reference.md | 46 +++++--- docs/src/content/docs/sdk/index.mdx | 4 +- go.mod | 20 ++-- go.sum | 40 +++---- tests/conformance/conformance_ts.mjs | 43 ++++++- 32 files changed, 599 insertions(+), 327 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e0ba485d..627487f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Go toolchain requirement bumped to 1.26.5** (`go.mod`): the `go` directive moves from `1.26.4` to `1.26.5` so local builds (via `GOTOOLCHAIN=auto`) and CI's `setup-go` (which reads `go.mod` via `go-version-file`) install a Go whose standard library clears the `govulncheck` findings GO-2026-5856 (`crypto/tls`) and GO-2026-4970 (`os`), both fixed in 1.26.5 — those findings were failing `make verify`'s `vulncheck` leaf (and with it the pre-commit hook) on every tree. Patch-level toolchain bump only — no source changes — and the released binaries pick up the patched stdlib too. +- **Dependency bumps clearing new govulncheck findings** (`go.mod`, `go.sum`): `google.golang.org/grpc` 1.81.1 → 1.82.1 (GO-2026-6061), `golang.org/x/text` 0.37.0 → 0.39.0 (GO-2026-5970), `github.com/klauspost/compress` 1.18.6 → 1.18.7 (GO-2026-5841) — all three were failing `make verify`'s `vulncheck` leaf after the advisories published. No API changes. + ### Security - **Policy `_in` is now enforced on both the row-`filter` and insert-`check` paths, closing a fail-open row-security gap** (`internal/policy/policy.go`, `internal/api/ingest.go`, `docs/src/content/docs/access-control.mdx`, plus tests in `internal/policy/policy_test.go`, `internal/api/ingest_test.go`): closes #224. The `Filter` schema accepted `_in` but the engine never read it: on the row-`filter`/SELECT path `resolveFilters` had no `_in` branch, so a row-security filter like `tenant_id: { _in: … }` produced **no `WHERE` predicate** and the role saw every row instead of its tenant subset (a fail-open, same family as #223); on the `check`/INSERT path only `_eq` was honored, silently dropping any other operator. `_in` now takes a single claim that resolves to a JSON **array** (the multi-tenant case — a token's `tenant_ids` list) and emits `col IN (?, …)` with one bound param per element; a scalar claim is a one-element set, and an empty/absent claim matches **no rows** (fail-closed) rather than widening to all of them. On the insert path an `_in` check requires the column be present and one of the set — there is no single value to auto-inject as `_eq` does, so an omitted column is rejected (`403 check failed`). The comparison operators are enforced on `filter` (`_eq`/`_neq`/`_gt`/`_lt`/`_in` all produce predicates now, so nothing is rejected there) and, on `check`, `_neq`/`_gt`/`_lt` become a loud config-load rejection (no insert-time semantics; `check` honors `_eq` + `_in`). The `_in` value stays a single templated string in the wire schema (Go `Filter.In`, SDK `PolicyFilter._in`), matching the established "set = array" shape of the caller-query `in` operator. diff --git a/Makefile b/Makefile index 0c614912..fdb9a7ab 100644 --- a/Makefile +++ b/Makefile @@ -364,7 +364,7 @@ lint-go: $(GOLANGCI_LINT) go-mod-download .PHONY: lint-go-sdk lint-go-sdk: $(GOLANGCI_LINT) - $(call run,golangci-lint (Go SDK),cd clients/go && $(GOLANGCI_LINT) run ./...,) + $(call run,golangci-lint (Go SDK),cd clients/go && $(GOLANGCI_LINT) run ./... --allow-parallel-runners,) .PHONY: lint-ts lint-ts: pnpm-install @@ -477,7 +477,7 @@ fix-prose: $(MISSPELL) # slowest tool, not the slowest *group* (e.g. golangci no longer drags Biome + # markdownlint along behind it). # -# Leaves (10): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck, +# Leaves (see verify-parallel prerequisites): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck, # verify-go-sdk (go vet + gofumpt on the nested clients/go module) on the Go # side; lint-ts (biome check) + lint-md (markdownlint) + lint-prose (misspell, # docs spelling) for JS/TS + Markdown + prose; @@ -744,6 +744,7 @@ test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVE .PHONY: test-all test-all: ## Run all suites sequentially + one consolidated Go + TS coverage report + gates @$(MAKE) test-unit COV_DEFER=1 + @$(MAKE) test-go-sdk @$(MAKE) test-ts COV_DEFER=1 @$(MAKE) test-integration COV_DEFER=1 @$(MAKE) test-e2e COV_DEFER=1 diff --git a/clients/go/README.md b/clients/go/README.md index 6a313cc4..c71e5644 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -204,7 +204,7 @@ See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference/# ## Error Handling -All SDK operations return `(T, error)`. Errors are `*wavehouse.Error` (use `errors.As`): +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors are `*wavehouse.Error` (use `errors.As`). Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`) deliver errors through callbacks instead: ```go page, err := client.From("clicks").Fetch(ctx) diff --git a/clients/go/client_test.go b/clients/go/client_test.go index 020124b1..22a45067 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -52,6 +52,7 @@ func TestClient_From(t *testing.T) { } _ = json.NewEncoder(w).Encode([]map[string]any{}) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) _, _ = c.From("events").Fetch(context.Background()) } @@ -68,6 +69,7 @@ func TestClient_SQL(t *testing.T) { } _ = json.NewEncoder(w).Encode([]map[string]any{{"x": 1}}) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) rows, err := SQL[map[string]any](context.Background(), c, "SELECT 1") if err != nil { @@ -88,3 +90,16 @@ func TestStaticToken(t *testing.T) { t.Fatalf("want abc, got %s", token) } } + +func TestPolicyFilter_MarshalOperators(t *testing.T) { + empty := "" + raw, err := json.Marshal(PolicyFilter{Eq: &empty}) + if err != nil { + t.Fatal(err) + } + // Intentional empty-string comparison survives; unset operators are + // omitted entirely, never sent as null. + if string(raw) != `{"_eq":""}` { + t.Fatalf(`want {"_eq":""}, got %s`, raw) + } +} diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 5dd9752d..9775b525 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -3,7 +3,7 @@ // // Usage: // -// wavehouse-codegen --url http://localhost:8080 --out ./db.go --auth +// WAVEHOUSE_AUTH= wavehouse-codegen --url http://localhost:8080 --out ./db.go package main import ( @@ -15,6 +15,7 @@ import ( "os" "slices" "strings" + "time" "unicode" ) @@ -56,11 +57,19 @@ Options: --url, -u WaveHouse base URL (default: http://localhost:8080) --out, -o Output .go file path (default: ./wavehouse_types.go) --auth, -a Bearer token for authenticated /v1/schema endpoint + (prefer the WAVEHOUSE_AUTH env var — argv leaks into + shell history and process listings) --package, -p Go package name (default: main) --help, -h Show this help`) os.Exit(0) + default: + fmt.Fprintf(os.Stderr, "Error: unknown argument %q (use --help)\n", os.Args[i]) + os.Exit(2) } } + if args.auth == "" { + args.auth = os.Getenv("WAVEHOUSE_AUTH") + } return args } @@ -80,14 +89,15 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc url := strings.TrimRight(baseURL, "/") + "/v1/schema" req, err := http.NewRequestWithContext(ctx, "GET", url, nil) if err != nil { - return nil, err + return nil, fmt.Errorf("build schema request for %s: %w", url, err) } if auth != "" { req.Header.Set("Authorization", "Bearer "+auth) } - resp, err := http.DefaultClient.Do(req) + client := &http.Client{Timeout: 30 * time.Second} + resp, err := client.Do(req) if err != nil { - return nil, err + return nil, fmt.Errorf("fetch schema from %s: %w", url, err) } defer func() { _ = resp.Body.Close() }() if resp.StatusCode != 200 { @@ -97,7 +107,7 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc // Server returns either []tableSchema or map[string]tableSchema. var raw json.RawMessage if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil { - return nil, err + return nil, fmt.Errorf("read schema response: %w", err) } // Try array first. var arr []tableSchema @@ -110,7 +120,7 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc } var m map[string]tableSchema if err := json.Unmarshal(raw, &m); err != nil { - return nil, err + return nil, fmt.Errorf("decode schema JSON: %w", err) } return m, nil } @@ -177,19 +187,25 @@ func chTypeToGo(chType string) string { } // Numeric — map lookup. if mapped, ok := map[string]string{ - "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", - "Int8": "int8", "Int16": "int16", "Int32": "int32", "Int64": "int64", + "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", + "Int8": "int8", "Int16": "int16", "Int32": "int32", "Float32": "float32", "Float64": "float64", "BFloat16": "float32", }[chType]; ok { return mapped } switch { case strings.HasPrefix(chType, "Decimal"), + strings.HasPrefix(chType, "UInt64"), strings.HasPrefix(chType, "UInt128"), strings.HasPrefix(chType, "UInt256"), + strings.HasPrefix(chType, "Int64"), strings.HasPrefix(chType, "Int128"), strings.HasPrefix(chType, "Int256"): - return "string" // big numbers are strings in JSON + // 64-bit and bigger integers (and decimals) are strings on the wire: + // ClickHouse's JSON output quotes them by default + // (output_format_json_quote_64bit_integers=1) and the server forwards + // the ClickHouse `data` array verbatim. + return "string" } // Array. if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { @@ -262,19 +278,33 @@ func sortedKeys(m map[string]tableSchema) []string { return names } -func generate(schemas map[string]tableSchema, pkg string) string { +func generate(schemas map[string]tableSchema, pkg string) (string, error) { var sb strings.Builder fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) names := sortedKeys(schemas) + // pascalCase is not injective ("user_id" and "userId" both yield + // "UserId"), and format.Source only parses — it doesn't type-check — so + // a duplicate identifier would be written as a non-compiling file with a + // success message. Fail loudly instead. + seenTypes := make(map[string]string, len(names)) for _, name := range names { schema := schemas[name] typeName := pascalCase(name) + "Row" + if prev, dup := seenTypes[typeName]; dup { + return "", fmt.Errorf("tables %q and %q both map to type %q; rename one or generate separately", prev, name, typeName) + } + seenTypes[typeName] = name fmt.Fprintf(&sb, "// %s represents a row in the %q table.\ntype %s struct {\n", typeName, name, typeName) + seenFields := make(map[string]string, len(schema.Columns)) for _, col := range schema.Columns { goType := chTypeToGo(col.Type) fieldName := pascalCase(col.Name) + if prev, dup := seenFields[fieldName]; dup { + return "", fmt.Errorf("table %q: columns %q and %q both map to field %q", name, prev, col.Name, fieldName) + } + seenFields[fieldName] = col.Name jsonTag := col.Name if col.HasDefault { jsonTag += ",omitempty" @@ -284,7 +314,7 @@ func generate(schemas map[string]tableSchema, pkg string) string { sb.WriteString("}\n\n") } - return sb.String() + return sb.String(), nil } func main() { @@ -304,7 +334,11 @@ func main() { names := sortedKeys(schemas) fmt.Printf("Found %d table(s): %s\n", len(schemas), strings.Join(names, ", ")) - output := generate(schemas, args.pkg) + output, err := generate(schemas, args.pkg) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } // gofmt the output. A failure here means the generated source is not // valid Go (e.g. a table/column name produced an invalid identifier); diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index 8809457a..e41c7b09 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -7,11 +7,22 @@ import ( "io" "net/http" "net/http/httptest" + "net/url" "reflect" "strings" "testing" ) +// logCallErr surfaces SDK-call errors that the conformance harness otherwise +// ignores — the assertions only inspect the captured request, but when a call +// fails before sending, the failure message should name the real cause. +func logCallErr(t *testing.T, err error) { + t.Helper() + if err != nil { + t.Logf("SDK call returned error (request may still be valid): %v", err) + } +} + // wireCasesJSON embeds the shared wire-format conformance fixture so the // test binary is self-contained: it works from a module archive or a // standalone checkout without depending on paths outside the Go module. @@ -103,18 +114,25 @@ func TestConformance_WireFormat(t *testing.T) { // Execute the case. switch tc.Endpoint { case "query": - q := c.From(tc.Table).Select() - q = applyOps(t, q, tc.Table, c, tc.Operations) - _, _ = q.FetchUntyped(ctx) + q := applyOps(t, tc.Table, c, tc.Operations) + _, err := q.FetchUntyped(ctx) + logCallErr(t, err) case "ingest": if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { + if len(tc.Operations[0].Args) == 0 { + t.Fatal("insert needs 1 arg, got 0") + } data := tc.Operations[0].Args[0] - _, _ = c.From(tc.Table).Insert(ctx, data) + _, err := c.From(tc.Table).Insert(ctx, data) + logCallErr(t, err) } case "ingest_batch": if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { + if len(tc.Operations[0].Args) == 0 { + t.Fatal("insert needs 1 arg, got 0") + } rawArr, ok := tc.Operations[0].Args[0].([]any) if !ok { t.Fatalf("batch insert args[0] is not an array") @@ -123,63 +141,73 @@ func TestConformance_WireFormat(t *testing.T) { for i, r := range rawArr { rows[i] = toStringMap(r) } - _, _ = c.From(tc.Table).Insert(ctx, rows) + _, err := c.From(tc.Table).Insert(ctx, rows) + logCallErr(t, err) } case "pipe": p := c.Pipe(tc.PipeName, tc.PipeParams) - _, _ = p.FetchUntyped(ctx) + _, err := p.FetchUntyped(ctx) + logCallErr(t, err) case "sql": - _, _ = SQL[map[string]any](ctx, c, tc.SQL) + _, err := SQL[map[string]any](ctx, c, tc.SQL) + logCallErr(t, err) case "health": - _ = c.Sys.Health(ctx) + logCallErr(t, c.Sys.Health(ctx)) case "schema_list": - _, _ = c.Schema.List(ctx) + _, err := c.Schema.List(ctx) + logCallErr(t, err) case "schema_refresh": - _ = c.Schema.Refresh(ctx) + logCallErr(t, c.Schema.Refresh(ctx)) case "policy_get": - _, _ = c.Policy.Get(ctx) + _, err := c.Policy.Get(ctx) + logCallErr(t, err) case "policy_set": var pol Policy if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { t.Fatalf("parse policy_body: %v", err) } - _ = c.Policy.Set(ctx, &pol) + logCallErr(t, c.Policy.Set(ctx, &pol)) case "policy_validate": var pol Policy if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { t.Fatalf("parse policy_body: %v", err) } - _, _ = c.Policy.Validate(ctx, &pol) + _, err := c.Policy.Validate(ctx, &pol) + logCallErr(t, err) case "dlq_list": - _, _ = c.DLQ.List(ctx) + _, err := c.DLQ.List(ctx) + logCallErr(t, err) case "dlq_table": - _, _ = c.DLQ.Table(ctx, tc.Table) + _, err := c.DLQ.Table(ctx, tc.Table) + logCallErr(t, err) case "pipes_list": - _, _ = c.Pipes.List(ctx) + _, err := c.Pipes.List(ctx) + logCallErr(t, err) case "pipes_get": - _, _ = c.Pipes.Get(ctx, tc.PipeName) + _, err := c.Pipes.Get(ctx, tc.PipeName) + logCallErr(t, err) case "pipes_set": var def PipeDef if err := json.Unmarshal(tc.PipeDefBody, &def); err != nil { t.Fatalf("parse pipe_def: %v", err) } - _ = c.Pipes.Set(ctx, tc.PipeName, def) + logCallErr(t, c.Pipes.Set(ctx, tc.PipeName, def)) case "pipes_delete": - _ = c.Pipes.Delete(ctx, tc.PipeName) + logCallErr(t, c.Pipes.Delete(ctx, tc.PipeName)) default: t.Skipf("unhandled endpoint: %s", tc.Endpoint) @@ -235,7 +263,7 @@ func TestConformance_WireFormat(t *testing.T) { // applyOps replays the operation chain from the fixture onto a QueryBuilder. // Fixtures always put select first (mirroring real usage), so rebuilding on // select is safe and keeps this simple. -func applyOps(t *testing.T, _ *QueryBuilder, table string, c *Client, ops []wireOp) *QueryBuilder { +func applyOps(t *testing.T, table string, c *Client, ops []wireOp) *QueryBuilder { t.Helper() q := c.From(table).Select() @@ -249,8 +277,15 @@ func applyOps(t *testing.T, _ *QueryBuilder, table string, c *Client, ops []wire if len(op.Args) != 3 { t.Fatalf("where needs 3 args, got %d", len(op.Args)) } - col := op.Args[0].(string) - opStr := FilterOp(op.Args[1].(string)) + col, ok := op.Args[0].(string) + if !ok { + t.Fatalf("where: column arg is %T, want string", op.Args[0]) + } + rawOp, ok := op.Args[1].(string) + if !ok { + t.Fatalf("where: operator arg is %T, want string", op.Args[1]) + } + opStr := FilterOp(rawOp) val := op.Args[2] q = q.Where(col, opStr, val) case "count": @@ -371,7 +406,14 @@ func normalizeJSON(v any) any { } } +// normalizePath compares request URIs by meaning: same path, same decoded +// query values regardless of + vs %20 spelling or parameter order. A raw +// string replace would also rewrite literal + characters and stop asserting +// the encoding at all. func normalizePath(p string) string { - // Normalize URL encoding differences (+ vs %20 for spaces). - return strings.ReplaceAll(p, "+", "%20") + u, err := url.ParseRequestURI(p) + if err != nil { + return p + } + return u.Path + "?" + u.Query().Encode() } diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go index 130e6db9..69b724e3 100644 --- a/clients/go/e2e_test.go +++ b/clients/go/e2e_test.go @@ -4,9 +4,11 @@ package wavehouse import ( "context" + "errors" "fmt" "net/http" "os" + "slices" "strings" "testing" "time" @@ -33,14 +35,15 @@ func e2eClient(t *testing.T) *Client { cfg.Auth = StaticToken(tok) } - // Probe the server before committing to the test. - probe, err := http.NewRequestWithContext( - context.Background(), "GET", base+"/v1/health", nil, - ) + // Probe the server before committing to the test. Bounded so a host that + // accepts the connection but never responds still yields the graceful skip. + probeCtx, cancel := context.WithTimeout(context.Background(), 3*time.Second) + defer cancel() + probe, err := http.NewRequestWithContext(probeCtx, "GET", base+"/v1/health", nil) if err != nil { t.Skipf("e2e: bad WAVEHOUSE_URL %q: %v", base, err) } - resp, err := http.DefaultClient.Do(probe) + resp, err := (&http.Client{Timeout: 3 * time.Second}).Do(probe) if err != nil { t.Skipf("e2e: server unreachable at %s: %v", base, err) } @@ -58,20 +61,47 @@ func marker(t *testing.T) string { return fmt.Sprintf("%s_%d", safe, time.Now().UnixNano()) } -// firstTable discovers a usable table from the schema list. Many E2E tests -// need a real table to insert/query — this avoids hardcoding a name. -func firstTable(t *testing.T, c *Client) string { +// firstTable discovers a usable table from the schema list and returns its +// schema alongside the name. Many E2E tests need a real table to insert/query +// — this avoids hardcoding a name, and returning the schema avoids a second +// Schema.List whose result set might no longer contain the chosen table. +// Sorted so every run picks the same table (map iteration order is random). +func firstTable(t *testing.T, c *Client) (string, TableSchema) { t.Helper() - ctx := context.Background() - schemas, err := c.Schema.List(ctx) + schemas, err := c.Schema.List(context.Background()) if err != nil { t.Skipf("e2e: cannot list schemas (auth?): %v", err) } + names := make([]string, 0, len(schemas)) for name := range schemas { - return name + names = append(names, name) + } + if len(names) == 0 { + t.Skip("e2e: no tables found — server has an empty schema") + } + slices.Sort(names) + return names[0], schemas[names[0]] +} + +// waitForRows polls the marker query until at least want rows are visible or +// the deadline expires. Ingestion is asynchronous — a fixed sleep fails on a +// loaded runner without any real defect. +func waitForRows(t *testing.T, c *Client, table, markerCol, mk string, want int) []map[string]any { + t.Helper() + deadline := time.Now().Add(10 * time.Second) + for { + page, err := c.From(table).Select(markerCol). + Where(markerCol, OpEq, mk). + Limit(max(want, 1)). + FetchUntyped(context.Background()) + if err != nil { + t.Fatalf("query for marker %q: %v", mk, err) + } + if len(page.Data) >= want || time.Now().After(deadline) { + return page.Data + } + time.Sleep(200 * time.Millisecond) } - t.Skip("e2e: no tables found — server has an empty schema") - return "" } // --------------------------------------------------------------------------- @@ -109,21 +139,9 @@ func TestE2E_SchemaList(t *testing.T) { func TestE2E_InsertAndQuery(t *testing.T) { c := e2eClient(t) ctx := context.Background() - table := firstTable(t, c) + table, ts := firstTable(t, c) mk := marker(t) - // Discover columns so we can build a valid row. We need at least one - // string-ish column to inject our marker. Fall back to skipping if the - // table's schema doesn't have one we can use. - schemas, err := c.Schema.List(ctx) - if err != nil { - t.Fatalf("Schema.List: %v", err) - } - ts, ok := schemas[table] - if !ok { - t.Skipf("table %q vanished between discovery and use", table) - } - row := buildMarkerRow(t, ts, mk) markerCol := markerColumn(t, ts) @@ -135,21 +153,11 @@ func TestE2E_InsertAndQuery(t *testing.T) { t.Fatalf("Insert into %s: OK=false", table) } - // Allow a moment for async ingestion to settle. - time.Sleep(500 * time.Millisecond) - - // Query it back. - page, err := c.From(table).Select(markerCol). - Where(markerCol, OpEq, mk). - Limit(1). - FetchUntyped(ctx) - if err != nil { - t.Fatalf("Query failed: %v", err) - } - if len(page.Data) == 0 { + rows := waitForRows(t, c, table, markerCol, mk, 1) + if len(rows) == 0 { t.Fatal("Query returned zero rows — expected the inserted marker row") } - got, _ := page.Data[0][markerCol].(string) + got, _ := rows[0][markerCol].(string) if got != mk { t.Errorf("marker mismatch: want %q, got %q", mk, got) } @@ -158,13 +166,7 @@ func TestE2E_InsertAndQuery(t *testing.T) { func TestE2E_BatchInsert(t *testing.T) { c := e2eClient(t) ctx := context.Background() - table := firstTable(t, c) - - schemas, err := c.Schema.List(ctx) - if err != nil { - t.Fatalf("Schema.List: %v", err) - } - ts := schemas[table] + table, ts := firstTable(t, c) mk := marker(t) markerCol := markerColumn(t, ts) @@ -183,30 +185,16 @@ func TestE2E_BatchInsert(t *testing.T) { t.Fatalf("Batch insert: OK=false") } - time.Sleep(500 * time.Millisecond) - - page, err := c.From(table).Select(markerCol). - Where(markerCol, OpEq, mk). - Limit(10). - FetchUntyped(ctx) - if err != nil { - t.Fatalf("Query after batch insert failed: %v", err) - } - if len(page.Data) < 3 { - t.Fatalf("expected >= 3 rows for marker %q, got %d", mk, len(page.Data)) + got := waitForRows(t, c, table, markerCol, mk, 3) + if len(got) < 3 { + t.Fatalf("expected >= 3 rows for marker %q, got %d", mk, len(got)) } } func TestE2E_QueryBuilder(t *testing.T) { c := e2eClient(t) ctx := context.Background() - table := firstTable(t, c) - - schemas, err := c.Schema.List(ctx) - if err != nil { - t.Fatalf("Schema.List: %v", err) - } - ts := schemas[table] + table, ts := firstTable(t, c) // Pick two columns for a minimal projection. var cols []string @@ -216,6 +204,9 @@ func TestE2E_QueryBuilder(t *testing.T) { break } } + if len(cols) == 0 { + t.Skipf("e2e: table %q has no columns", table) + } page, err := c.From(table). Select(cols...). @@ -235,7 +226,7 @@ func TestE2E_QueryBuilder(t *testing.T) { func TestE2E_TypedFetch(t *testing.T) { c := e2eClient(t) ctx := context.Background() - table := firstTable(t, c) + table, _ := firstTable(t, c) q := c.From(table).SelectAll().Limit(3) page, err := FetchTyped[map[string]any](ctx, q) @@ -408,7 +399,10 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { case strings.Contains(ct, "bool"): row[col.Name] = false default: - row[col.Name] = "" + // No safe synthetic value for this type (Array, Map, Tuple, UUID, + // ...) — an empty string would make the insert fail with a type + // error that looks like an SDK defect. + t.Skipf("e2e: table %q requires column %q of unsupported type %q", ts.Name, col.Name, col.Type) } } if !markerSet { @@ -426,12 +420,10 @@ func skipIfUnauthorized(t *testing.T, err error, op string) { } } -// isHTTPStatus checks whether err is a wavehouse.Error with the given status. +// isHTTPStatus checks whether err wraps a wavehouse.Error with the given status. func isHTTPStatus(err error, status int) bool { - if err == nil { - return false - } - if e, ok := err.(*Error); ok { + var e *Error + if errors.As(err, &e) { return e.Status == status } return false diff --git a/clients/go/errors.go b/clients/go/errors.go index f2ea1b24..8ec3b2e8 100644 --- a/clients/go/errors.go +++ b/clients/go/errors.go @@ -53,7 +53,7 @@ func parseErrorResponse(res *http.Response) *Error { msg = http.StatusText(res.StatusCode) } - retryable := res.StatusCode == http.StatusServiceUnavailable || res.StatusCode >= 500 + retryable := res.StatusCode >= 500 || res.StatusCode == http.StatusTooManyRequests return &Error{ Status: res.StatusCode, Code: fmt.Sprintf("HTTP_%d", res.StatusCode), diff --git a/clients/go/errors_test.go b/clients/go/errors_test.go index c5af80df..160cb5c4 100644 --- a/clients/go/errors_test.go +++ b/clients/go/errors_test.go @@ -76,24 +76,28 @@ func TestParseErrorResponse(t *testing.T) { func TestParseErrorResponse_5xxRetryable(t *testing.T) { tests := []struct { + name string status int retryable bool }{ - {400, false}, - {403, false}, - {500, true}, - {503, true}, + {"BadRequest", 400, false}, + {"Forbidden", 403, false}, + {"TooManyRequests", 429, true}, + {"InternalServerError", 500, true}, + {"ServiceUnavailable", 503, true}, } for _, tt := range tests { - res := &http.Response{ - StatusCode: tt.status, - Body: io.NopCloser(strings.NewReader(`{"error":"test"}`)), - Header: http.Header{}, - } - e := parseErrorResponse(res) - if e.Retryable != tt.retryable { - t.Errorf("status %d: want retryable=%v, got %v", tt.status, tt.retryable, e.Retryable) - } + t.Run(tt.name, func(t *testing.T) { + res := &http.Response{ + StatusCode: tt.status, + Body: io.NopCloser(strings.NewReader(`{"error":"test"}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Retryable != tt.retryable { + t.Errorf("status %d: want retryable=%v, got %v", tt.status, tt.retryable, e.Retryable) + } + }) } } diff --git a/clients/go/example_test.go b/clients/go/example_test.go index 03d8b518..b8e566c7 100644 --- a/clients/go/example_test.go +++ b/clients/go/example_test.go @@ -17,8 +17,9 @@ func ExampleNewClient() { }) // Health check — returns nil when the server is reachable. - err := client.Sys.Health(context.Background()) - _ = err + if err := client.Sys.Health(context.Background()); err != nil { + log.Fatal(err) + } } func ExampleNewClient_withAuth() { @@ -40,12 +41,15 @@ func ExampleClient_From() { }) // Query with the builder. - page, _ := client.From("clicks"). + page, err := client.From("clicks"). Select("page", "button"). Where("page", wavehouse.OpEq, "/home"). OrderBy("page", "asc"). Limit(10). FetchUntyped(context.Background()) + if err != nil { + log.Fatal(err) + } for _, row := range page.Data { fmt.Println(row["page"]) diff --git a/clients/go/http.go b/clients/go/http.go index 33c3785c..f56a1235 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -8,6 +8,7 @@ import ( "fmt" "io" "math" + "math/rand/v2" "net/http" "net/url" "strconv" @@ -16,6 +17,10 @@ import ( var errAborted = &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} +// maxRetryAfter caps server-supplied Retry-After delays so a hostile or +// misconfigured server can't park the calling goroutine for hours. +const maxRetryAfter = 30 * time.Second + // httpContext carries per-client state needed by every request. type httpContext struct { baseURL string @@ -130,10 +135,10 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a apiErr := parseErrorResponse(res) _ = res.Body.Close() - // 503 with Retry-After: wait the specified duration. - if res.StatusCode == http.StatusServiceUnavailable { + // 503/429 with Retry-After: wait the specified duration (capped). + if res.StatusCode == http.StatusServiceUnavailable || res.StatusCode == http.StatusTooManyRequests { if ra := res.Header.Get("Retry-After"); ra != "" && attempt < maxAttempts-1 { - delay := 30 * time.Second + delay := maxRetryAfter if secs, parseErr := strconv.Atoi(ra); parseErr == nil && secs > 0 { delay = time.Duration(secs) * time.Second } else if parsed, parseErr := http.ParseTime(ra); parseErr == nil { @@ -141,6 +146,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a delay = d } } + delay = min(delay, maxRetryAfter) if sleepErr := sleepWithContext(ctx, delay); sleepErr != nil { return errAborted } @@ -177,6 +183,8 @@ func backoff(attempt int) time.Duration { if ms > 30000 { ms = 30000 } + // ±20% jitter so clients failing at the same moment don't retry in lockstep. + ms *= 0.8 + 0.4*rand.Float64() //nolint:gosec // retry jitter, not cryptographic return time.Duration(ms) * time.Millisecond } diff --git a/clients/go/http_test.go b/clients/go/http_test.go index 5a095c62..ede6543c 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -200,19 +200,25 @@ func TestDoRequest_EmptyResponse(t *testing.T) { func TestBackoff(t *testing.T) { tests := []struct { + name string attempt int - want time.Duration + base time.Duration }{ - {0, 1 * time.Second}, - {1, 2 * time.Second}, - {2, 4 * time.Second}, - {3, 8 * time.Second}, - {10, 30 * time.Second}, // capped at 30s + {"Attempt0", 0, 1 * time.Second}, + {"Attempt1", 1, 2 * time.Second}, + {"Attempt2", 2, 4 * time.Second}, + {"Attempt3", 3, 8 * time.Second}, + {"CappedAt30s", 10, 30 * time.Second}, } for _, tt := range tests { - got := backoff(tt.attempt) - if got != tt.want { - t.Errorf("backoff(%d) = %v, want %v", tt.attempt, got, tt.want) - } + t.Run(tt.name, func(t *testing.T) { + // backoff applies ±20% jitter around the exponential base. + lo := time.Duration(float64(tt.base) * 0.8) + hi := time.Duration(float64(tt.base) * 1.2) + got := backoff(tt.attempt) + if got < lo || got > hi { + t.Errorf("backoff(%d) = %v, want within [%v, %v]", tt.attempt, got, lo, hi) + } + }) } } diff --git a/clients/go/live_query.go b/clients/go/live_query.go index 1af2b770..d47ce618 100644 --- a/clients/go/live_query.go +++ b/clients/go/live_query.go @@ -3,6 +3,7 @@ package wavehouse import ( "context" "sync" + "time" ) // LiveQueryHandle controls a live query that combines historical backfill @@ -10,7 +11,13 @@ import ( type LiveQueryHandle struct { stream *StreamController cancel context.CancelFunc + unsub func() closeOnce sync.Once + + mu sync.Mutex + buffer []StreamEvent + buffering bool + closed bool } // newLiveQuery starts a live query: opens the stream immediately, fetches @@ -19,43 +26,50 @@ func newLiveQuery( stream *StreamController, fetchFn func(ctx context.Context) ([]map[string]any, error), sub *StreamSubscriber, - filters []QueryFilter, ) *LiveQueryHandle { ctx, cancel := context.WithCancel(context.Background()) //nolint:gosec // cancel is called in Close() lq := &LiveQueryHandle{ - stream: stream, - cancel: cancel, + stream: stream, + cancel: cancel, + buffering: true, } - var ( - mu sync.Mutex - buffer []StreamEvent - buffering = true - closed = false - ) - - // Step 1: Subscribe to live events and buffer them. - stream.Subscribe(&StreamSubscriber{ + // Step 1: Subscribe to live events and buffer them. User callbacks are + // invoked outside lq.mu so a subscriber may call Close() without + // deadlocking. + lq.unsub = stream.Subscribe(&StreamSubscriber{ Next: func(event StreamEvent) { - mu.Lock() - defer mu.Unlock() - if closed { + lq.mu.Lock() + if lq.closed { + lq.mu.Unlock() + return + } + if lq.buffering { + lq.buffer = append(lq.buffer, event) + lq.mu.Unlock() return } - if buffering { - buffer = append(buffer, event) - } else if sub.Next != nil { + lq.mu.Unlock() + if sub.Next != nil { sub.Next(event) } }, - Status: sub.Status, - Error: sub.Error, + Status: func(s StreamStatus) { + if !lq.isClosed() && sub.Status != nil { + sub.Status(s) + } + }, + Error: func(err error) { + if !lq.isClosed() && sub.Error != nil { + sub.Error(err) + } + }, }) // Step 2–5: Fetch historical and flush. go func() { rows, err := fetchFn(ctx) - if ctx.Err() != nil { + if ctx.Err() != nil || lq.isClosed() { return } @@ -65,19 +79,23 @@ func newLiveQuery( } if err != nil { - mu.Lock() - buffering = false - buffer = nil - mu.Unlock() + lq.mu.Lock() + lq.buffering = false + lq.buffer = nil + lq.mu.Unlock() return } - // Step 4: Deduplicate buffered events. - var lastTimestamp string - if len(rows) > 0 { - lastRow := rows[len(rows)-1] - if ts, ok := lastRow["received_timestamp"].(string); ok { - lastTimestamp = ts + // Step 4: Dedup bound — the maximum backfilled timestamp, compared as + // parsed times. OrderBy(..., "desc") makes the *last* row the oldest, + // and RFC3339 strings with varying fractional digits don't sort + // lexically, so neither "last row" nor raw string compare is safe. + var lastTS time.Time + for _, row := range rows { + if s, ok := row["received_timestamp"].(string); ok { + if ts, perr := time.Parse(time.RFC3339Nano, s); perr == nil && ts.After(lastTS) { + lastTS = ts + } } } @@ -85,31 +103,31 @@ func newLiveQuery( // buffer is provably empty under the lock — prevents concurrent // sub.Next calls and preserves delivery order. for { - mu.Lock() - if closed { - mu.Unlock() + lq.mu.Lock() + if lq.closed { + lq.mu.Unlock() return } - pending := buffer - buffer = nil + pending := lq.buffer + lq.buffer = nil if len(pending) == 0 { - buffering = false - mu.Unlock() + lq.buffering = false + lq.mu.Unlock() break } - mu.Unlock() + lq.mu.Unlock() for _, event := range pending { - mu.Lock() - c := closed - mu.Unlock() - if c { + if lq.isClosed() { return } - // <= dedupes events already delivered in the backfill. - // Sub-millisecond received_timestamp precision makes collisions rare. - if lastTimestamp != "" && event.Timestamp <= lastTimestamp { - continue + // Skip events already delivered in the backfill. + // Sub-millisecond received_timestamp precision makes + // boundary collisions rare. + if !lastTS.IsZero() { + if ts, perr := time.Parse(time.RFC3339Nano, event.Timestamp); perr == nil && !ts.After(lastTS) { + continue + } } if sub.Next != nil { sub.Next(event) @@ -118,21 +136,25 @@ func newLiveQuery( } }() - // Cleanup on context cancel. - go func() { - <-ctx.Done() - mu.Lock() - closed = true - buffer = nil - mu.Unlock() - }() - return lq } -// Close shuts down the live query and the underlying stream. +func (lq *LiveQueryHandle) isClosed() bool { + lq.mu.Lock() + defer lq.mu.Unlock() + return lq.closed +} + +// Close shuts down the live query and the underlying stream. The close state +// is applied synchronously: no new subscriber callbacks start after Close +// returns (a callback already in flight may still complete). func (lq *LiveQueryHandle) Close() { lq.closeOnce.Do(func() { + lq.mu.Lock() + lq.closed = true + lq.buffer = nil + lq.mu.Unlock() + lq.unsub() lq.cancel() lq.stream.Close() }) diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index 2801c1ab..76acc176 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -5,11 +5,14 @@ import ( "encoding/json" "net/http" "net/http/httptest" + "sync" "testing" ) -func nsClient(handler http.Handler) *Client { +func nsClient(t *testing.T, handler http.Handler) *Client { + t.Helper() srv := httptest.NewServer(handler) + t.Cleanup(srv.Close) return NewClient(Config{ BaseURL: srv.URL, HTTPClient: srv.Client(), @@ -18,7 +21,7 @@ func nsClient(handler http.Handler) *Client { } func TestSysNamespace_Health(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/v1/health" { t.Errorf("want /v1/health, got %s", r.URL.Path) } @@ -31,7 +34,7 @@ func TestSysNamespace_Health(t *testing.T) { } func TestSchemaNamespace_List(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/v1/schema" { t.Errorf("want /v1/schema, got %s", r.URL.Path) } @@ -49,22 +52,27 @@ func TestSchemaNamespace_List(t *testing.T) { } func TestSchemaNamespace_Refresh(t *testing.T) { + var mu sync.Mutex var gotMethod string - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() gotMethod = r.Method + mu.Unlock() w.WriteHeader(200) })) err := c.Schema.Refresh(context.Background()) if err != nil { t.Fatal(err) } + mu.Lock() + defer mu.Unlock() if gotMethod != "POST" { t.Fatalf("want POST, got %s", gotMethod) } } func TestPolicyNamespace_GetSetValidate(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) @@ -99,7 +107,7 @@ func TestPolicyNamespace_GetSetValidate(t *testing.T) { func TestDLQNamespace(t *testing.T) { t.Run("List", func(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) })) stats, err := c.DLQ.List(context.Background()) @@ -112,15 +120,20 @@ func TestDLQNamespace(t *testing.T) { }) t.Run("Table", func(t *testing.T) { + var mu sync.Mutex var gotParam string - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() gotParam = r.URL.Query().Get("table") + mu.Unlock() _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) })) _, err := c.DLQ.Table(context.Background(), "clicks") if err != nil { t.Fatal(err) } + mu.Lock() + defer mu.Unlock() if gotParam != "clicks" { t.Fatalf("want table=clicks, got %s", gotParam) } @@ -128,7 +141,7 @@ func TestDLQNamespace(t *testing.T) { } func TestPipesNamespace_CRUD(t *testing.T) { - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": if r.URL.Path == "/v1/admin/pipes" { @@ -171,12 +184,15 @@ func TestPipesNamespace_CRUD(t *testing.T) { } func TestPipeRef_Fetch(t *testing.T) { + var mu sync.Mutex var gotPath, gotMethod string var gotBody map[string]any - c := nsClient(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() gotPath = r.URL.Path gotMethod = r.Method _ = json.NewDecoder(r.Body).Decode(&gotBody) + mu.Unlock() _ = json.NewEncoder(w).Encode([]map[string]any{{"count": 42}}) })) rows, err := Fetch[map[string]any](context.Background(), c.Pipe("top_pages", map[string]any{"limit": 10})) @@ -186,6 +202,8 @@ func TestPipeRef_Fetch(t *testing.T) { if len(rows) != 1 { t.Fatalf("want 1 row, got %d", len(rows)) } + mu.Lock() + defer mu.Unlock() if gotPath != "/v1/pipes/top_pages" { t.Fatalf("want /v1/pipes/top_pages, got %s", gotPath) } diff --git a/clients/go/pipes.go b/clients/go/pipes.go index 767078a6..71b25720 100644 --- a/clients/go/pipes.go +++ b/clients/go/pipes.go @@ -2,6 +2,7 @@ package wavehouse import ( "context" + "fmt" "net/url" ) @@ -17,7 +18,7 @@ func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { method: "GET", path: "/v1/admin/pipes", }, &pipes); err != nil { - return nil, err + return nil, fmt.Errorf("list pipes: %w", err) } return pipes, nil } @@ -29,26 +30,32 @@ func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { method: "GET", path: "/v1/admin/pipes/" + url.PathEscape(name), }, &pipe); err != nil { - return nil, err + return nil, fmt.Errorf("get pipe %q: %w", name, err) } return &pipe, nil } // Set creates or updates a pipe. Admin-only. func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) error { - return doRequest(ctx, p.ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "PUT", path: "/v1/admin/pipes/" + url.PathEscape(name), body: def, - }, nil) + }, nil); err != nil { + return fmt.Errorf("set pipe %q: %w", name, err) + } + return nil } // Delete removes a pipe by name. Admin-only. func (p *PipesNamespace) Delete(ctx context.Context, name string) error { - return doRequest(ctx, p.ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "DELETE", path: "/v1/admin/pipes/" + url.PathEscape(name), - }, nil) + }, nil); err != nil { + return fmt.Errorf("delete pipe %q: %w", name, err) + } + return nil } // PipeDef is the definition body for creating/updating a pipe (Pipe minus name). @@ -79,7 +86,7 @@ func Fetch[Row any](ctx context.Context, p *PipeRef) ([]Row, error) { path: "/v1/pipes/" + url.PathEscape(p.name), body: body, }, &rows); err != nil { - return nil, err + return nil, fmt.Errorf("execute pipe %q: %w", p.name, err) } return rows, nil } diff --git a/clients/go/policy.go b/clients/go/policy.go index c8a2306b..9fbc5bb6 100644 --- a/clients/go/policy.go +++ b/clients/go/policy.go @@ -1,6 +1,9 @@ package wavehouse -import "context" +import ( + "context" + "fmt" +) // PolicyNamespace provides admin-only access-control policy management. type PolicyNamespace struct { @@ -14,18 +17,21 @@ func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { method: "GET", path: "/v1/admin/policy", }, &pol); err != nil { - return nil, err + return nil, fmt.Errorf("get policy: %w", err) } return &pol, nil } // Set replaces the entire access-control policy. Admin-only. func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { - return doRequest(ctx, p.ctx, requestOptions{ + if err := doRequest(ctx, p.ctx, requestOptions{ method: "PUT", path: "/v1/admin/policy", body: pol, - }, nil) + }, nil); err != nil { + return fmt.Errorf("set policy: %w", err) + } + return nil } // Validate checks a policy without applying it (dry run). Admin-only. @@ -36,7 +42,7 @@ func (p *PolicyNamespace) Validate(ctx context.Context, pol *Policy) (*Validatio path: "/v1/admin/policy/validate", body: pol, }, &result); err != nil { - return nil, err + return nil, fmt.Errorf("validate policy: %w", err) } return &result, nil } diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 65679ad1..2855d367 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -212,7 +212,7 @@ func (q *QueryBuilder) LiveQuery(sub *StreamSubscriber, opts *StreamOptions) *Li } return page.Data, nil } - return newLiveQuery(stream, fetchFn, sub, q.state.filters) + return newLiveQuery(stream, fetchFn, sub) } func (q *QueryBuilder) aggDefault(fn, prefix, column, alias string) *QueryBuilder { @@ -293,6 +293,14 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro } next := q.clone(func(s *queryState) { + // Replace an existing cursor filter instead of appending — otherwise + // page N carries N stacked filters on the cursor column. + for i := range s.filters { + if s.filters[i].Column == cursor.Column && s.filters[i].Op == cursorOp { + s.filters[i].Value = lastValue + return + } + } s.filters = append(s.filters, QueryFilter{ Column: cursor.Column, Op: cursorOp, diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index 84261a05..71d2808e 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -3,32 +3,44 @@ package wavehouse import ( "context" "encoding/json" + "io" "net/http" "net/http/httptest" + "sync" "testing" ) -func queryTestCtx(handler http.Handler) (*Client, *httptest.Server) { +func queryTestCtx(t *testing.T, handler http.Handler) *Client { + t.Helper() srv := httptest.NewServer(handler) - c := NewClient(Config{ + t.Cleanup(srv.Close) + return NewClient(Config{ BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}, }) - return c, srv } func captureQueryBody(t *testing.T, handler http.Handler) (*Client, func() map[string]any) { t.Helper() + // body is written on the server goroutine and read on the test goroutine; + // the mutex is what makes that visible under -race. + var mu sync.Mutex var body []byte wrapper := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - raw := make([]byte, 32*1024) - n, _ := r.Body.Read(raw) - body = raw[:n] + raw, err := io.ReadAll(r.Body) + if err != nil { + t.Errorf("read request body: %v", err) + } + mu.Lock() + body = raw + mu.Unlock() handler.ServeHTTP(w, r) }) - c, _ := queryTestCtx(wrapper) + c := queryTestCtx(t, wrapper) return c, func() map[string]any { + mu.Lock() + defer mu.Unlock() var m map[string]any _ = json.Unmarshal(body, &m) return m @@ -41,7 +53,7 @@ var emptyRows = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { }) func TestQueryBuilder_Immutability(t *testing.T) { - c, _ := queryTestCtx(emptyRows) + c := queryTestCtx(t, emptyRows) b1 := c.From("clicks").Select("page") b2 := b1.Where("score", OpGt, 10) if b1 == b2 { @@ -121,14 +133,16 @@ func TestQueryBuilder_AllOperators(t *testing.T) { {OpNotLike, "not_like"}, } for _, tt := range ops { - c, getBody := captureQueryBody(t, emptyRows) - _, _ = c.From("clicks").Select("x").Where("col", tt.sdk, "v").FetchUntyped(context.Background()) - body := getBody() - filters := body["filters"].([]any) - f := filters[0].(map[string]any) - if f["op"] != tt.wire { - t.Errorf("%s: want wire op %s, got %s", tt.sdk, tt.wire, f["op"]) - } + t.Run(tt.wire, func(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("x").Where("col", tt.sdk, "v").FetchUntyped(context.Background()) + body := getBody() + filters := body["filters"].([]any) + f := filters[0].(map[string]any) + if f["op"] != tt.wire { + t.Errorf("want wire op %s, got %s", tt.wire, f["op"]) + } + }) } } @@ -198,10 +212,9 @@ func TestQueryBuilder_TimeRange(t *testing.T) { } func TestQueryBuilder_Pagination_HasMore(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) })) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) if err != nil { @@ -216,10 +229,9 @@ func TestQueryBuilder_Pagination_HasMore(t *testing.T) { } func TestQueryBuilder_Pagination_NoOrderNoNext(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) })) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client(), Options: &ClientOptions{MaxRetries: 0}}) page, err := c.From("clicks").Select("id").Limit(2).FetchUntyped(context.Background()) if err != nil { diff --git a/clients/go/stream.go b/clients/go/stream.go index 2fabf1e2..af14cd7f 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -201,7 +201,7 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str return } - lastID, err := sc.connect(ctx, hctx, table, since) + lastID, live, err := sc.connect(ctx, hctx, table, since) // Persist the last event ID so the next reconnect resumes from it. if lastID != "" { since = lastID @@ -210,6 +210,12 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str return } + // A connection that reached "live" resets the backoff so a long-lived + // stream doesn't inherit a maxed-out delay on its first drop. + if live { + attempt = 0 + } + if err != nil { sc.emitError(&Error{ Status: 0, @@ -232,11 +238,12 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str } // connect opens a single SSE connection and reads events until it closes. -// Returns the last seen event ID (empty if none) and any error. -func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table, since string) (string, error) { +// Returns the last seen event ID (empty if none), whether the connection +// reached the live state, and any error. +func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table, since string) (string, bool, error) { u, err := url.Parse(hctx.baseURL + "/v1/stream") if err != nil { - return "", err + return "", false, err } q := u.Query() q.Set("table", table) @@ -249,7 +256,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if hctx.auth != nil { token, err := hctx.auth(ctx) if err != nil { - return "", fmt.Errorf("auth: %w", err) + return "", false, fmt.Errorf("auth: %w", err) } if token != "" { authHeader = "Bearer " + token @@ -260,7 +267,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table req, err := http.NewRequestWithContext(ctx, "GET", u.String(), nil) if err != nil { - return "", err + return "", false, err } req.Header.Set("Accept", "text/event-stream") req.Header.Set("Cache-Control", "no-cache") @@ -270,12 +277,12 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table resp, err := hctx.httpClient.Do(req) if err != nil { - return "", err + return "", false, err } defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { - return "", fmt.Errorf("SSE connect failed: HTTP %d", resp.StatusCode) + return "", false, fmt.Errorf("SSE connect failed: HTTP %d", resp.StatusCode) } sc.setStatus(StatusLive) @@ -288,7 +295,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table for scanner.Scan() { if ctx.Err() != nil { - return lastID, nil + return lastID, true, nil } line := scanner.Text() @@ -324,7 +331,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table } } - return lastID, scanner.Err() + return lastID, true, scanner.Err() } // sseMessage matches the server's SSE event JSON shape. diff --git a/clients/go/sys.go b/clients/go/sys.go index e52c252c..ed875cdb 100644 --- a/clients/go/sys.go +++ b/clients/go/sys.go @@ -1,6 +1,9 @@ package wavehouse -import "context" +import ( + "context" + "fmt" +) // SysNamespace provides system health checks. type SysNamespace struct { @@ -10,8 +13,11 @@ type SysNamespace struct { // Health pings the server's public /v1/health endpoint. Returns nil when the // server is reachable and past boot, or an error describing the failure. func (s *SysNamespace) Health(ctx context.Context) error { - return doRequest(ctx, s.ctx, requestOptions{ + if err := doRequest(ctx, s.ctx, requestOptions{ method: "GET", path: "/v1/health", - }, nil) + }, nil); err != nil { + return fmt.Errorf("health check: %w", err) + } + return nil } diff --git a/clients/go/table.go b/clients/go/table.go index d466d5b4..02bf4a0a 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -84,7 +84,7 @@ func (t *TableRef) Schema(ctx context.Context) (*TableSchema, error) { path: "/v1/schema", params: url.Values{"table": {t.table}}, }, &schema); err != nil { - return nil, err + return nil, fmt.Errorf("get schema for table %q: %w", t.table, err) } return &schema, nil } @@ -105,7 +105,7 @@ func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, e params: url.Values{"table": {t.table}}, body: data, }, &res); err != nil { - return nil, err + return nil, fmt.Errorf("insert into %q: %w", t.table, err) } ok := true if res.OK != nil { @@ -181,7 +181,7 @@ func (t *TableRef) sendNDJSON(ctx context.Context, ndjson string) (*InsertResult rawBody: ndjson, contentType: "application/x-ndjson", }, &res); err != nil { - return nil, err + return nil, fmt.Errorf("ingest into %q: %w", t.table, err) } result := &InsertResult{ OK: res.Failed == 0, diff --git a/clients/go/testdata/wire_cases.json b/clients/go/testdata/wire_cases.json index 4ec8613d..2c27b46a 100644 --- a/clients/go/testdata/wire_cases.json +++ b/clients/go/testdata/wire_cases.json @@ -543,5 +543,20 @@ "pipe_name": "my_pipe", "expected_path": "/v1/admin/pipes/my_pipe", "expected_method": "DELETE" + }, + { + "name": "cacheTTL is client-side only and not sent on the wire", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "cacheTTL", "args": [60] }, + { "method": "limit", "args": [5] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "select_all": true, + "limit": 5 + } } ] diff --git a/clients/go/types.go b/clients/go/types.go index b59333fd..0d17699f 100644 --- a/clients/go/types.go +++ b/clients/go/types.go @@ -179,15 +179,16 @@ type RolePermissions struct { MaxMemoryUsage any `json:"max_memory_usage,omitempty"` } -// PolicyFilter describes a policy filter predicate. Fields are pointers so an -// intentional empty-string comparison (e.g. Eq pointing at "") round-trips -// distinctly from an absent operator, matching the server's semantics. +// PolicyFilter describes a policy filter predicate. Fields are pointers with +// omitempty so an intentional empty-string comparison (e.g. Eq pointing at "") +// is sent as "", while an unset operator is omitted entirely — never null — +// matching the server's absent-operator semantics. type PolicyFilter struct { - Eq *string `json:"_eq"` - Neq *string `json:"_neq"` - Gt *string `json:"_gt"` - Lt *string `json:"_lt"` - In *string `json:"_in"` + Eq *string `json:"_eq,omitempty"` + Neq *string `json:"_neq,omitempty"` + Gt *string `json:"_gt,omitempty"` + Lt *string `json:"_lt,omitempty"` + In *string `json:"_in,omitempty"` } // ValidationResult is the response from policy validation. diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index 3e645eaf..a4d338e8 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -8,11 +8,12 @@ // client := wavehouse.NewClient(wavehouse.Config{ // BaseURL: "http://localhost:8080", // }) -// rows, err := client.From("clicks").SelectAll().Fetch(ctx) +// rows, err := client.From("clicks").SelectAll().FetchUntyped(ctx) package wavehouse import ( "context" + "fmt" "net/http" "strings" ) @@ -73,7 +74,9 @@ func NewClient(cfg Config) *Client { hc := cfg.HTTPClient if hc == nil { - hc = http.DefaultClient + // Not http.DefaultClient: it's mutable global state another package + // could reconfigure (timeout, transport, redirects) after we're built. + hc = &http.Client{} } c := &Client{ @@ -125,7 +128,7 @@ func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { body: map[string]string{"sql": query}, }, &rows) if err != nil { - return nil, err + return nil, fmt.Errorf("sql query: %w", err) } return rows, nil } diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md index a0941433..957713fd 100644 --- a/docs/src/content/docs/sdk/go/admin.md +++ b/docs/src/content/docs/sdk/go/admin.md @@ -43,7 +43,7 @@ policy, err := wh.Policy.Get(ctx) // Update policy. tenantFilter := "{{ jwt.app_metadata.tenant_id }}" -err = wh.Policy.Set(ctx, &wavehouse.Policy{ +policyDraft := &wavehouse.Policy{ DefaultRole: "viewer", Tables: map[string]wavehouse.TablePolicy{ "clicks": { @@ -58,7 +58,8 @@ err = wh.Policy.Set(ctx, &wavehouse.Policy{ }, }, }, -}) +} +err = wh.Policy.Set(ctx, policyDraft) // Validate without applying (dry run). result, err := wh.Policy.Validate(ctx, policyDraft) diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index d95a83d5..2202f95d 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -83,7 +83,7 @@ wh := wavehouse.NewClient(wavehouse.Config{ | `BaseURL` | `string` | — | WaveHouse server URL (required) | | `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider, called before each request. `nil` means unauthenticated access | | `Options` | `*ClientOptions` | `nil` | Transport tuning (see below) | -| `HTTPClient` | `*http.Client` | `http.DefaultClient` | Override for custom TLS, proxies, or test transports | +| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports | ### `ClientOptions` @@ -118,6 +118,13 @@ parameter fallback to worry about (that's a TypeScript-SDK-in-the-browser concern only; see its [equivalent note](/sdk#creating-a-client)). ::: +:::caution[Use HTTPS for authenticated non-local servers] +The SDK doesn't forbid `http://` base URLs — local development and +private-network deployments rely on them — but a bearer token sent over +plaintext HTTP is readable by anything on the path. Point authenticated +clients at `https://` endpoints outside a trusted network. +::: + ## Typed Rows (Generics) Pass a row type as a type parameter to get results decoded straight into @@ -148,7 +155,7 @@ parameter. ## Error Handling -All SDK operations return `(T, error)`. Errors are `*wavehouse.Error`; unwrap with `errors.As`: +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors are `*wavehouse.Error`; unwrap with `errors.As`. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close` — deliver errors through callbacks instead; see [Streaming](/sdk/go/streaming).) ```go page, err := wh.From("clicks").Fetch(ctx) @@ -161,7 +168,7 @@ if err != nil { } ``` -See [Reference → Error Handling](/sdk/go/reference/#error-handling) for retry behavior and error codes. +See [Reference → Error Handling](/sdk/go/reference#error-handling) for retry behavior and error codes. ## Differences from the TypeScript SDK diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 2b56fd5b..59c588bc 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -361,10 +361,12 @@ for page.HasMore && page.Next != nil { ## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` -Execute a raw SQL query. `/v1/admin/query` is admin-only: the caller's JWT -must resolve to the policy admin role (`admin_role`, `"admin"` by default). -A request with no token, or an invalid/expired one, falls back to the -`default_role` and is rejected. Package-level generic function — use +Execute a raw SQL query. `/v1/admin/query` is admin-only: for JWT callers, +the token must resolve to the policy admin role (`admin_role`, `"admin"` by +default) — a JWT request with no token, or an invalid/expired one, falls +back to the `default_role` and is rejected. Alternatively, a configured +operator key (`Authorization: Operator ` or `X-Operator-Key`) +authorizes `/v1/admin/*` without a JWT. Package-level generic function — use `map[string]any` for a dynamic/unknown schema. ```go diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 7e6e90cb..75cf8cad 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -4,7 +4,7 @@ description: "Error codes, context cancellation, the full API tree, and the code --- Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: -cancellation, the error model behind every SDK call's `(T, error)` return, +cancellation, the error model behind every request-response call's `(T, error)` return, the complete API tree at a glance, and the `wavehouse-codegen` tool that ships with the module. Compare with the TypeScript SDK's [Reference & CLI](/sdk/reference) page. @@ -38,10 +38,13 @@ background goroutine, torn down explicitly via `.Close()`. See ## Error Handling -The SDK never panics on API or network failures — every operation returns -`(T, error)`, and errors are always `*wavehouse.Error` (unwrap with -`errors.As`). This is the direct Go equivalent of the TypeScript SDK's "the -SDK never throws" guarantee. +The SDK never panics on API or network failures — every request-response +operation (queries, ingest, pipes, admin) returns `(T, error)`, and errors +are always `*wavehouse.Error` (unwrap with `errors.As`). Streaming lifecycle +methods (`Stream`, `Subscribe`, `Close`) don't return `(T, error)`; stream +errors are delivered via the subscriber's `Error` callback. This is the +direct Go equivalent of the TypeScript SDK's "the SDK never throws" +guarantee. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| @@ -135,9 +138,9 @@ Generate Go structs from a running WaveHouse instance. The module ships a `wavehouse-codegen` command under `cmd/`: ```bash +export WAVEHOUSE_AUTH= # avoids leaking the token via argv go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ --url http://localhost:8080 \ - --auth \ --out ./db_types.go \ --package myapp ``` @@ -149,8 +152,9 @@ go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go ``` Codegen reads `/v1/schema`, which is **admin-only**. Against a non-dev -server, pass an admin-role token with `--auth ` or the request is -denied with `403`. +server, provide an admin-role token or the request is denied with `403`. +Prefer the `WAVEHOUSE_AUTH` environment variable — a token passed with +`--auth ` ends up in shell history and process listings. **Options:** @@ -158,7 +162,7 @@ denied with `403`. |------|-------------|---------| | `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | | `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | -| `--auth`, `-a` | Bearer token (if auth required) | — | +| `--auth`, `-a` | Bearer token (if auth required); prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | | `--package`, `-p` | Go package name for the generated file | `main` | | `--help`, `-h` | Show usage and exit | — | @@ -195,11 +199,11 @@ tag. |------------------|---------| | `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | | `Bool` | `bool` | -| `UInt8` / `UInt16` / `UInt32` / `UInt64` | `uint8` / `uint16` / `uint32` / `uint64` | -| `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | +| `UInt8` / `UInt16` / `UInt32` | `uint8` / `uint16` / `uint32` | +| `Int8` / `Int16` / `Int32` | `int8` / `int16` / `int32` | | `Float32` | `float32` | | `Float64` | `float64` | -| `Decimal*`, `UInt128`/`UInt256`, `Int128`/`Int256` | `string` (big numbers are strings in JSON) | +| `UInt64`/`Int64`, `Decimal*`, `UInt128`/`UInt256`, `Int128`/`Int256` | `string` (ClickHouse quotes 64-bit-and-wider integers in JSON output — `output_format_json_quote_64bit_integers` — and the server forwards them verbatim) | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | | `Array(T)` | `[]T` | @@ -207,9 +211,10 @@ tag. | anything unrecognized | `any` | This differs from the TypeScript SDK's mapping in one notable way: Go's -codegen preserves ClickHouse's integer **widths** (`UInt32` → `uint32`, not -a generic `number`), since Go — unlike TypeScript — has native fixed-width -integer types. +codegen preserves ClickHouse's integer **widths** up to 32 bits (`UInt32` → +`uint32`, not a generic `number`), since Go — unlike TypeScript — has +native fixed-width integer types. 64-bit integers stay `string` because +that is what actually arrives on the wire. ## Testing @@ -227,6 +232,17 @@ cd clients/go go test ./... ``` +E2E tests (build tag `e2e`) run against a live WaveHouse instance and have +their own Make target, separate from the repo's `make test-e2e`: + +```bash +WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH= make test-go-sdk-e2e +``` + +`WAVEHOUSE_URL` defaults to `http://localhost:8080`; `WAVEHOUSE_AUTH` is +optional (admin-only cases skip without it). When the server is unreachable +the suite skips instead of failing. + Unlike the TypeScript SDK, the Go SDK isn't (yet) wired into the repo's `make test-e2e` harness — see the TypeScript SDK's [E2E Testing](/sdk/reference#e2e-testing) section for that suite's diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index aa4ca864..659fe9fe 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -13,7 +13,9 @@ WaveHouse also ships an official Go SDK `context.Context`-first, generics for typed rows. See the [Go SDK docs](/sdk/go). The two clients speak the same wire format, so everything below about tables, the query builder, streaming, and admin -endpoints carries over conceptually — only the language idioms differ. +endpoints carries over conceptually, but API and lifecycle details differ — +Go uses context-first calls and package-level generics, and streams must be +closed explicitly. ::: ## Installation diff --git a/go.mod b/go.mod index 0ea6370e..db6d4e11 100644 --- a/go.mod +++ b/go.mod @@ -47,7 +47,7 @@ require ( go.opentelemetry.io/otel/trace v1.44.0 go.opentelemetry.io/proto/otlp v1.10.0 golang.org/x/sync v0.21.0 - google.golang.org/grpc v1.81.1 + google.golang.org/grpc v1.82.1 gopkg.in/yaml.v3 v3.0.1 ) @@ -131,7 +131,7 @@ require ( github.com/jedib0t/go-pretty/v6 v6.7.10 // indirect github.com/jmespath/go-jmespath v0.4.0 // indirect github.com/joho/godotenv v1.5.1 // indirect - github.com/klauspost/compress v1.18.6 // indirect + github.com/klauspost/compress v1.18.7 // indirect github.com/knadh/profiler v0.2.0 // indirect github.com/kr/pretty v0.3.1 // indirect github.com/kr/text v0.2.0 // indirect @@ -196,17 +196,17 @@ require ( go.yaml.in/yaml/v2 v2.4.4 // indirect go.yaml.in/yaml/v3 v3.0.4 // indirect golang.org/x/arch v0.26.0 // indirect - golang.org/x/crypto v0.52.0 // indirect + golang.org/x/crypto v0.53.0 // indirect golang.org/x/exp v0.0.0-20260218203240-3dfff04db8fa // indirect golang.org/x/image v0.38.0 // indirect - golang.org/x/mod v0.35.0 // indirect - golang.org/x/net v0.55.0 // indirect - golang.org/x/sys v0.45.0 // indirect - golang.org/x/telemetry v0.0.0-20260421165255-392afab6f40e // indirect - golang.org/x/term v0.43.0 // indirect - golang.org/x/text v0.37.0 // indirect + golang.org/x/mod v0.37.0 // indirect + golang.org/x/net v0.56.0 // indirect + golang.org/x/sys v0.46.0 // indirect + golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 // indirect + golang.org/x/term v0.44.0 // indirect + golang.org/x/text v0.39.0 // indirect golang.org/x/time v0.15.0 // indirect - golang.org/x/tools v0.44.0 // indirect + golang.org/x/tools v0.47.0 // indirect golang.org/x/vuln v1.3.0 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect diff --git a/go.sum b/go.sum index d34ebedc..4990f509 100644 --- a/go.sum +++ b/go.sum @@ -221,8 +221,8 @@ github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwA github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= github.com/klauspost/compress v1.13.6/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk= -github.com/klauspost/compress v1.18.6 h1:2jupLlAwFm95+YDR+NwD2MEfFO9d4z4Prjl1XXDjuao= -github.com/klauspost/compress v1.18.6/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= +github.com/klauspost/compress v1.18.7 h1:aUyZsS4kH3QTKurYhAOwAHxllVPnOthb3vPfnF1Ehjw= +github.com/klauspost/compress v1.18.7/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ= github.com/knadh/profiler v0.2.0 h1:jaY0xlQs8iaWxKdvGHOftaZnX7d8l7yrCGQPSecwnng= github.com/knadh/profiler v0.2.0/go.mod h1:LqNkAu++MfFkbEDA63AmRaIf6UkGrLXyZ5VQQdekZiI= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= @@ -439,23 +439,23 @@ golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACk golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4= -golang.org/x/crypto v0.52.0 h1:RMs7fP2rXdep0CftQlK8Uf+kibLm7qkCcradZWYz988= -golang.org/x/crypto v0.52.0/go.mod h1:1QgfPxDqh0T2M/elOJtp9RvuR95kVjir0e6/BvEmGbc= +golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto= +golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio= golang.org/x/exp v0.0.0-20260218203240-3dfff04db8fa h1:Zt3DZoOFFYkKhDT3v7Lm9FDMEV06GpzjG2jrqW+QTE0= golang.org/x/exp v0.0.0-20260218203240-3dfff04db8fa/go.mod h1:K79w1Vqn7PoiZn+TkNpx3BUWUQksGO3JcVX6qIjytmA= golang.org/x/image v0.38.0 h1:5l+q+Y9JDC7mBOMjo4/aPhMDcxEptsX+Tt3GgRQRPuE= golang.org/x/image v0.38.0/go.mod h1:/3f6vaXC+6CEanU4KJxbcUZyEePbyKbaLoDOe4ehFYY= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= -golang.org/x/mod v0.35.0 h1:Ww1D637e6Pg+Zb2KrWfHQUnH2dQRLBQyAtpr/haaJeM= -golang.org/x/mod v0.35.0/go.mod h1:+GwiRhIInF8wPm+4AoT6L0FA1QWAad3OMdTRx4tFYlU= +golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ= +golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= -golang.org/x/net v0.55.0 h1:bcvxaJn3e1U6InsFWt1JUq1aSjnRxLzT2rtD2KfkDF8= -golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww= +golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o= +golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= @@ -475,27 +475,27 @@ golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBc golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/sys v0.45.0 h1:dO4czNzziLiiXplLQgBCEpCvXQ3dnkn0SdaZSYdQ+FY= -golang.org/x/sys v0.45.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= -golang.org/x/telemetry v0.0.0-20260421165255-392afab6f40e h1:OXgN37M6hqjaAvb7CJK9vJ+7Z/6lvIm5bXho5poo/Wk= -golang.org/x/telemetry v0.0.0-20260421165255-392afab6f40e/go.mod h1:kHjTxDEnAu6/Nl9lDkzjWpR+bmKfxeiRuSDlsMb70gE= +golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw= +golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57 h1:nwGZBCt+FnXUrGsj5vjzAsEmkcaFvd82BbOjECiFYZc= +golang.org/x/telemetry v0.0.0-20260625142307-59b4966ccb57/go.mod h1:3AWMyWHS+caVoiEXpiq6+tzKA40J4vQT3MYr80ZtQpc= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= -golang.org/x/term v0.43.0 h1:S4RLU2sB31O/NCl+zFN9Aru9A/Cq2aqKpTZJ6B+DwT4= -golang.org/x/term v0.43.0/go.mod h1:lrhlHNdQJHO+1qVYiHfFKVuVioJIheAc3fBSMFYEIsk= +golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc= +golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= -golang.org/x/text v0.37.0 h1:Cqjiwd9eSg8e0QAkyCaQTNHFIIzWtidPahFWR83rTrc= -golang.org/x/text v0.37.0/go.mod h1:a5sjxXGs9hsn/AJVwuElvCAo9v8QYLzvavO5z2PiM38= +golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus= +golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM= golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U= golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= -golang.org/x/tools v0.44.0 h1:UP4ajHPIcuMjT1GqzDWRlalUEoY+uzoZKnhOjbIPD2c= -golang.org/x/tools v0.44.0/go.mod h1:KA0AfVErSdxRZIsOVipbv3rQhVXTnlU6UhKxHd1seDI= +golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q= +golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA= golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM= golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY= golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM= @@ -512,8 +512,8 @@ google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa h1: google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:q4lMZS6kskjT5HvCPrnnypcDPVJqT/f4nfxmkE7gryY= google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk= google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= -google.golang.org/grpc v1.81.1 h1:VnnIIZ88UzOOKLukQi+ImGz8O1Wdp8nAGGnvOfEIWQQ= -google.golang.org/grpc v1.81.1/go.mod h1:xGH9GfzOyMTGIOXBJmXt+BX/V0kcdQbdcuwQ/zNw42I= +google.golang.org/grpc v1.82.1 h1:NnAxzGRA0677vCa4BUkOAnO5+FfQqVl9iUXeD0IqcGE= +google.golang.org/grpc v1.82.1/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 8eafc796..6d8630df 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -18,7 +18,14 @@ import { fileURLToPath } from "node:url"; const __dirname = dirname(fileURLToPath(import.meta.url)); // Import the built SDK. -const { createClient } = await import(join(__dirname, "../../clients/ts/dist/index.js")); +let createClient; +try { + ({ createClient } = await import(join(__dirname, "../../clients/ts/dist/index.js"))); +} catch (err) { + console.error("Cannot load the TypeScript SDK build. Run the SDK build first (e.g. `pnpm --dir clients/ts build`)."); + console.error(err.message); + process.exit(1); +} const cases = JSON.parse(readFileSync(join(__dirname, "../../clients/go/testdata/wire_cases.json"), "utf-8")); @@ -43,7 +50,7 @@ const server = createServer((req, res) => { if (req.url?.startsWith("/v1/dlq")) { res.end(JSON.stringify({ tables: {}, total: 0 })); } else if (req.url?.startsWith("/v1/schema") && req.method === "GET") { - res.end(JSON.stringify({})); + res.end(JSON.stringify([])); } else if (req.url === "/v1/admin/policy/validate" && req.method === "POST") { res.end(JSON.stringify({ valid: true })); } else if (req.url?.startsWith("/v1/admin/policy") && req.method === "GET") { @@ -53,13 +60,15 @@ const server = createServer((req, res) => { } else if (req.url === "/v1/admin/pipes" && req.method === "GET") { res.end(JSON.stringify([])); } else if (req.url?.startsWith("/v1/ingest")) { + // Same shapes the real server returns (internal/api/ingest.go). if (lastCapture.contentType === "application/x-ndjson") { res.end(JSON.stringify({ total: 0, succeeded: 0, failed: 0, duplicates: 0 })); } else { res.end(JSON.stringify({ ok: true })); } } else if (req.url === "/v1/health") { - res.end(""); + // Real server shape (internal/api/health.go). + res.end(JSON.stringify({ status: "ok" })); } else { res.end(JSON.stringify([])); } @@ -124,8 +133,17 @@ function applyQueryOps(wh, table, operations) { return q; } +// Compare request URIs by meaning: same path, same decoded query values, +// regardless of + vs %20 spelling or parameter order (mirrors the Go harness). function normalizePath(p) { - return p.replace(/\+/g, "%20"); + let u; + try { + u = new URL(p, "http://conformance.invalid"); + } catch { + return p; + } + u.searchParams.sort(); + return `${u.pathname}?${u.searchParams.toString()}`; } function deepEqual(a, b) { @@ -147,6 +165,8 @@ function sortKeys(v) { let passed = 0; let failed = 0; +let skipped = 0; +const skippedNames = []; const failures = []; for (const tc of cases) { @@ -213,7 +233,10 @@ for (const tc of cases) { await wh.pipes.delete(tc.pipe_name); break; default: - passed++; + // Not a pass — the Go harness skips these too. Fixture cases with a + // new endpoint value must be wired up here before they count. + skipped++; + skippedNames.push(`${tc.name} (endpoint: ${tc.endpoint})`); continue; } @@ -261,9 +284,16 @@ for (const tc of cases) { } } +server.closeAllConnections?.(); server.close(); -console.log(`\nWire-format conformance (TS SDK): ${passed} passed, ${failed} failed, ${cases.length} total\n`); +console.log( + `\nWire-format conformance (TS SDK): ${passed} passed, ${failed} failed, ${skipped} skipped, ${cases.length} total\n`, +); + +for (const name of skippedNames) { + console.log(` - skipped: ${name}`); +} for (const f of failures) { console.log(` ✗ ${f.name}`); @@ -276,4 +306,5 @@ if (failed > 0) { process.exit(1); } else { console.log(" ✓ All cases passed\n"); + process.exit(0); } From fc6346e312ea6919680b346d9fbec33a4bde628a Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 13:20:46 -0400 Subject: [PATCH 08/40] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=201=20=E2=80=94=20stream=20close=20race,=20tests,=20do?= =?UTF-8?q?c=20sync?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Streaming correctness: - Filtered stream no longer panics with send-on-closed-channel when Close() races in-flight inner deliveries: channel close and channel send now serialize under the controller mutex, and the wrapper unsubscribes from the inner stream before closing - Events() channel is fed only once Events() has been called, so Subscribe-only consumers no longer overflow a channel nobody reads; the buffer-full drop is logged once, not per event - Malformed-SSE log omits the payload (can carry tenant/PII fields) - Unparseable Retry-After falls back to backoff(attempt), not the 30s max; parsing extracted to retryAfterDelay for testability Tests (SDK coverage 33% → 80%+; streaming subsystem was 0%): - stream_test.go: SSE lifecycle over httptest, filtered close-under-load regression test, Events()/Connected, full filter-engine tables - live_query_test.go: backfill buffering, desc-order dedup bound, fetch error, no-callbacks-after-Close - http_test.go: retryAfterDelay table + live 429 Retry-After flow - Remaining unclosed httptest servers in table_test/http_test now use t.Cleanup Build plumbing: - test-go-sdk runs with -race; test/lint/fix aggregates now include the nested clients/go module; mangled test-go-sdk comment restored - New test-conformance-ts target runs the TS conformance runner (was wired to nothing); CI unit job runs it Docs: - development.md synced: suites/targets tables, CI unit job, project structure with clients/, Go SDK in the dev-loop section - reference.md: real codegen output (EventId not EventID; no bare int), initialism note, missing type-mapping rows (SimpleAggregateFunction, Time/Time64, Boolean, BFloat16), two-runner conformance wording, 401 row corrected (missing token → 403) here and in sdk/reference.md - Go SDK added alongside TS on the landing page, getting-started, and why-wavehouse comparison tables --- .github/workflows/ci.yml | 2 +- Makefile | 19 +- clients/go/http.go | 26 +- clients/go/http_test.go | 79 +++++- clients/go/live_query_test.go | 173 +++++++++++++ clients/go/stream.go | 59 ++++- clients/go/stream_test.go | 301 ++++++++++++++++++++++ clients/go/table_test.go | 8 + docs/src/content/docs/development.md | 23 +- docs/src/content/docs/getting-started.md | 3 +- docs/src/content/docs/index.mdx | 11 +- docs/src/content/docs/sdk/go/index.md | 5 +- docs/src/content/docs/sdk/go/reference.md | 38 ++- docs/src/content/docs/sdk/reference.md | 2 +- docs/src/content/docs/why-wavehouse.md | 4 +- 15 files changed, 686 insertions(+), 67 deletions(-) create mode 100644 clients/go/live_query_test.go create mode 100644 clients/go/stream_test.go diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8646bf53..1de4d6ab 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -205,7 +205,7 @@ jobs: with: go-cache-suffix: "-unit" - name: Run Go unit tests + SDK vitest + Go SDK tests - run: make test-unit test-ts test-go-sdk COV_DEFER=1 + run: make test-unit test-ts test-go-sdk test-conformance-ts COV_DEFER=1 - name: Upload coverage fragment uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: diff --git a/Makefile b/Makefile index fdb9a7ab..6e3d1c28 100644 --- a/Makefile +++ b/Makefile @@ -356,7 +356,7 @@ fmt-ts: pnpm-install $(call run,Biome (format),$(PNPM) -s -w run format,run make fix to apply formatting) .PHONY: lint -lint: lint-go lint-ts lint-md lint-prose ## Lint across Go (golangci-lint) + TS/JSON (Biome) + Markdown (markdownlint) + docs prose (misspell). Run `make fix` to apply --fix. +lint: lint-go lint-go-sdk lint-ts lint-md lint-prose ## Lint across Go (root + clients/go golangci-lint) + TS/JSON (Biome) + Markdown (markdownlint) + docs prose (misspell). Run `make fix` to apply --fix. .PHONY: lint-go lint-go: $(GOLANGCI_LINT) go-mod-download @@ -451,6 +451,8 @@ fix-go: $(GOLANGCI_LINT) @$(GOFUMPT) -w $(GO_DIRS) @$(GOIMPORTS) -w $(GO_DIRS) @$(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners + @echo "$(CYAN)==> Applying Go auto-fixes (Go SDK — nested module, outside GO_DIRS)...$(RESET)" + @cd clients/go && go mod tidy && $(GOFUMPT) -w . && $(GOIMPORTS) -w . && $(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners .PHONY: fix-ts fix-ts: pnpm-install @@ -681,7 +683,7 @@ test-unit: go-mod-download ## Run Go unit tests + render coverage + gate thresho # Hidden alias: `make test` matches `go test ./...` muscle memory; test-unit # is the explicit form. .PHONY: test -test: test-unit +test: test-unit test-go-sdk .PHONY: test-integration test-integration: go-mod-download ## Run Go integration tests + render coverage + gate threshold (requires Docker) @@ -726,11 +728,20 @@ test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against sui # test-go-sdk: unit tests for clients/go/ — a nested Go module (its own # go.mod), so it's outside test-unit's ./internal/... ./cmd/... scope and -# test-go-sdk: nested module — needs its own target. +# needs its own target. -race because the SDK's streaming subsystem is the +# most concurrent code in the repo. .PHONY: test-go-sdk test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" - @cd clients/go && go test ./... + @cd clients/go && go test -race ./... + +# test-conformance-ts: the TS half of the cross-SDK wire-format conformance +# suite (the Go half is clients/go/conformance_test.go, run by test-go-sdk). +# Both replay clients/go/testdata/wire_cases.json. +.PHONY: test-conformance-ts +test-conformance-ts: build-ts ## Run TS SDK wire-format conformance against the shared fixture + @printf "$(CYAN)==> Running TS wire-format conformance...$(RESET)\n" + @node tests/conformance/conformance_ts.mjs # test-go-sdk-e2e: E2E against live server. WAVEHOUSE_URL + WAVEHOUSE_AUTH env vars. .PHONY: test-go-sdk-e2e diff --git a/clients/go/http.go b/clients/go/http.go index f56a1235..0fc59d9e 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -138,16 +138,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a // 503/429 with Retry-After: wait the specified duration (capped). if res.StatusCode == http.StatusServiceUnavailable || res.StatusCode == http.StatusTooManyRequests { if ra := res.Header.Get("Retry-After"); ra != "" && attempt < maxAttempts-1 { - delay := maxRetryAfter - if secs, parseErr := strconv.Atoi(ra); parseErr == nil && secs > 0 { - delay = time.Duration(secs) * time.Second - } else if parsed, parseErr := http.ParseTime(ra); parseErr == nil { - if d := time.Until(parsed); d > 0 { - delay = d - } - } - delay = min(delay, maxRetryAfter) - if sleepErr := sleepWithContext(ctx, delay); sleepErr != nil { + if sleepErr := sleepWithContext(ctx, retryAfterDelay(ra, attempt)); sleepErr != nil { return errAborted } lastErr = apiErr @@ -178,6 +169,21 @@ func buildURL(base, path string, params url.Values) string { return u } +// retryAfterDelay resolves a Retry-After header (delta-seconds or HTTP-date) +// into a wait, clamped to maxRetryAfter. An unparseable header falls back to +// the ordinary backoff for this attempt, not the maximum. +func retryAfterDelay(ra string, attempt int) time.Duration { + delay := backoff(attempt) + if secs, err := strconv.Atoi(ra); err == nil && secs > 0 { + delay = time.Duration(secs) * time.Second + } else if parsed, err := http.ParseTime(ra); err == nil { + if d := time.Until(parsed); d > 0 { + delay = d + } + } + return min(delay, maxRetryAfter) +} + func backoff(attempt int) time.Duration { ms := 1000 * math.Pow(2, float64(attempt)) if ms > 30000 { diff --git a/clients/go/http_test.go b/clients/go/http_test.go index ede6543c..db1693b3 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -10,8 +10,10 @@ import ( "time" ) -func testCtx(handler http.Handler) httpContext { +func testCtx(t *testing.T, handler http.Handler) httpContext { + t.Helper() srv := httptest.NewServer(handler) + t.Cleanup(srv.Close) return httpContext{ baseURL: srv.URL, maxRetries: 0, @@ -20,7 +22,7 @@ func testCtx(handler http.Handler) httpContext { } func TestDoRequest_SuccessfulGET(t *testing.T) { - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) })) @@ -41,7 +43,7 @@ func TestDoRequest_SuccessfulGET(t *testing.T) { func TestDoRequest_POSTWithBody(t *testing.T) { var gotBody map[string]string var gotCT string - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotCT = r.Header.Get("Content-Type") _ = json.NewDecoder(r.Body).Decode(&gotBody) w.WriteHeader(200) @@ -66,7 +68,7 @@ func TestDoRequest_POSTWithBody(t *testing.T) { func TestDoRequest_RawBody(t *testing.T) { var gotBody string var gotCT string - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotCT = r.Header.Get("Content-Type") raw := make([]byte, 1024) n, _ := r.Body.Read(raw) @@ -93,7 +95,7 @@ func TestDoRequest_RawBody(t *testing.T) { func TestDoRequest_AuthInjection(t *testing.T) { var gotAuth string - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotAuth = r.Header.Get("Authorization") w.WriteHeader(200) })) @@ -113,7 +115,7 @@ func TestDoRequest_AuthInjection(t *testing.T) { func TestDoRequest_4xxNotRetried(t *testing.T) { var count atomic.Int32 - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { count.Add(1) w.Header().Set("Content-Type", "application/json") w.WriteHeader(404) @@ -136,7 +138,7 @@ func TestDoRequest_4xxNotRetried(t *testing.T) { func TestDoRequest_5xxRetried(t *testing.T) { var count atomic.Int32 - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { n := count.Add(1) if n < 3 { w.Header().Set("Content-Type", "application/json") @@ -162,7 +164,7 @@ func TestDoRequest_5xxRetried(t *testing.T) { } func TestDoRequest_AbortedOnCancel(t *testing.T) { - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { time.Sleep(5 * time.Second) })) @@ -180,7 +182,7 @@ func TestDoRequest_AbortedOnCancel(t *testing.T) { } func TestDoRequest_EmptyResponse(t *testing.T) { - hctx := testCtx(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(200) })) @@ -222,3 +224,62 @@ func TestBackoff(t *testing.T) { }) } } + +func TestRetryAfterDelay(t *testing.T) { + tests := []struct { + name string + ra string + want time.Duration + }{ + {"DeltaSeconds", "5", 5 * time.Second}, + {"ClampedToMax", "3600", maxRetryAfter}, + {"HTTPDateFuture", time.Now().Add(10 * time.Second).UTC().Format(http.TimeFormat), 0}, // range-checked below + {"Garbage", "not-a-delay", 0}, // range-checked below + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := retryAfterDelay(tt.ra, 0) + switch tt.name { + case "HTTPDateFuture": + if got <= 0 || got > 10*time.Second { + t.Fatalf("want ~10s, got %v", got) + } + case "Garbage": + // Falls back to backoff(0): 1s ±20% jitter. + if got < 800*time.Millisecond || got > 1200*time.Millisecond { + t.Fatalf("want backoff(0) fallback, got %v", got) + } + default: + if got != tt.want { + t.Fatalf("want %v, got %v", tt.want, got) + } + } + }) + } +} + +func TestDoRequest_429RetriesWithRetryAfter(t *testing.T) { + var calls atomic.Int64 + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + if calls.Add(1) == 1 { + w.Header().Set("Retry-After", "1") + w.WriteHeader(http.StatusTooManyRequests) + return + } + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + })) + hctx.maxRetries = 1 + + start := time.Now() + var result map[string]string + err := doRequest(context.Background(), hctx, requestOptions{method: "GET", path: "/x"}, &result) + if err != nil { + t.Fatalf("want success after 429 retry, got %v", err) + } + if got := calls.Load(); got != 2 { + t.Fatalf("want 2 attempts, got %d", got) + } + if elapsed := time.Since(start); elapsed < 900*time.Millisecond { + t.Fatalf("Retry-After: 1 not honored — retried after only %v", elapsed) + } +} diff --git a/clients/go/live_query_test.go b/clients/go/live_query_test.go new file mode 100644 index 00000000..f703a68e --- /dev/null +++ b/clients/go/live_query_test.go @@ -0,0 +1,173 @@ +package wavehouse + +import ( + "context" + "errors" + "sync/atomic" + "testing" + "time" +) + +// bareStream builds a StreamController that never dials anything — events are +// injected with emitEvent, exactly how the run loop feeds real ones. +func bareStream() *StreamController { + return &StreamController{ + status: StatusLive, + eventCh: make(chan StreamEvent, 16), + done: make(chan struct{}), + cancel: func() {}, + } +} + +func liveEvent(ts string) StreamEvent { + return StreamEvent{Table: "clicks", Timestamp: ts, Data: map[string]any{"page": "/home"}} +} + +func awaitInitial(t *testing.T, ch <-chan []map[string]any) []map[string]any { + t.Helper() + select { + case rows := <-ch: + return rows + case <-time.After(5 * time.Second): + t.Fatal("Initial never fired") + return nil + } +} + +func TestLiveQuery_InitialThenLiveWithDedup(t *testing.T) { + sc := bareStream() + fetched := []map[string]any{ + {"page": "/a", "received_timestamp": "2026-01-01T00:00:05Z"}, + // Descending order: the max timestamp is NOT the last row. + {"page": "/b", "received_timestamp": "2026-01-01T00:00:03Z"}, + } + gate := make(chan struct{}) + initialCh := make(chan []map[string]any, 1) + nextCh := make(chan StreamEvent, 8) + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { + <-gate + return fetched, nil + }, + &StreamSubscriber{ + Initial: func(rows []map[string]any, err error) { + if err != nil { + t.Errorf("Initial err: %v", err) + } + initialCh <- rows + }, + Next: func(e StreamEvent) { nextCh <- e }, + }) + defer lq.Close() + + // Buffered while the backfill is in flight; deduped against the *max* + // backfilled timestamp (5Z despite descending order) on flush. + sc.emitEvent(liveEvent("2026-01-01T00:00:04Z")) // ≤ max backfill → skipped + sc.emitEvent(liveEvent("2026-01-01T00:00:06Z")) // newer → delivered + close(gate) + + rows := awaitInitial(t, initialCh) + if len(rows) != 2 { + t.Fatalf("want 2 backfill rows, got %d", len(rows)) + } + + select { + case e := <-nextCh: + if e.Timestamp != "2026-01-01T00:00:06Z" { + t.Fatalf("want the newer event only, got %s", e.Timestamp) + } + case <-time.After(5 * time.Second): + t.Fatal("live event never delivered") + } + select { + case e := <-nextCh: + t.Fatalf("stale event delivered despite dedup: %s", e.Timestamp) + case <-time.After(100 * time.Millisecond): + } +} + +func TestLiveQuery_BuffersDuringBackfill(t *testing.T) { + sc := bareStream() + gate := make(chan struct{}) + initialCh := make(chan []map[string]any, 1) + nextCh := make(chan StreamEvent, 8) + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { + <-gate + return []map[string]any{{"received_timestamp": "2026-01-01T00:00:01Z"}}, nil + }, + &StreamSubscriber{ + Initial: func(rows []map[string]any, _ error) { initialCh <- rows }, + Next: func(e StreamEvent) { nextCh <- e }, + }) + defer lq.Close() + + // Events arriving mid-backfill are buffered, then flushed post-Initial. + sc.emitEvent(liveEvent("2026-01-01T00:00:02Z")) + sc.emitEvent(liveEvent("2026-01-01T00:00:00Z")) // older than backfill → dropped in flush + close(gate) + + awaitInitial(t, initialCh) + select { + case e := <-nextCh: + if e.Timestamp != "2026-01-01T00:00:02Z" { + t.Fatalf("want buffered 02Z event, got %s", e.Timestamp) + } + case <-time.After(5 * time.Second): + t.Fatal("buffered event never flushed") + } + select { + case e := <-nextCh: + t.Fatalf("pre-backfill event should have been deduped: %s", e.Timestamp) + case <-time.After(100 * time.Millisecond): + } +} + +func TestLiveQuery_FetchErrorReportedOnce(t *testing.T) { + sc := bareStream() + errCh := make(chan error, 1) + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { return nil, errors.New("boom") }, + &StreamSubscriber{ + Initial: func(_ []map[string]any, err error) { errCh <- err }, + }) + defer lq.Close() + + select { + case err := <-errCh: + if err == nil || err.Error() != "boom" { + t.Fatalf("want boom, got %v", err) + } + case <-time.After(5 * time.Second): + t.Fatal("Initial never fired on fetch error") + } +} + +func TestLiveQuery_NoCallbacksAfterClose(t *testing.T) { + sc := bareStream() + initialCh := make(chan []map[string]any, 1) + var delivered atomic.Int64 + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { return nil, nil }, + &StreamSubscriber{ + Initial: func(rows []map[string]any, _ error) { initialCh <- rows }, + Next: func(StreamEvent) { delivered.Add(1) }, + Status: func(StreamStatus) { delivered.Add(1) }, + Error: func(error) { delivered.Add(1) }, + }) + awaitInitial(t, initialCh) + + lq.Close() + before := delivered.Load() + sc.emitEvent(liveEvent("2026-01-01T00:00:09Z")) + sc.emitError(errors.New("late")) + sc.setStatus(StatusReconnecting) + time.Sleep(50 * time.Millisecond) + if got := delivered.Load(); got != before { + t.Fatalf("callbacks fired after Close: before=%d after=%d", before, got) + } +} diff --git a/clients/go/stream.go b/clients/go/stream.go index af14cd7f..4cb9582f 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -18,13 +18,15 @@ import ( // StreamController manages a live SSE event stream. Use Subscribe for // callback-based consumption or Events for channel-based consumption. type StreamController struct { - mu sync.Mutex - status StreamStatus - subscribers []*StreamSubscriber - eventCh chan StreamEvent // ponytail: single buffered channel for Go-native consumption - cancel context.CancelFunc - done chan struct{} - closed bool + mu sync.Mutex + status StreamStatus + subscribers []*StreamSubscriber + eventCh chan StreamEvent // ponytail: single buffered channel for Go-native consumption + chanRequested bool // set by Events(); until then emitEvent skips the channel + dropLogOnce sync.Once + cancel context.CancelFunc + done chan struct{} + closed bool } // newStreamController opens an SSE connection for the given table. @@ -75,8 +77,13 @@ func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { } // Events returns a read-only channel that receives stream events. -// The channel is closed when the stream closes. +// The channel is closed when the stream closes. Events are fed to the +// channel only from the first Events() call onward — a Subscribe-only +// consumer never fills (and overflows) a channel it isn't reading. func (sc *StreamController) Events() <-chan StreamEvent { + sc.mu.Lock() + sc.chanRequested = true + sc.mu.Unlock() return sc.eventCh } @@ -162,11 +169,20 @@ func (sc *StreamController) emitEvent(event StreamEvent) { } } - // Non-blocking send to the channel. + // Non-blocking send to the channel. Guarded by mu so the send and + // closeEventCh serialize — a late event can never hit a closed channel — + // and skipped entirely until Events() opts in. + sc.mu.Lock() + defer sc.mu.Unlock() + if sc.closed || !sc.chanRequested { + return + } select { case sc.eventCh <- event: default: - log.Printf("[wavehouse] stream event dropped: channel buffer full") + sc.dropLogOnce.Do(func() { + log.Printf("[wavehouse] stream event dropped: Events() channel buffer full (further drops not logged)") + }) } } @@ -182,11 +198,20 @@ func (sc *StreamController) emitError(err error) { } } +// closeEventCh marks the controller closed and closes the events channel. +// Must serialize with emitEvent's send via mu. +func (sc *StreamController) closeEventCh() { + sc.mu.Lock() + sc.closed = true + close(sc.eventCh) + sc.mu.Unlock() +} + // run is the SSE connection loop with reconnect/backoff. func (sc *StreamController) run(ctx context.Context, hctx httpContext, table string, opts *StreamOptions) { defer func() { sc.setStatus(StatusClosed) - close(sc.eventCh) + sc.closeEventCh() close(sc.done) }() @@ -344,7 +369,9 @@ type sseMessage struct { func (sc *StreamController) handleSSEData(data, eventID string) { var msg sseMessage if err := json.Unmarshal([]byte(data), &msg); err != nil { - log.Printf("[wavehouse] SSE received malformed message: %s", data) + // Deliberately omits the payload: event data can carry tenant/PII + // fields and this goes to the process-global logger. + log.Printf("[wavehouse] SSE received malformed message (%d bytes): %v", len(data), err) return } @@ -370,11 +397,14 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, go func() { defer func() { sc.setStatus(StatusClosed) - close(sc.eventCh) + // closeEventCh serializes with any in-flight emitEvent (which + // runs on the inner controller's goroutine), so the channel is + // never closed under a pending send. + sc.closeEventCh() close(sc.done) }() - inner.Subscribe(&StreamSubscriber{ + unsub := inner.Subscribe(&StreamSubscriber{ Next: func(event StreamEvent) { if !matchesFilters(event.Data, filters) { return @@ -394,6 +424,7 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, select { case <-ctx.Done(): + unsub() inner.Close() case <-inner.done: } diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go new file mode 100644 index 00000000..4da86d51 --- /dev/null +++ b/clients/go/stream_test.go @@ -0,0 +1,301 @@ +package wavehouse + +import ( + "context" + "fmt" + "io" + "net/http" + "net/http/httptest" + "sync/atomic" + "testing" + "time" +) + +// sseServer serves the given SSE frames on any request, then holds the +// connection open until the client disconnects. +func sseServer(t *testing.T, frames []string) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + fl, ok := w.(http.Flusher) + if !ok { + t.Error("response writer is not a flusher") + return + } + w.WriteHeader(200) + fl.Flush() + for _, f := range frames { + _, _ = io.WriteString(w, f) + fl.Flush() + } + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + return srv +} + +func sseFrame(ts, page string) string { + return "id: " + ts + "\n" + + `data: {"table_name":"clicks","received_timestamp":"` + ts + `","data":{"page":"` + page + `"}}` + + "\n\n" +} + +func streamClient(t *testing.T, srv *httptest.Server) *Client { + t.Helper() + return NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) +} + +func TestStream_SubscribeReceivesEvents(t *testing.T) { + srv := sseServer(t, []string{ + sseFrame("2026-01-01T00:00:01Z", "/home"), + sseFrame("2026-01-01T00:00:02Z", "/about"), + }) + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + got := make(chan StreamEvent, 8) + stream.Subscribe(&StreamSubscriber{ + Next: func(e StreamEvent) { got <- e }, + }) + + e1 := recvEvent(t, got) + if e1.Table != "clicks" || e1.Data["page"] != "/home" { + t.Fatalf("unexpected first event: %+v", e1) + } + e2 := recvEvent(t, got) + if e2.Data["page"] != "/about" || e2.Timestamp != "2026-01-01T00:00:02Z" { + t.Fatalf("unexpected second event: %+v", e2) + } +} + +func recvEvent(t *testing.T, ch <-chan StreamEvent) StreamEvent { + t.Helper() + select { + case e := <-ch: + return e + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for stream event") + return StreamEvent{} + } +} + +func TestStream_EventsChannel(t *testing.T) { + srv := sseServer(t, []string{sseFrame("2026-01-01T00:00:01Z", "/home")}) + stream := streamClient(t, srv).From("clicks").Stream(nil) + + ch := stream.Events() + e := recvEvent(t, ch) + if e.Data["page"] != "/home" { + t.Fatalf("unexpected event: %+v", e) + } + + stream.Close() + select { + case _, open := <-ch: + if open { + // A buffered event may arrive before close; drain once more. + if _, open2 := <-ch; open2 { + t.Fatal("events channel not closed after Close") + } + } + case <-time.After(5 * time.Second): + t.Fatal("events channel never closed after Close") + } +} + +func TestStream_Connected(t *testing.T) { + srv := sseServer(t, nil) + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := stream.Connected(ctx); err != nil { + t.Fatalf("Connected: %v", err) + } + if s := stream.Status(); s != StatusLive { + t.Fatalf("want live, got %s", s) + } +} + +func TestStream_FilteredDeliversMatchesAndProjects(t *testing.T) { + srv := sseServer(t, []string{ + sseFrame("2026-01-01T00:00:01Z", "/home"), + sseFrame("2026-01-01T00:00:02Z", "/miss"), + sseFrame("2026-01-01T00:00:03Z", "/home"), + }) + stream := streamClient(t, srv).From("clicks"). + Select("page"). + Where("page", OpEq, "/home"). + Stream(nil) + defer stream.Close() + + got := make(chan StreamEvent, 8) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { got <- e }}) + + for range 2 { + e := recvEvent(t, got) + if e.Data["page"] != "/home" { + t.Fatalf("filter leaked event: %+v", e) + } + if len(e.Data) != 1 { + t.Fatalf("projection kept extra columns: %+v", e.Data) + } + } + select { + case e := <-got: + t.Fatalf("unexpected third event: %+v", e) + case <-time.After(100 * time.Millisecond): + } +} + +// TestStream_FilteredCloseUnderLoad exercises the wrapper-close path while the +// inner stream is still delivering — the send-on-closed-channel regression. +func TestStream_FilteredCloseUnderLoad(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + fl := w.(http.Flusher) + w.WriteHeader(200) + fl.Flush() + for i := 0; ; i++ { + select { + case <-r.Context().Done(): + return + default: + } + _, err := io.WriteString(w, sseFrame(fmt.Sprintf("2026-01-01T00:00:%02dZ", i%60), "/home")) + if err != nil { + return + } + fl.Flush() + } + })) + t.Cleanup(srv.Close) + + stream := streamClient(t, srv).From("clicks"). + Select(). + Where("page", OpEq, "/home"). + Stream(nil) + + var n atomic.Int64 + stream.Subscribe(&StreamSubscriber{Next: func(StreamEvent) { n.Add(1) }}) + // Also exercise the Events() channel feed path during close. + _ = stream.Events() + + deadline := time.Now().Add(5 * time.Second) + for n.Load() < 10 && time.Now().Before(deadline) { + time.Sleep(5 * time.Millisecond) + } + if n.Load() == 0 { + t.Fatal("no events delivered before close") + } + stream.Close() // must not panic or race with in-flight emits + + select { + case <-stream.done: + case <-time.After(5 * time.Second): + t.Fatal("filtered stream goroutine never exited") + } +} + +func TestStream_HandleMalformedSSEData(t *testing.T) { + sc := &StreamController{eventCh: make(chan StreamEvent, 1)} + sc.handleSSEData("not json", "id1") // must not panic or emit + select { + case e := <-sc.eventCh: + t.Fatalf("malformed data emitted event: %+v", e) + default: + } +} + +// --------------------------------------------------------------------------- +// Client-side filter engine +// --------------------------------------------------------------------------- + +func TestEvaluateFilter(t *testing.T) { + tests := []struct { + name string + actual any + op string + expected any + want bool + }{ + {"EqNumericCrossType", float64(10), "eq", 10, true}, + {"EqString", "a", "eq", "a", true}, + {"EqMismatch", "a", "eq", "b", false}, + {"Neq", "a", "neq", "b", true}, + {"GtTrue", float64(11), "gt", 10, true}, + {"GtFalse", float64(10), "gt", 10, false}, + {"Gte", float64(10), "gte", 10, true}, + {"Lt", float64(9), "lt", 10, true}, + {"LteString", "a", "lte", "b", true}, + {"GtIncomparable", "a", "gt", 10, false}, + {"InAnySlice", "b", "in", []any{"a", "b"}, true}, + {"InTypedSlice", float64(2), "in", []int{1, 2}, true}, + {"InMiss", "c", "in", []any{"a", "b"}, false}, + {"InNotASlice", "a", "in", "a", false}, + {"Like", "hello world", "like", "hello%", true}, + {"LikeCaseInsensitive", "HELLO", "like", "hello", true}, + {"LikeUnderscore", "cat", "like", "c_t", true}, + {"LikeAnchored", "xhello", "like", "hello%", false}, + {"NotLike", "abc", "not_like", "x%", true}, + {"LikeNonString", 5, "like", "5", false}, + {"UnknownOp", "a", "regex", "a", false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + if got := evaluateFilter(tt.actual, tt.op, tt.expected); got != tt.want { + t.Errorf("evaluateFilter(%v, %q, %v) = %v, want %v", tt.actual, tt.op, tt.expected, got, tt.want) + } + }) + } +} + +func TestMatchesFilters_AllMustMatch(t *testing.T) { + row := map[string]any{"page": "/home", "score": float64(10)} + both := []QueryFilter{ + {Column: "page", Op: "eq", Value: "/home"}, + {Column: "score", Op: "gt", Value: 5}, + } + if !matchesFilters(row, both) { + t.Fatal("want match when every filter passes") + } + oneFails := append(append([]QueryFilter(nil), both...), QueryFilter{Column: "score", Op: "gt", Value: 99}) + if matchesFilters(row, oneFails) { + t.Fatal("want no match when any filter fails") + } + if !matchesFilters(row, nil) { + t.Fatal("want match with no filters") + } +} + +func TestCompareOrdered(t *testing.T) { + if c, ok := compareOrdered(float64(1), 2); !ok || c != -1 { + t.Fatalf("numeric compare: got (%d, %v)", c, ok) + } + if c, ok := compareOrdered("b", "a"); !ok || c != 1 { + t.Fatalf("string compare: got (%d, %v)", c, ok) + } + if _, ok := compareOrdered(map[string]any{}, 1); ok { + t.Fatal("incomparable types must return ok=false") + } +} + +func TestToFloat64(t *testing.T) { + for _, v := range []any{float64(1), float32(1), int(1), int64(1)} { + if f, ok := toFloat64(v); !ok || f != 1 { + t.Fatalf("toFloat64(%T) = (%v, %v)", v, f, ok) + } + } + if _, ok := toFloat64("1"); ok { + t.Fatal("strings must not convert") + } +} + +func TestProjectColumns(t *testing.T) { + row := map[string]any{"a": 1, "b": 2, "c": 3} + got := projectColumns(row, []string{"a", "c", "missing"}) + if len(got) != 2 || got["a"] != 1 || got["c"] != 3 { + t.Fatalf("unexpected projection: %+v", got) + } +} diff --git a/clients/go/table_test.go b/clients/go/table_test.go index 707e13ed..5116d53f 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -17,6 +17,7 @@ func TestTableRef_InsertSingle(t *testing.T) { _ = json.NewDecoder(r.Body).Decode(&gotBody) _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/home"}) if err != nil { @@ -44,6 +45,7 @@ func TestTableRef_InsertBatch(t *testing.T) { "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []map[string]any{ {"page": "/a"}, @@ -83,6 +85,7 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { "total": 2, "succeeded": 1, "failed": 1, "duplicates": 0, }) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []ClickRow{ {Page: "/a"}, @@ -116,6 +119,7 @@ func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { gotPath = r.URL.Path _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) if err != nil { @@ -133,6 +137,7 @@ func TestTableRef_InsertEmptyBatch(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { t.Fatal("should not make a request for empty batch") })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []map[string]any{}) if err != nil { @@ -155,6 +160,7 @@ func TestTableRef_InsertNDJSON(t *testing.T) { "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) ndjson := `{"page":"/a"}` + "\n" + `{"page":"/b"}` result, err := c.From("clicks").InsertNDJSON(context.Background(), ndjson) @@ -181,6 +187,7 @@ func TestTableRef_Schema(t *testing.T) { }, }) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) schema, err := c.From("clicks").Schema(context.Background()) if err != nil { @@ -198,6 +205,7 @@ func TestTableRef_InsertDuplicate(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) })) + t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/dup"}) if err != nil { diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 476ae773..14128047 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -188,7 +188,7 @@ They block the terminal and stream logs; simply press `Ctrl+C` to instantly tear ### Using the SDK against `make dev` -There's no bundled playground — point the published `@wavehouse/sdk` client at your local server (`baseURL: "http://localhost:8080"`), with the dev policy seeded so requests are authorized: +There's no bundled playground — point the published `@wavehouse/sdk` client (or the Go SDK, `github.com/Wave-RF/WaveHouse/clients/go`) at your local server (`baseURL: "http://localhost:8080"`), with the dev policy seeded so requests are authorized: ```bash WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml make dev @@ -196,7 +196,7 @@ WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml make dev See the [SDK guide](/sdk) for the client API and examples. -Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. +Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. Go services do the same with `wavehouse.NewClient(wavehouse.Config{BaseURL: "http://localhost:8080"})` — see the [Go SDK docs](/sdk/go). ### Validating tokens @@ -337,7 +337,10 @@ Each test target writes `covdata` to `tmp/coverage//data/`, renders a tex | Category | Location | Docker? | Command | | -------- | -------- | ------- | ------- | | Unit tests | `internal/*/_test.go` | No | `make test` | -| SDK unit tests | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | +| SDK unit tests (TS) | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | +| SDK unit tests (Go) | `clients/go/*_test.go` (nested Go module) | No | `make test-go-sdk` (runs with `-race`) | +| Wire-format conformance | `clients/go/conformance_test.go` + `tests/conformance/conformance_ts.mjs`, both replaying `clients/go/testdata/wire_cases.json` | No | Go half via `make test-go-sdk`; TS half via `make test-conformance-ts` | +| SDK E2E (Go, live server) | `clients/go/e2e_test.go` (`//go:build e2e`) | No | `make test-go-sdk-e2e` (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | | Integration tests (Go) | `tests/integration/*_test.go` | Yes | `make test-integration` | | E2E tests (SDK) | `tests/e2e/sdk/*.test.ts` | Yes | `make test-e2e` | @@ -429,8 +432,13 @@ WaveHouse/ │ ├── policy/ # Access control policies (evaluation + NATS KV store) │ ├── query/ # Structured query AST + SQL builder │ └── testutil/ # Shared test helpers and mocks +├── clients/ # Official SDKs +│ ├── ts/ # TypeScript SDK (@wavehouse/sdk) +│ └── go/ # Go SDK — a NESTED Go module (own go.mod, invisible +│ # to root `go list`; hence the *-go-sdk make targets) ├── tests/ # Integration & E2E tests │ ├── integration/ # Go integration tests (//go:build integration) +│ ├── conformance/ # TS half of the cross-SDK wire-format conformance suite │ └── e2e/ # E2E suite (orchestrator + ClickHouse testcontainer) │ ├── fixtures/ # ClickHouse DDL + config/policy fixtures │ └── sdk/ # E2E specs driven through the TypeScript SDK (Vitest) @@ -476,7 +484,7 @@ Run `make help` to see all targets. Key ones: | **Static checks** | | | `make fmt` | Check formatting across Go (`gofumpt`) + TS (Biome). Run `make fix` to apply. | | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | -| `make lint` | Run linters across Go (`golangci-lint`) + TS (Biome) | +| `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) | | `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | | `make verify` | Repo-wide static checks: Go (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | | `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`) and TS (Biome `--write`) | @@ -486,8 +494,11 @@ Run `make help` to see all targets. Key ones: | `make build-cover` | Coverage-instrumented build → `bin/wavehouse-cov` (used by E2E) | | `make build-ts` | Build TypeScript SDK → `clients/ts/dist/` | | **Test** | | -| `make test` | Alias for `test-unit` | +| `make test` | Alias for `test-unit` + `test-go-sdk` | | `make test-unit` | Go unit tests + render coverage + gate suite threshold | +| `make test-go-sdk` | Go SDK (`clients/go`, nested module) unit tests with `-race` | +| `make test-go-sdk-e2e` | Go SDK E2E against a live server (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | +| `make test-conformance-ts` | TS SDK wire-format conformance against the shared `wire_cases.json` fixture (builds the TS SDK first) | | `make test-integration` | Go integration tests (requires Docker) + coverage gate | | `make test-ts` | SDK vitest unit tests + v8 coverage + gate against `suites.ts-unit` (matches Go's "always coverage" pattern) | | `make cov` | Merge Go + TS coverage and gate against thresholds. Auto-runs after `make test-all` and `make ci`; standalone `make cov` is "show me the merged numbers without re-running." Each side skips silently if its data is missing, but `make cov` fails if *both* are empty (you ran it before any test target). | @@ -581,7 +592,7 @@ If the title doesn't match, a sticky comment posts on the PR explaining the form The `main branch protection` ruleset requires one status check to pass before any PR can merge: -- `CI` — the aggregator job of `.github/workflows/ci.yml`. The workflow is a job DAG over the same Makefile targets local `make ci` runs: `lint` (`make verify`), `unit` (`make test-unit test-ts`), `integration` (`make test-integration`), `e2e` (`make -j test-e2e` — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), `coverage` (`make cov` over every suite's uploaded coverage fragment + threshold gates, like local `make ci`'s final step), `docs-build` (`make build-docs` when docs-affecting files changed, uploading the docs dist artifact), `PR title` (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run's Summary page gets a per-job wall-clock table from the non-gating `Timing summary` job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md). +- `CI` — the aggregator job of `.github/workflows/ci.yml`. The workflow is a job DAG over the same Makefile targets local `make ci` runs: `lint` (`make verify`), `unit` (`make test-unit test-ts test-go-sdk test-conformance-ts`), `integration` (`make test-integration`), `e2e` (`make -j test-e2e` — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), `coverage` (`make cov` over every suite's uploaded coverage fragment + threshold gates, like local `make ci`'s final step), `docs-build` (`make build-docs` when docs-affecting files changed, uploading the docs dist artifact), `PR title` (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run's Summary page gets a per-job wall-clock table from the non-gating `Timing summary` job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md). The `PR housekeeping` workflow still runs on every PR (labels + the title explainer comment) but is no longer a required check. diff --git a/docs/src/content/docs/getting-started.md b/docs/src/content/docs/getting-started.md index 5ae0e978..d8cf0800 100644 --- a/docs/src/content/docs/getting-started.md +++ b/docs/src/content/docs/getting-started.md @@ -76,7 +76,7 @@ curl -s -X POST "http://localhost:8080/v1/query?table=clicks" \ `POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached in-process (L1 Ristretto) with singleflight coalescing — duplicate concurrent queries hit ClickHouse once. For raw SQL there's `POST /v1/admin/query` (an admin escape hatch that never caches, emitting `Cache-Control: no-store`), but it's **admin-only** — the trial `public` role can't reach it. To use it, swap the public default for real auth: configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). :::tip[Prefer a type-safe client?] -The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). +The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming; the [Go SDK](/sdk/go) offers the same builder with generics for typed rows. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). ::: ## 5. Subscribe to real-time updates @@ -106,6 +106,7 @@ The handful of things that most often trip up a first session — each is expect - **[Architecture](/architecture)** — how ingest, query, cache, and streaming fit together. - **[API Reference](/api)** — every endpoint, request/response shape, and error code. - **[TypeScript SDK](/sdk)** — zero-dependency client with query builder, live queries, and codegen. +- **[Go SDK](/sdk/go)** — the same surface for Go: context-first, generics for typed rows, codegen CLI. - **[Configuration](/configuration)** — full YAML + environment variable reference. - **[Deployment](/deployment)** — Docker images, releases, health checks. - **[Development](/development)** — building from source, running tests, hot-reload workflow. diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index b10e97b1..1492292e 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -95,8 +95,8 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click Per-table, per-role column and row-level policies with JWT claim templating. Stored in NATS KV with file-based bootstrap and cluster sync. - - `@wavehouse/sdk` — zero-dependency client with type-safe query builder, live queries, real-time streaming, and codegen from your schemas. + + `@wavehouse/sdk` and `clients/go` — zero-dependency clients with type-safe query builders, live queries, real-time streaming, and codegen from your schemas, speaking one shared wire format. @@ -104,7 +104,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click ## Query it like a database. Subscribe to it like a socket. -The zero-dependency [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming: +The zero-dependency [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming. Writing Go? The official [Go SDK](/sdk/go) mirrors the same feature set: @@ -192,6 +192,11 @@ WaveHouse is fail-closed, so the standalone stack ships a permissive trial polic description="Query builder, live queries, streaming, and schema codegen." href="/sdk" /> +
diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 2202f95d..280fab00 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -173,8 +173,9 @@ See [Reference → Error Handling](/sdk/go/reference#error-handling) for retry b ## Differences from the TypeScript SDK The two SDKs share a wire format and mirror each other's feature set closely -(a shared `testdata/wire_cases.json` conformance fixture in the repo asserts -both produce identical HTTP requests for equivalent builder calls), but the +(a shared `wire_cases.json` conformance fixture is replayed by a test runner +per SDK — both run in CI — asserting each produces the expected HTTP request +for equivalent builder calls), but the languages pull the API shape in different directions: - **No `Result` union.** Go returns `(T, error)`; nothing is wrapped in diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 75cf8cad..54e2d2be 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -49,7 +49,7 @@ guarantee. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Missing or invalid JWT | +| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 500 | `HTTP_500` | Yes | Server error (retried per `ClientOptions.MaxRetries`) | @@ -179,14 +179,21 @@ package myapp // ClicksRow represents a row in the "clicks" table. type ClicksRow struct { - EventID string `json:"event_id"` - Page string `json:"page"` - UserID string `json:"user_id"` - DurationMS int `json:"duration_ms"` - ReceivedTimestamp string `json:"received_timestamp"` + Page string `json:"page"` + Button string `json:"button"` + Score float64 `json:"score"` + ReceivedTimestamp string `json:"received_timestamp,omitempty"` } ``` +(That's the exact output for the `clicks` table from the +[development quick-start](/development#quick-start) — `received_timestamp` +gets `,omitempty` because it has a `DEFAULT` clause.) + +Note the generator does **not** special-case initialisms: `event_id` becomes +`EventId`, not the Go-idiomatic `EventID` — each `_`-separated part simply +gets its first letter upper-cased. + Table and column names are converted to `PascalCase` for Go field/type names (a leading digit gets an `X` prefix — e.g. a table named `2fa_events` becomes `X2faEventsRow` — to stay a valid Go identifier). A column with @@ -197,17 +204,18 @@ tag. | ClickHouse Type | Go Type | |------------------|---------| -| `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | -| `Bool` | `bool` | +| `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Time`/`Time64`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | +| `Bool` / `Boolean` | `bool` | | `UInt8` / `UInt16` / `UInt32` | `uint8` / `uint16` / `uint32` | | `Int8` / `Int16` / `Int32` | `int8` / `int16` / `int32` | -| `Float32` | `float32` | +| `Float32`, `BFloat16` | `float32` | | `Float64` | `float64` | | `UInt64`/`Int64`, `Decimal*`, `UInt128`/`UInt256`, `Int128`/`Int256` | `string` (ClickHouse quotes 64-bit-and-wider integers in JSON output — `output_format_json_quote_64bit_integers` — and the server forwards them verbatim) | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | | `Array(T)` | `[]T` | | `Map(K, V)` | `map[K]V` (falls back to `map[string]any` if `K`/`V` can't be split) | +| `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | This differs from the TypeScript SDK's mapping in one notable way: Go's @@ -220,11 +228,13 @@ that is what actually arrives on the wire. The Go SDK ships with unit tests colocated in `clients/go/` (its own Go module — `clients/go/go.mod` — separate from the root `WaveHouse` module), -plus a wire-format **conformance suite** -(`clients/go/conformance_test.go` + `clients/go/testdata/wire_cases.json`) -that replays a shared fixture of builder calls and asserts the Go SDK -produces the exact same HTTP method, path, content type, and body as the -TypeScript SDK for each one — keeping the two clients honest about the wire +plus the Go half of the cross-language wire-format **conformance suite**: +`clients/go/conformance_test.go` replays the shared fixture +(`clients/go/testdata/wire_cases.json`) and asserts the Go SDK produces the +expected HTTP method, path, content type, and body for each case. The +TypeScript half — `tests/conformance/conformance_ts.mjs`, run with +`make test-conformance-ts` (it builds the TS SDK first) — replays the same +fixture, and CI runs both, keeping the two clients honest about the wire format they both speak. ```bash diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index f631d1a7..2c9bd2d5 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -30,7 +30,7 @@ The SDK **never throws**. All errors are returned in `Result.error`. | Status | Code | Retryable | Description | |--------|------|-----------|-------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Missing or invalid JWT | +| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 500 | `HTTP_500` | Yes | Server error (retried per `maxRetries`) | diff --git a/docs/src/content/docs/why-wavehouse.md b/docs/src/content/docs/why-wavehouse.md index 9cb5b63d..5cce04ba 100644 --- a/docs/src/content/docs/why-wavehouse.md +++ b/docs/src/content/docs/why-wavehouse.md @@ -154,7 +154,7 @@ flowchart TB | Schema validation | Custom code in ingest API | Built in (discovers `system.columns`) | | Row/column access control | Custom middleware or a dedicated service | Built in (Hasura-style, JWT-driven) | | Dead letter queue | Custom retry + dead topic on Kafka | Built in (`WAVEHOUSE_DLQ`) | -| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript, zero-dep, codegen) | +| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript) + `clients/go` (Go) — zero-dep, codegen | The DIY path works — big teams run it — but the ops cost is not small. You're paying for a Kafka cluster (or Confluent bill), a second service you wrote from scratch, and all the debugging hours when the batching consumer stalls at 3 a.m. @@ -194,7 +194,7 @@ Tinybird wins on "zero ops to start." WaveHouse wins on "own your data plane and | Thundering-herd coalescing | ✗ | Custom | ✓ | ✓ Ristretto + singleflight | | Row/column policies with JWT claims | ✗ | Custom | Tokens only | ✓ Hasura-style | | Named parameterized pipes | ✗ | Custom | ✓ | ✓ stored in NATS KV | -| Type-safe client SDK with codegen | ✗ | Per team | Partial | ✓ `@wavehouse/sdk` | +| Type-safe client SDK with codegen | ✗ | Per team | Partial | ✓ TypeScript + Go SDKs | | Cost model | Infra only | Infra + eng time | Per-vCPU SaaS | Infra only | ## Part IV — End-to-end data journey From e783f5b8af1fd6773886591f578915288eca6047 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 13:37:44 -0400 Subject: [PATCH 09/40] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=202=20=E2=80=94=20terminal=20SSE=20errors,=20codegen?= =?UTF-8?q?=20pointers,=20precision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SDK behavior: - Non-retryable SSE connect errors (401/403/404) are now terminal: connect surfaces the parsed API error, run emits it and closes the stream instead of reconnecting forever; Connected() unblocks with "stream closed" - codegen: defaulted columns generate pointer fields (*T + omitempty) — the Go spelling of the TS codegen's `field?: T` — so an explicit zero value is sent instead of silently dropped in favor of the server default - Typed-row pagination cursor decodes with json.Number, keeping int64 cursor values past 2^53 exact Tests: - Pagination: page.Next walked across three pages asserting exactly one replaced cursor filter with the right op/value, desc → lt, quiet end when the projection omits the order column, int64 precision regression test - Terminal 403 stream test: error surfaced, StatusClosed, Connected fails - e2e: buildMarkerRow returns the column it used (markerColumn could pick a defaulted column the row never wrote); http_test raw-body via io.ReadAll - conformance_ts exits non-zero when nothing ran or any case was skipped Docs/Makefile: - Error-model claim scoped: HTTP-exchange errors are *wavehouse.Error; pre-request failures (auth provider, marshal) are plain wrapped errors — examples gain the else branch (go/index, go/reference, README) - codegen README example uses WAVEHOUSE_AUTH; reference.md documents the pointer rule and sample output - queries.md: real aggregation signatures/alias defaults, pagination example gains the OrderBy it needs - development.md: coverage sentence scoped to instrumented suites, "four suites" count dropped, Releasing the SDKs covers the Go module - access-control.mdx + pipes.mdx list the Go SDK method equivalents - make ci runs test-conformance-ts (parity with the CI unit job) --- Makefile | 2 +- clients/go/README.md | 4 +- clients/go/cmd/wavehouse-codegen/main.go | 7 + clients/go/e2e_test.go | 32 ++--- clients/go/http_test.go | 6 +- clients/go/query_builder.go | 7 +- clients/go/query_builder_test.go | 159 ++++++++++++++++++++++ clients/go/stream.go | 12 +- clients/go/stream_test.go | 41 ++++++ docs/src/content/docs/access-control.mdx | 2 +- docs/src/content/docs/development.md | 12 +- docs/src/content/docs/pipes.mdx | 2 +- docs/src/content/docs/sdk/go/index.md | 4 +- docs/src/content/docs/sdk/go/queries.md | 14 +- docs/src/content/docs/sdk/go/reference.md | 28 ++-- docs/src/content/docs/sdk/index.mdx | 2 +- tests/conformance/conformance_ts.mjs | 4 +- 17 files changed, 282 insertions(+), 56 deletions(-) diff --git a/Makefile b/Makefile index 6e3d1c28..243ddcfa 100644 --- a/Makefile +++ b/Makefile @@ -786,7 +786,7 @@ cov: ## Consolidated coverage report (Go + TS) + gate against thresholds (auto-r # marker that standalone `make verify` writes is instead written by ci's own # `ci-marker.sh write` below — it touches both the ci and verify markers. .PHONY: ci-parallel -ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts test-go-sdk +ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts test-go-sdk test-conformance-ts .PHONY: ci ci: ## Full pipeline — parallel checks, then sequential heavy suites + coverage diff --git a/clients/go/README.md b/clients/go/README.md index c71e5644..bd991359 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -193,9 +193,9 @@ rows, _ := wavehouse.SQL[map[string]any](ctx, client, "SELECT count() FROM click Generate Go structs from a running WaveHouse instance: ```bash +export WAVEHOUSE_AUTH= # avoids leaking the token via argv go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ --url http://localhost:8080 \ - --auth \ --out ./db_types.go \ --package myapp ``` @@ -204,7 +204,7 @@ See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference/# ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors are `*wavehouse.Error` (use `errors.As`). Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`) deliver errors through callbacks instead: +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors originating from the HTTP exchange are `*wavehouse.Error` — unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`, and `Connected`) deliver errors through callbacks or plain errors instead: ```go page, err := client.From("clicks").Fetch(ctx) diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 9775b525..793e1938 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -307,7 +307,14 @@ func generate(schemas map[string]tableSchema, pkg string) (string, error) { seenFields[fieldName] = col.Name jsonTag := col.Name if col.HasDefault { + // Pointer + omitempty is the Go spelling of the TS codegen's + // `field?: T`: nil omits the field (server default applies), + // while a pointer to the zero value still sends an explicit + // 0/false/"" instead of silently dropping it. jsonTag += ",omitempty" + if !strings.HasPrefix(goType, "*") { + goType = "*" + goType + } } fmt.Fprintf(&sb, "\t%s %s `json:%q`\n", fieldName, goType, jsonTag) } diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go index 69b724e3..23f59221 100644 --- a/clients/go/e2e_test.go +++ b/clients/go/e2e_test.go @@ -142,8 +142,7 @@ func TestE2E_InsertAndQuery(t *testing.T) { table, ts := firstTable(t, c) mk := marker(t) - row := buildMarkerRow(t, ts, mk) - markerCol := markerColumn(t, ts) + row, markerCol := buildMarkerRow(t, ts, mk) res, err := c.From(table).Insert(ctx, row) if err != nil { @@ -169,12 +168,12 @@ func TestE2E_BatchInsert(t *testing.T) { table, ts := firstTable(t, c) mk := marker(t) - markerCol := markerColumn(t, ts) // Build 3 rows, each with the same marker so we can count them. rows := make([]map[string]any, 3) + markerCol := "" for i := range rows { - rows[i] = buildMarkerRow(t, ts, mk) + rows[i], markerCol = buildMarkerRow(t, ts, mk) } res, err := c.From(table).Insert(ctx, rows) @@ -358,26 +357,14 @@ func TestE2E_PipesCRUD(t *testing.T) { // Helpers // --------------------------------------------------------------------------- -// markerColumn finds the first String/LowCardinality(String) column in the -// schema that we can use to inject a test marker value. -func markerColumn(t *testing.T, ts TableSchema) string { - t.Helper() - for _, col := range ts.Columns { - ct := strings.ToLower(col.Type) - if ct == "string" || strings.Contains(ct, "string") { - return col.Name - } - } - t.Skipf("e2e: table %q has no string column for marker injection", ts.Name) - return "" -} - // buildMarkerRow constructs a minimal valid row for the table, injecting the -// marker into the first string column and using sensible defaults for other -// required columns. -func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { +// marker into the first non-default string column and using sensible values +// for other required columns. It returns the row and the marker column, so +// callers query back the exact column the marker went into. +func buildMarkerRow(t *testing.T, ts TableSchema, mk string) (map[string]any, string) { t.Helper() row := make(map[string]any) + markerCol := "" markerSet := false for _, col := range ts.Columns { if col.HasDefault { @@ -387,6 +374,7 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { switch { case !markerSet && strings.Contains(ct, "string"): row[col.Name] = mk + markerCol = col.Name markerSet = true case strings.Contains(ct, "string"): row[col.Name] = "e2e" @@ -408,7 +396,7 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) map[string]any { if !markerSet { t.Skipf("e2e: table %q has no non-default string column for marker", ts.Name) } - return row + return row, markerCol } // skipIfUnauthorized skips the test when err indicates a 401 or 403, diff --git a/clients/go/http_test.go b/clients/go/http_test.go index db1693b3..aef9aa0f 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -3,6 +3,7 @@ package wavehouse import ( "context" "encoding/json" + "io" "net/http" "net/http/httptest" "sync/atomic" @@ -70,9 +71,8 @@ func TestDoRequest_RawBody(t *testing.T) { var gotCT string hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { gotCT = r.Header.Get("Content-Type") - raw := make([]byte, 1024) - n, _ := r.Body.Read(raw) - gotBody = string(raw[:n]) + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) _ = json.NewEncoder(w).Encode(map[string]int{"total": 1}) })) diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 2855d367..c1c540e6 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -1,6 +1,7 @@ package wavehouse import ( + "bytes" "context" "encoding/json" "net/url" @@ -275,9 +276,13 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro m, ok := lastRow.(map[string]any) if !ok { // ponytail: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. + // UseNumber keeps int64 cursor values exact; plain float64 decoding + // corrupts IDs past 2^53 and pagination would repeat or skip a row. raw, _ := json.Marshal(lastRow) m = make(map[string]any) - _ = json.Unmarshal(raw, &m) + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.UseNumber() + _ = dec.Decode(&m) } lastValue, exists := m[cursor.Column] if !exists { diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index 71d2808e..4c4b7656 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -1,8 +1,10 @@ package wavehouse import ( + "bytes" "context" "encoding/json" + "fmt" "io" "net/http" "net/http/httptest" @@ -269,3 +271,160 @@ func TestQueryBuilder_ComplexQuery(t *testing.T) { t.Fatal("wrong group_by") } } + +// pagingServer returns limit-sized pages of rows and captures each request +// body, so tests can walk page.Next and inspect the cursor filters sent. +func pagingServer(t *testing.T, pages [][]map[string]any) (*Client, func() []map[string]any) { + t.Helper() + var mu sync.Mutex + var bodies []map[string]any + call := 0 + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, _ := io.ReadAll(r.Body) + var body map[string]any + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.UseNumber() // keep int64 cursor values exact on the capture side too + _ = dec.Decode(&body) + mu.Lock() + bodies = append(bodies, body) + idx := call + call++ + mu.Unlock() + page := []map[string]any{} + if idx < len(pages) { + page = pages[idx] + } + _ = json.NewEncoder(w).Encode(page) + })) + return c, func() []map[string]any { + mu.Lock() + defer mu.Unlock() + return append([]map[string]any(nil), bodies...) + } +} + +func filtersOf(t *testing.T, body map[string]any) []map[string]any { + t.Helper() + raw, ok := body["filters"].([]any) + if !ok { + return nil + } + out := make([]map[string]any, len(raw)) + for i, f := range raw { + out[i] = f.(map[string]any) + } + return out +} + +func TestQueryBuilder_Pagination_NextWalksPages(t *testing.T) { + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": "a"}, {"id": "b"}}, + {{"id": "c"}, {"id": "d"}}, + {{"id": "e"}}, + }) + + page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + page2, err := page.Next(context.Background()) + if err != nil { + t.Fatal(err) + } + if page2.Data[0]["id"] != "c" || !page2.HasMore || page2.Next == nil { + t.Fatalf("unexpected page 2: %+v", page2) + } + page3, err := page2.Next(context.Background()) + if err != nil { + t.Fatal(err) + } + if len(page3.Data) != 1 || page3.HasMore { + t.Fatalf("unexpected page 3: %+v", page3) + } + + bodies := getBodies() + if len(bodies) != 3 { + t.Fatalf("want 3 requests, got %d", len(bodies)) + } + if f := filtersOf(t, bodies[0]); len(f) != 0 { + t.Fatalf("page 1 must have no cursor filter, got %v", f) + } + // Page 2 and 3: exactly ONE cursor filter (replaced, not stacked), with + // the ascending op and the previous page's last cursor value. + for i, want := range []string{"b", "d"} { + f := filtersOf(t, bodies[i+1]) + if len(f) != 1 { + t.Fatalf("page %d: want exactly 1 cursor filter, got %v", i+2, f) + } + if f[0]["column"] != "id" || f[0]["op"] != "gt" || f[0]["value"] != want { + t.Fatalf("page %d: unexpected cursor filter %v", i+2, f[0]) + } + } +} + +func TestQueryBuilder_Pagination_DescUsesLt(t *testing.T) { + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": "z"}, {"id": "y"}}, + {{"id": "x"}}, + }) + + page, err := c.From("clicks").Select("id").OrderBy("id", "desc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if _, err := page.Next(context.Background()); err != nil { + t.Fatal(err) + } + + f := filtersOf(t, getBodies()[1]) + if len(f) != 1 || f[0]["op"] != "lt" || f[0]["value"] != "y" { + t.Fatalf("desc cursor filter wrong: %v", f) + } +} + +func TestQueryBuilder_Pagination_CursorColumnMissingEndsQuietly(t *testing.T) { + c, _ := pagingServer(t, [][]map[string]any{ + {{"other": "1"}, {"other": "2"}}, // projection omits the order column + }) + + page, err := c.From("clicks").Select("other").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + next, err := page.Next(context.Background()) + if err != nil { + t.Fatal(err) + } + if len(next.Data) != 0 || next.HasMore || next.Next != nil { + t.Fatalf("want quiet empty page, got %+v", next) + } +} + +func TestQueryBuilder_Pagination_TypedInt64CursorKeepsPrecision(t *testing.T) { + type idRow struct { + ID int64 `json:"id"` + } + const bigID = int64(9007199254740993) // 2^53 + 1: float64 round-trip corrupts it + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": 1}, {"id": bigID}}, + {}, + }) + + q := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2) + page, err := FetchTyped[idRow](context.Background(), q) + if err != nil { + t.Fatal(err) + } + if _, err := page.Next(context.Background()); err != nil { + t.Fatal(err) + } + + f := filtersOf(t, getBodies()[1]) + if len(f) != 1 { + t.Fatalf("want 1 cursor filter, got %v", f) + } + // json.Number survives the round-trip; float64 would have sent ...992. + if got := fmt.Sprint(f[0]["value"]); got != "9007199254740993" { + t.Fatalf("cursor value lost precision: %s", got) + } +} diff --git a/clients/go/stream.go b/clients/go/stream.go index 4cb9582f..418b11ec 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -4,6 +4,7 @@ import ( "bufio" "context" "encoding/json" + "errors" "fmt" "log" "net/http" @@ -242,6 +243,15 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str } if err != nil { + // A non-retryable API error (401/403/404, ...) is terminal: + // reconnecting can't fix a bad token or a missing table, and the + // TS SDK's EventSource likewise ends up closed on a non-200. + // Emit it and exit — the deferred cleanup sets StatusClosed. + var apiErr *Error + if errors.As(err, &apiErr) && !apiErr.Retryable { + sc.emitError(apiErr) + return + } sc.emitError(&Error{ Status: 0, Code: "SSE_ERROR", @@ -307,7 +317,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { - return "", false, fmt.Errorf("SSE connect failed: HTTP %d", resp.StatusCode) + return "", false, parseErrorResponse(resp) } sc.setStatus(StatusLive) diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 4da86d51..18216aba 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -2,6 +2,7 @@ package wavehouse import ( "context" + "errors" "fmt" "io" "net/http" @@ -299,3 +300,43 @@ func TestProjectColumns(t *testing.T) { t.Fatalf("unexpected projection: %+v", got) } } + +// TestStream_NonRetryableConnectErrorIsTerminal: a 403 must close the stream +// (no infinite reconnect) and surface the API error to Error subscribers. +func TestStream_NonRetryableConnectErrorIsTerminal(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, `{"error":"forbidden"}`, http.StatusForbidden) + })) + t.Cleanup(srv.Close) + + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + errCh := make(chan error, 4) + stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) || apiErr.Status != http.StatusForbidden || apiErr.Retryable { + t.Fatalf("want non-retryable HTTP_403, got %v", err) + } + case <-time.After(5 * time.Second): + t.Fatal("error never surfaced") + } + + select { + case <-stream.done: + case <-time.After(5 * time.Second): + t.Fatal("stream never closed after non-retryable connect error") + } + if s := stream.Status(); s != StatusClosed { + t.Fatalf("want closed, got %s", s) + } + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := stream.Connected(ctx); err == nil { + t.Fatal("Connected must fail on a terminally-closed stream") + } +} diff --git a/docs/src/content/docs/access-control.mdx b/docs/src/content/docs/access-control.mdx index dda1e0b5..69c21cb7 100644 --- a/docs/src/content/docs/access-control.mdx +++ b/docs/src/content/docs/access-control.mdx @@ -399,7 +399,7 @@ curl -X PUT http://localhost:8080/v1/admin/policy \ -d @policy.json ``` -The full request and response shapes for these endpoints live in the [API Reference](/api); the [TypeScript SDK](/sdk) wraps them as `client.policy.get()`, `client.policy.set(policy)`, and `client.policy.validate(policy)`. +The full request and response shapes for these endpoints live in the [API Reference](/api); the [TypeScript SDK](/sdk) wraps them as `client.policy.get()`, `client.policy.set(policy)`, and `client.policy.validate(policy)`, and the [Go SDK](/sdk/go/admin) as `wh.Policy.Get(ctx)`, `wh.Policy.Set(ctx, policy)`, and `wh.Policy.Validate(ctx, policy)`. ## Bootstrapping and the policy lifecycle diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 14128047..a7bbf169 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -313,7 +313,7 @@ make test-ts # E2E SDK suite against bin/wavehouse-cov make test-e2e -# All four suites sequentially + merged coverage +# All suites sequentially + merged coverage make test-all # Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts, @@ -324,7 +324,7 @@ make ci make cov ``` -Each test target writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. +Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. **Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). @@ -503,7 +503,7 @@ Run `make help` to see all targets. Key ones: | `make test-ts` | SDK vitest unit tests + v8 coverage + gate against `suites.ts-unit` (matches Go's "always coverage" pattern) | | `make cov` | Merge Go + TS coverage and gate against thresholds. Auto-runs after `make test-all` and `make ci`; standalone `make cov` is "show me the merged numbers without re-running." Each side skips silently if its data is missing, but `make cov` fails if *both* are empty (you ran it before any test target). | | `make test-e2e` | E2E SDK suite against `bin/wavehouse-cov` + coverage gate | -| `make test-all` | All four suites sequentially + merged coverage gate | +| `make test-all` | All suites sequentially + merged coverage gate | | `make ci` | Full pipeline: parallel `verify` + builds + unit/SDK tests, then integration + E2E + cov | | **Analysis** (informational, not in CI) | | | `make size` | Binary size analysis → `tmp/analysis/` (text + SVG + interactive HTML) | @@ -550,9 +550,11 @@ PRs are grouped per config to reduce noise. The npm config is pointed at the wor **No auto-merge.** Dependabot PRs go through the same merge gate as any other PR — an approval from the `@Wave-RF/wavehouse-admins` team (the ruleset's `required_reviewers` rule) plus the required checks. (The former `dependabot-automerge.yml`, which auto-approved and merged patch/minor bumps hands-off, was removed — every bump now gets a human admin review.) -## Releasing the SDK +## Releasing the SDKs -The TypeScript SDK (`@wavehouse/sdk`, in `clients/ts/`) publishes to npm via `.github/workflows/publish-npm.yml` using OIDC trusted publishing — no `NPM_TOKEN`. It is independent of the server's Go/Docker release (`release.yml`): the `v*` (server) and `sdk-v*` (SDK) tag globs are disjoint, so the two never collide. There are two channels: +**Go SDK** (`github.com/Wave-RF/WaveHouse/clients/go`): no tagged releases yet — `go get` resolves a pseudo-version from `main`. A tagged release requires a `clients/go/vX.Y.Z` tag (Go's nested-module tag form); the server's `v*` and npm's `sdk-v*` tag globs deliberately don't cover it, and no release workflow exists for it yet. + +**TypeScript SDK** (`@wavehouse/sdk`, in `clients/ts/`) publishes to npm via `.github/workflows/publish-npm.yml` using OIDC trusted publishing — no `NPM_TOKEN`. It is independent of the server's Go/Docker release (`release.yml`): the `v*` (server) and `sdk-v*` (SDK) tag globs are disjoint, so the two never collide. There are two channels: - **Dev snapshots.** Every push to `main` publishes `0.0.0-dev.` under the `dev` dist-tag — but only when the built `dist/` actually changed (the version is a hash of the build output, so an unchanged build resolves to an already-published version and is skipped). Install the bleeding edge with `npm install @wavehouse/sdk@dev`. - **Tagged releases.** Pushing a `sdk-vX.Y.Z` tag publishes that version and creates a GitHub Release. A stable version goes to the `latest` dist-tag; a prerelease (`sdk-v0.2.0-rc.1`) is published under `alpha`/`beta`/`rc`/`next` — derived from the suffix — and marked as a GitHub pre-release. The tag **must** match `clients/ts/package.json`'s `version`, or the job fails fast. diff --git a/docs/src/content/docs/pipes.mdx b/docs/src/content/docs/pipes.mdx index dc9f262c..9aa39486 100644 --- a/docs/src/content/docs/pipes.mdx +++ b/docs/src/content/docs/pipes.mdx @@ -148,7 +148,7 @@ curl -X PUT http://localhost:8080/v1/admin/pipes/top_pages \ }' ``` -A `PUT` is a full replace of that named pipe; `name` is taken from the URL. Definitions are stored in NATS KV and synced across nodes, so a create/update/delete applies cluster-wide without a restart. The [TypeScript SDK](/sdk) exposes the same operations as `client.pipes.list()`, `client.pipes.get(name)`, `client.pipes.set(name, def)`, and `client.pipes.delete(name)`. +A `PUT` is a full replace of that named pipe; `name` is taken from the URL. Definitions are stored in NATS KV and synced across nodes, so a create/update/delete applies cluster-wide without a restart. The [TypeScript SDK](/sdk) exposes the same operations as `client.pipes.list()`, `client.pipes.get(name)`, `client.pipes.set(name, def)`, and `client.pipes.delete(name)`; the [Go SDK](/sdk/go/pipes) as `wh.Pipes.List(ctx)`, `wh.Pipes.Get(ctx, name)`, `wh.Pipes.Set(ctx, name, def)`, and `wh.Pipes.Delete(ctx, name)`. ## Executing a pipe diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 280fab00..49d35c72 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -155,7 +155,7 @@ parameter. ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors are `*wavehouse.Error`; unwrap with `errors.As`. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close` — deliver errors through callbacks instead; see [Streaming](/sdk/go/streaming).) +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors originating from the HTTP exchange are `*wavehouse.Error`; unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close`, `Connected` — deliver errors through callbacks or plain errors instead; see [Streaming](/sdk/go/streaming).) ```go page, err := wh.From("clicks").Fetch(ctx) @@ -163,6 +163,8 @@ if err != nil { var whErr *wavehouse.Error if errors.As(err, &whErr) { fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } else { + fmt.Println("client-side failure:", err) // auth provider, marshal, ... } return err } diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 59c588bc..c7e342fb 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -225,10 +225,12 @@ clicks.Select("page"). Aggregate("uniqExact", "user_id", "unique_users") // custom fn ``` -Each aggregation method signature: `(column, alias string) *QueryBuilder`. -`Count` defaults to `column="*"` when `column` is `""`, and `alias="count"` -when `alias` is `""`; the other aggregations default `alias` to -`"_"` when left empty. +`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias +string)`; `Aggregate` takes `(fn, column, alias string)`. Empty-alias +defaults: `Count` → `count` (and `column=""` becomes `*`); `Sum`/`Avg`/ +`Min`/`Max` → `sum_`/`avg_`/`min_`/`max_`; +`CountDistinct` → `count_distinct_`. `Aggregate` has **no** alias +default — pass one explicitly or the query is sent with `"alias": ""`. #### `.GroupBy(...columns)` @@ -303,10 +305,10 @@ Execute the query and decode rows into `[]map[string]any`. The ordinary (non-generic) method form of `FetchTyped`. ```go -page, err := clicks.Select("page").Limit(50).FetchUntyped(ctx) +page, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUntyped(ctx) if page.HasMore && page.Next != nil { - page2, err := page.Next(ctx) // cursor-based pagination + page2, err := page.Next(ctx) // cursor-based pagination — needs OrderBy } ``` diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 54e2d2be..9e040031 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -39,12 +39,15 @@ background goroutine, torn down explicitly via `.Close()`. See ## Error Handling The SDK never panics on API or network failures — every request-response -operation (queries, ingest, pipes, admin) returns `(T, error)`, and errors -are always `*wavehouse.Error` (unwrap with `errors.As`). Streaming lifecycle -methods (`Stream`, `Subscribe`, `Close`) don't return `(T, error)`; stream -errors are delivered via the subscriber's `Error` callback. This is the -direct Go equivalent of the TypeScript SDK's "the SDK never throws" -guarantee. +operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors +originating from the HTTP exchange are `*wavehouse.Error` (unwrap with +`errors.As`); client-side failures before a request goes out (an `Auth` +provider error, a request-body marshal failure) are plain wrapped errors, +so handle the `errors.As == false` case too. Streaming lifecycle methods +(`Stream`, `Subscribe`, `Close`) don't return `(T, error)` — stream errors +are delivered via the subscriber's `Error` callback — and `Connected(ctx)` +returns plain errors. This is the direct Go equivalent of the TypeScript +SDK's "the SDK never throws" guarantee. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| @@ -63,6 +66,8 @@ if err != nil { var whErr *wavehouse.Error if errors.As(err, &whErr) { fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } else { + fmt.Println("client-side failure:", err) // auth provider, marshal, ... } return err } @@ -182,13 +187,13 @@ type ClicksRow struct { Page string `json:"page"` Button string `json:"button"` Score float64 `json:"score"` - ReceivedTimestamp string `json:"received_timestamp,omitempty"` + ReceivedTimestamp *string `json:"received_timestamp,omitempty"` } ``` (That's the exact output for the `clicks` table from the [development quick-start](/development#quick-start) — `received_timestamp` -gets `,omitempty` because it has a `DEFAULT` clause.) +becomes `*string` + `,omitempty` because it has a `DEFAULT` clause.) Note the generator does **not** special-case initialisms: `event_id` becomes `EventId`, not the Go-idiomatic `EventID` — each `_`-separated part simply @@ -197,8 +202,11 @@ gets its first letter upper-cased. Table and column names are converted to `PascalCase` for Go field/type names (a leading digit gets an `X` prefix — e.g. a table named `2fa_events` becomes `X2faEventsRow` — to stay a valid Go identifier). A column with -`has_default: true` in the schema gets `,omitempty` appended to its JSON -tag. +`has_default: true` in the schema becomes a **pointer field** with +`,omitempty` — the Go spelling of the TS codegen's `field?: T`: leave it +`nil` to omit the field (the server default applies), or point it at a +value to send it — including an explicit `0`/`false`/`""`, which a plain +value field with `omitempty` would silently drop. **ClickHouse → Go type mapping:** diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 659fe9fe..5724e623 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -69,7 +69,7 @@ closed explicitly. either way. A bare CDN URL tracks the latest published release; use a range (`@0`, `@0.1`) to float within a major or minor, or the `@dev` tag for unreleased builds from `main` (see - [Releasing the SDK](/development#releasing-the-sdk)). + [Releasing the SDKs](/development#releasing-the-sdks)). diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 6d8630df..d430bcfe 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -302,7 +302,9 @@ for (const f of failures) { } } -if (failed > 0) { +if (failed > 0 || skipped > 0 || passed === 0) { + if (passed === 0) console.log(" ✗ nothing ran — every case skipped or the fixture is empty\n"); + if (skipped > 0) console.log(" ✗ skipped cases break cross-SDK parity — wire up the endpoint above\n"); process.exit(1); } else { console.log(" ✓ All cases passed\n"); From 6e166da11be3d386301b3c0f0c5a3b443694fbeb Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 13:54:57 -0400 Subject: [PATCH 10/40] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=203=20=E2=80=94=20cursor=20ceiling=20honesty,=20Array(?= =?UTF-8?q?UInt8),=20codegen=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Untyped-path cursor precision: acknowledged as a documented ceiling rather than claimed fixed — FetchUntyped rows are float64-decoded before pagination sees them (same 2^53 ceiling as the TS SDK's JS numbers), so the code comment now says exactly that, queries.md documents the caveat next to the pagination example, and a regression test pins the behavior; FetchTyped and codegen structs remain exact - codegen: Array(UInt8) no longer generates []uint8 ([]byte, which encoding/json base64-encodes and the server rejects) — widened to []uint16; new main_test.go covers chTypeToGo (incl. this case), pascalCase's digit guard, findTopLevelComma, pointer-default output, and both collision failures (codegen package was 0% covered) - Docs: streaming.md documents terminal non-retryable stream errors, SSE_ERROR row added to both SDK reference tables, make test/fmt/ci descriptions synced, internal/stream added to the project tree, SQL example no longer redeclares rows :=, CONTRIBUTING gains the SDK-sync bullet (+ configuration.mdx path, also in SUPPORT.md), AGENTS.md drops the nonexistent "wavehouse-go" name for the real module path, landing/ why-wavehouse name the Go module importably, 404 page links the Go SDK --- AGENTS.md | 6 +- CONTRIBUTING.md | 3 +- SUPPORT.md | 2 +- clients/go/cmd/wavehouse-codegen/main.go | 10 +- clients/go/cmd/wavehouse-codegen/main_test.go | 139 ++++++++++++++++++ clients/go/query_builder.go | 9 +- clients/go/query_builder_test.go | 28 ++++ docs/src/content/docs/404.md | 1 + docs/src/content/docs/development.md | 11 +- docs/src/content/docs/index.mdx | 2 +- docs/src/content/docs/sdk/go/index.md | 7 +- docs/src/content/docs/sdk/go/queries.md | 9 +- docs/src/content/docs/sdk/go/reference.md | 1 + docs/src/content/docs/sdk/go/streaming.md | 6 + docs/src/content/docs/sdk/reference.md | 1 + docs/src/content/docs/why-wavehouse.md | 2 +- 16 files changed, 217 insertions(+), 20 deletions(-) create mode 100644 clients/go/cmd/wavehouse-codegen/main_test.go diff --git a/AGENTS.md b/AGENTS.md index 4ff5f47c..658706f8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,7 +60,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 11. **Hasura-style access control: fail-closed (security)** — `policy.IsAdmin` (role == `admin_role`, **exact case-sensitive**, default `"admin"`) is the single admin check, shared by `Evaluate`/`ResolveRole`/`Validate`/the `/v1/admin` gate/`RoleAllowed`. Empty/absent role matches nothing (no `"*"` wildcard); `Validate` rejects empty role keys; a `nil` policy (deleted) denies **everyone incl. admin** via a role — a total lockout for token-based callers, so bootstrap from the policy file, never an implicit admin grant (**exception:** the operator key's `auth.IsOperator` bit passes the `/v1/admin` gate even under a `nil` policy — a deliberate break-glass restore over HTTP, see #7). `default_role` is the one sanctioned roleless exception (`ResolveRole` maps empty → it pre-eval); `default_role == admin_role` is permitted but dev-only and loudly warned (`policy.DefaultRoleGrantsAdmin`). Preserve when touching `internal/policy` (policy twin of #13; see #159). Detail: architecture.md § `policy/`. 12. **Structured queries: column authz fail-closed (security)** — `POST /v1/query?table={table}`: typed AST validated against schema, permission-enforced, timestamp-bucketed for cache, `DefaultMaxRows` (10,000) cap. Every column reference — projection, aggregation args, `filters`, `group_by`, `order_by`, `time_range` — is authorized inside `query.Build` (the single chokepoint that enumerates them all), so no clause can skip the role's `allow_columns`/`deny_columns` check (#223). A `select_all` read by a *column-restricted* role expands to its allowed columns via `policy.AllowedProjection`, never a bare `SELECT *`; *unrestricted*/admin roles keep `SELECT *` (`policy.RestrictsColumns` decides). Omitting `columns` selects nothing (`ErrEmptyProjection` → `200 []`); `["*"]` is the literal column `*` (schema-gated, not a wildcard); a table-granted role with no readable columns fails closed (`ErrNoReadableColumns` → `403`). Structured and live-stream (`stream.filterColumns`) reads share the one per-column decision `policy.IsColumnAllowed`, so column visibility can't drift. Preserve when touching `internal/query` or the structured-query handler. Detail: architecture.md § `query/`. 13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). Detail: architecture.md § `pipes/`. -14. **Client SDKs** — TypeScript (`@wavehouse/sdk` in `clients/ts/`) and Go (`wavehouse-go` in `clients/go/`) are both canonical, officially supported clients with full API-tree parity. Zero third-party runtime dependencies in both. Each ships a typed query builder, real-time SSE streaming, live queries, and a codegen CLI. See §SDK Sync. +14. **Client SDKs** — TypeScript (`@wavehouse/sdk` in `clients/ts/`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`, package `wavehouse`, in `clients/go/`) are both canonical, officially supported clients with full API-tree parity. Zero third-party runtime dependencies in both. Each ships a typed query builder, real-time SSE streaming, live queries, and a codegen CLI. See §SDK Sync. 15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. The OTLP endpoint/TLS/custom-CA/mTLS/headers are delegated to the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` env vars — `InitProvider` passes **no** endpoint/header options. Known gap, intentionally not patched in WaveHouse app code: the pinned gRPC logs exporter (`otlploggrpc` v0.19/v0.20) ignores the env TLS-cert vars, so a custom/private CA and mutual TLS apply to traces/metrics but **not** the logs signal (public-CA/system-roots TLS and plaintext still work for logs) — upstream bug open-telemetry/opentelemetry-go#6661. A malformed `OTEL_EXPORTER_OTLP_HEADERS` is logged and skipped by the SDK (fail-soft), not fatal. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`. 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops. @@ -331,7 +331,7 @@ Diagrams render inside the Starlight content column (~46–58rem wide) as build- ## SDK Sync -The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) and Go SDK (`wavehouse-go` in `clients/go/`) are both canonical, officially supported clients. Both ship from this repo with full API-tree parity. When backend changes alter the public API surface, both SDKs need corresponding updates. The `pre-commit` git hook flags likely misses informationally; consult this table when deciding what to update. +The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) and Go SDK (`github.com/Wave-RF/WaveHouse/clients/go`, in `clients/go/`) are both canonical, officially supported clients. Both ship from this repo with full API-tree parity. When backend changes alter the public API surface, both SDKs need corresponding updates. The `pre-commit` git hook flags likely misses informationally; consult this table when deciding what to update. | Backend change | SDK considerations | | -------------- | ------------------ | @@ -388,7 +388,7 @@ Internal-only backend changes (middleware refactors, observability internals, de ```text cmd/ → Binary entry points (thin — just wiring) clients/ts/ → TypeScript SDK (@wavehouse/sdk) -clients/go/ → Go SDK (wavehouse-go) +clients/go/ → Go SDK (github.com/Wave-RF/WaveHouse/clients/go) wavehouse.go, http.go, errors.go, types.go → Client core (constructor, transport, errors, shared types) query_builder.go, table.go → Structured query builder + per-table typed client stream.go, live_query.go → SSE streaming + live queries diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 82372627..1b459bd7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -43,9 +43,10 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t 3. Update documentation if your change affects: - API endpoints → update `docs/src/content/docs/api.md` - - Configuration options → update `docs/src/content/docs/configuration.md` + - Configuration options → update `docs/src/content/docs/configuration.mdx` - Deployment → update `docs/src/content/docs/deployment.md` - Architecture → update `docs/src/content/docs/architecture.md` + - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), their doc trees (`docs/src/content/docs/sdk/` and `.../sdk/go/`), and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync 4. Follow the commit message format (see below). diff --git a/SUPPORT.md b/SUPPORT.md index 30f185cc..eb331fe3 100644 --- a/SUPPORT.md +++ b/SUPPORT.md @@ -23,7 +23,7 @@ WaveHouse is in **alpha**. We're a small team building publicly while shipping t - Bug reports against the latest tagged release or `main` HEAD. - Reproducible regressions vs. the previous tag. -- Documentation gaps or wrong examples (especially `getting-started.md`, `api.md`, `configuration.md`). +- Documentation gaps or wrong examples (especially `getting-started.md`, `api.md`, `configuration.mdx`). - Configuration questions where the docs disagree with reality. ## Out of scope during alpha diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 793e1938..0ecfb885 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -209,8 +209,14 @@ func chTypeToGo(chType string) string { } // Array. if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { - inner := chType[6 : len(chType)-1] - return "[]" + chTypeToGo(inner) + inner := chTypeToGo(chType[6 : len(chType)-1]) + // []uint8 is []byte, which encoding/json base64-encodes as a string — + // the server requires a real JSON array for Array(...) columns, so + // widen the element type instead. + if inner == "uint8" { + inner = "uint16" + } + return "[]" + inner } // Map. if strings.HasPrefix(chType, "Map(") && strings.HasSuffix(chType, ")") { diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go new file mode 100644 index 00000000..2fe23a0e --- /dev/null +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -0,0 +1,139 @@ +package main + +import ( + "strings" + "testing" +) + +func TestChTypeToGo(t *testing.T) { + tests := []struct { + ch string + want string + }{ + {"String", "string"}, + {"FixedString(16)", "string"}, + {"UUID", "string"}, + {"DateTime64(3, 'UTC')", "string"}, + {"Date", "string"}, + {"Time64(3)", "string"}, + {"Enum8('a' = 1)", "string"}, + {"IPv4", "string"}, + {"Bool", "bool"}, + {"Boolean", "bool"}, + {"UInt8", "uint8"}, + {"UInt16", "uint16"}, + {"UInt32", "uint32"}, + {"Int8", "int8"}, + {"Int32", "int32"}, + {"Float32", "float32"}, + {"BFloat16", "float32"}, + {"Float64", "float64"}, + // 64-bit and wider integers are quoted in ClickHouse JSON output. + {"UInt64", "string"}, + {"Int64", "string"}, + {"UInt128", "string"}, + {"Int256", "string"}, + {"Decimal(18, 4)", "string"}, + {"Nullable(Int32)", "*int32"}, + {"Nullable(Int64)", "*string"}, + {"LowCardinality(String)", "string"}, + {"LowCardinality(Nullable(String))", "*string"}, + {"SimpleAggregateFunction(sum, UInt32)", "uint32"}, + {"SimpleAggregateFunction(any)", "any"}, + {"Array(String)", "[]string"}, + {"Array(Nullable(Int32))", "[]*int32"}, + // []uint8 is []byte → base64 on marshal; must widen. + {"Array(UInt8)", "[]uint16"}, + {"Map(String, UInt32)", "map[string]uint32"}, + {"Map(String, Map(UInt32, String))", "map[string]map[uint32]string"}, + {"Tuple(String, UInt8)", "any"}, + {"SomethingNew", "any"}, + } + for _, tt := range tests { + if got := chTypeToGo(tt.ch); got != tt.want { + t.Errorf("chTypeToGo(%q) = %q, want %q", tt.ch, got, tt.want) + } + } +} + +func TestPascalCase(t *testing.T) { + tests := []struct { + in string + want string + }{ + {"clicks", "Clicks"}, + {"user_id", "UserId"}, + {"received_timestamp", "ReceivedTimestamp"}, + {"multi-part.name here", "MultiPartNameHere"}, + {"2fa_events", "X2faEvents"}, // leading digit gets the X prefix + {"", ""}, + } + for _, tt := range tests { + if got := pascalCase(tt.in); got != tt.want { + t.Errorf("pascalCase(%q) = %q, want %q", tt.in, got, tt.want) + } + } +} + +func TestFindTopLevelComma(t *testing.T) { + tests := []struct { + in string + want int + }{ + {"String, UInt32", 6}, + {"Map(String, String), UInt8", 19}, + {"NoComma", -1}, + } + for _, tt := range tests { + if got := findTopLevelComma(tt.in); got != tt.want { + t.Errorf("findTopLevelComma(%q) = %d, want %d", tt.in, got, tt.want) + } + } +} + +func TestGenerate_Basic(t *testing.T) { + out, err := generate(map[string]tableSchema{ + "clicks": {Name: "clicks", Columns: []column{ + {Name: "page", Type: "String"}, + {Name: "score", Type: "Float64"}, + {Name: "received_timestamp", Type: "DateTime64(3, 'UTC')", HasDefault: true}, + }}, + }, "myapp") + if err != nil { + t.Fatal(err) + } + for _, want := range []string{ + "package myapp", + "type ClicksRow struct {", + "Page string `json:\"page\"`", + "Score float64 `json:\"score\"`", + // Defaulted column: pointer + omitempty so an explicit zero still sends. + "ReceivedTimestamp *string `json:\"received_timestamp,omitempty\"`", + } { + if !strings.Contains(out, want) { + t.Errorf("generated output missing %q:\n%s", want, out) + } + } +} + +func TestGenerate_FieldCollisionFails(t *testing.T) { + _, err := generate(map[string]tableSchema{ + "t": {Name: "t", Columns: []column{ + {Name: "user_id", Type: "String"}, + {Name: "userId", Type: "String"}, + }}, + }, "main") + if err == nil || !strings.Contains(err.Error(), "UserId") { + t.Fatalf("want field-collision error naming UserId, got %v", err) + } +} + +func TestGenerate_TypeCollisionFails(t *testing.T) { + _, err := generate(map[string]tableSchema{ + "2fa": {Name: "2fa", Columns: []column{{Name: "a", Type: "String"}}}, + "x2fa": {Name: "x2fa", Columns: []column{{Name: "a", Type: "String"}}}, + }, "main") + if err == nil || !strings.Contains(err.Error(), "X2faRow") { + t.Fatalf("want type-collision error naming X2faRow, got %v", err) + } +} diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index c1c540e6..07c16783 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -276,8 +276,13 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro m, ok := lastRow.(map[string]any) if !ok { // ponytail: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. - // UseNumber keeps int64 cursor values exact; plain float64 decoding - // corrupts IDs past 2^53 and pagination would repeat or skip a row. + // UseNumber keeps typed int64 cursor values exact past 2^53. The + // untyped path (FetchUntyped / TableRef.Fetch) doesn't get this + // protection: its rows were already decoded to float64 by + // encoding/json, so precision above 2^53 is gone before we get here — + // the same ceiling the TS SDK has with JS numbers. Use FetchTyped (or + // codegen structs, whose 64-bit int columns are strings) when paging + // on >2^53 integer cursors. raw, _ := json.Marshal(lastRow) m = make(map[string]any) dec := json.NewDecoder(bytes.NewReader(raw)) diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index 4c4b7656..02431746 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -428,3 +428,31 @@ func TestQueryBuilder_Pagination_TypedInt64CursorKeepsPrecision(t *testing.T) { t.Fatalf("cursor value lost precision: %s", got) } } + +// TestQueryBuilder_Pagination_UntypedCursorFloat64Ceiling documents the known +// ceiling on the untyped path: rows decode to float64, so an integer cursor +// past 2^53 loses precision before pagination sees it (same as the TS SDK's +// JS-number ceiling). Use FetchTyped or codegen structs past 2^53. +func TestQueryBuilder_Pagination_UntypedCursorFloat64Ceiling(t *testing.T) { + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": 1}, {"id": int64(9007199254740993)}}, + {}, + }) + + page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if _, err := page.Next(context.Background()); err != nil { + t.Fatal(err) + } + + f := filtersOf(t, getBodies()[1]) + if len(f) != 1 { + t.Fatalf("want 1 cursor filter, got %v", f) + } + // float64 rounds 2^53+1 down to 2^53 — the documented untyped ceiling. + if got := fmt.Sprint(f[0]["value"]); got != "9007199254740992" { + t.Fatalf("untyped ceiling changed (update docs if intentional): %s", got) + } +} diff --git a/docs/src/content/docs/404.md b/docs/src/content/docs/404.md index 36e006ca..70e2069b 100644 --- a/docs/src/content/docs/404.md +++ b/docs/src/content/docs/404.md @@ -44,6 +44,7 @@ head: ArchitectureHow the pieces fit together API referenceEndpoints, payloads, and error semantics TypeScript SDKTyped client for browser and Node + Go SDKTyped client for Go services

Followed a link that should have worked? File an issue — broken links are bugs.

diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index a7bbf169..ea642152 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -297,7 +297,7 @@ All tests run with Go's **race detector** (`-race`) enabled by default. WaveHous ```bash # Prefix any test target with V=1 for verbose output, e.g. `V=1 make test` -# Unit tests (compact output) — alias for `test-unit` +# Unit tests + Go SDK tests (compact output) — alias for `test-unit` + `test-go-sdk` make test # Run specific test(s) @@ -316,8 +316,8 @@ make test-e2e # All suites sequentially + merged coverage make test-all -# Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts, -# then test-integration + test-e2e + cov +# Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts + +# test-conformance-ts, then test-integration + test-e2e + cov make ci # Merge available covdata + gate against total threshold @@ -431,6 +431,7 @@ WaveHouse/ │ ├── pipes/ # Named query pipes (NATS KV + .sql bootstrap) │ ├── policy/ # Access control policies (evaluation + NATS KV store) │ ├── query/ # Structured query AST + SQL builder +│ ├── stream/ # SSE fan-out (event Hub, subscriber queues, keepalive) │ └── testutil/ # Shared test helpers and mocks ├── clients/ # Official SDKs │ ├── ts/ # TypeScript SDK (@wavehouse/sdk) @@ -482,11 +483,11 @@ Run `make help` to see all targets. Key ones: | `make obs-grafana` | Grafana alternative to aspire, more advanced and complicated | | `make obs-front` | Custom graphs like grafana, but is simpler and easier to configure like aspire | | **Static checks** | | -| `make fmt` | Check formatting across Go (`gofumpt`) + TS (Biome). Run `make fix` to apply. | +| `make fmt` | Check formatting across root-module Go (`gofumpt`) + TS (Biome); the nested `clients/go` module's gofumpt check runs under `make verify` (`verify-go-sdk`). Run `make fix` to apply everywhere. | | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | | `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) | | `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | -| `make verify` | Repo-wide static checks: Go (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | +| `make verify` | Repo-wide static checks: Go incl. `clients/go` (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | | `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`) and TS (Biome `--write`) | | **Build** | | | `make build` | Compile `wavehouse` → `bin/wavehouse` (debug symbols kept) | diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index 1492292e..9308a38e 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -96,7 +96,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click Per-table, per-role column and row-level policies with JWT claim templating. Stored in NATS KV with file-based bootstrap and cluster sync. - `@wavehouse/sdk` and `clients/go` — zero-dependency clients with type-safe query builders, live queries, real-time streaming, and codegen from your schemas, speaking one shared wire format. + `@wavehouse/sdk` (TypeScript) and `github.com/Wave-RF/WaveHouse/clients/go` — zero-dependency clients with type-safe query builders, live queries, real-time streaming, and codegen from your schemas, speaking one shared wire format. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 49d35c72..0cc2f4c3 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -198,9 +198,10 @@ languages pull the API shape in different directions: - **No implicit "await."** A `QueryBuilder` isn't `PromiseLike` — call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly; there's no bare `await builder` shortcut. -- **`Insert` accepts typed row slices, not just maps.** Passing - `[]ClickRow{...}` (any slice type, detected via reflection) batches as - NDJSON exactly like `[]map[string]any` — see +- **Any slice batches, not just `[]map[string]any`.** Go detects slice-ness + via reflection, so `[]ClickRow{...}` takes the same NDJSON batch path as + `[]map[string]any` (the TS SDK's `insert` likewise accepts arrays of typed + rows — this bullet is about the Go mechanics, not a TS gap) — see [Queries → Insert](/sdk/go/queries#insertctx-data). ## Explore the Go SDK diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index c7e342fb..6ef9380f 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -340,6 +340,13 @@ add an `.OrderBy()` to paginate. If the order column was left out of an explicit `.Select(...)` projection, `Next` quietly returns an empty page instead of erroring (there is no cursor value to read). +One precision caveat on the untyped path (`FetchUntyped` / `TableRef.Fetch`): +rows decode into `map[string]any`, where JSON numbers become `float64`, so an +integer cursor column loses exactness past 2^53 and pagination can repeat or +skip a row at that scale — the same ceiling the TypeScript SDK has with JS +numbers. `FetchTyped` with an `int64` field keeps the cursor exact, and +codegen structs are unaffected (their 64-bit integer columns are `string`). + ```go page, err := clicks.Select(). OrderBy("received_timestamp", "desc"). @@ -380,7 +387,7 @@ type PageTotal struct { Page string `json:"page"` Total int `json:"total"` } -rows, err := wavehouse.SQL[PageTotal](ctx, wh, +typed, err := wavehouse.SQL[PageTotal](ctx, wh, "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") ``` diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 9e040031..271143b6 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -59,6 +59,7 @@ SDK's "the SDK never throws" guarantee. | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `context.Context` | +| 0 | `SSE_ERROR` | Yes | Stream connection failure, delivered to the subscriber's `Error` callback; the stream reconnects automatically | ```go page, err := wh.From("clicks").Fetch(ctx) diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 55b31243..90107898 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -149,6 +149,12 @@ type StreamEvent struct { | --------- | --------- | -------- | | SSE | Automatic, with exponential backoff (capped at 30s) and gap-fill replay via the last-seen event ID | HTTP/2 recommended | +Reconnect covers transport failures and retryable (5xx) responses. A +non-retryable response (401/403/404) is terminal: the error is delivered to +the subscriber's `Error` callback, status goes to `StatusClosed`, and the +stream does not reconnect — fix the cause (refresh the token, correct the +table) and open a new stream. + Auth is sent as an `Authorization: Bearer` header on every stream (re)connection — see [the note in the Getting Started guide](/sdk/go#creating-a-client). The diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index 2c9bd2d5..5d54ca4b 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -37,6 +37,7 @@ The SDK **never throws**. All errors are returned in `Result.error`. | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries with `Retry-After`) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `AbortSignal` | +| 0 | `SSE_ERROR` | Yes | Stream connection failure, delivered to the stream's error callback; the stream reconnects automatically | --- diff --git a/docs/src/content/docs/why-wavehouse.md b/docs/src/content/docs/why-wavehouse.md index 5cce04ba..8302d3a4 100644 --- a/docs/src/content/docs/why-wavehouse.md +++ b/docs/src/content/docs/why-wavehouse.md @@ -154,7 +154,7 @@ flowchart TB | Schema validation | Custom code in ingest API | Built in (discovers `system.columns`) | | Row/column access control | Custom middleware or a dedicated service | Built in (Hasura-style, JWT-driven) | | Dead letter queue | Custom retry + dead topic on Kafka | Built in (`WAVEHOUSE_DLQ`) | -| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript) + `clients/go` (Go) — zero-dep, codegen | +| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript) + `github.com/Wave-RF/WaveHouse/clients/go` — zero-dep, codegen | The DIY path works — big teams run it — but the ops cost is not small. You're paying for a Kafka cluster (or Confluent bill), a second service you wrote from scratch, and all the debugging hours when the batching consumer stalls at 3 a.m. From 4b199b617400f5f47c592b7b007e684e191a27b5 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 14:51:03 -0400 Subject: [PATCH 11/40] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=204=20=E2=80=94=2064-bit=20codegen=20mapping,=20LIKE?= =?UTF-8?q?=20compile,=20doc=20precision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - codegen: Int64/UInt64 map to int64/uint64 and 128/256-bit ints to json.Number (with a conditional encoding/json import) — generated structs target /v1/query and /v1/pipes/*, where the server scans ClickHouse values into Go types and re-marshals them as UNQUOTED numbers; the round-1 string mapping only held for /v1/admin/query, which forwards ClickHouse's own quoted JSON (use map[string]any with SQL[Row] there). Decode round-trip test pins the wire shape; docs type table and parity paragraph updated, pagination caveat now notes 64-bit codegen columns decode exactly - Filtered streams compile LIKE patterns once at construction — the process-global likeRegexCache sync.Map (unbounded, keyed on caller input) is gone, and per-event matching is a plain regex call - Docs: /sdk/go Quick Start is compilable (package main + func main, like the README); client concurrency-safety documented; Events() first-call feeding note; SELECT * expansion claim scoped to column-restricted roles (both SDK pages); Array(UInt8) exception in the type table; codegen go run uses @latest so it works outside the repo; stale TableRef "NOT safe for mutations" comment corrected (it holds no mutable state) - AGENTS.md: configuration.mdx path, both SDK readmes in the prose list - CHANGELOG: Go SDK entry expanded to house style (module path, nested- module caveat, new targets, conformance wiring) --- AGENTS.md | 8 +-- CHANGELOG.md | 2 +- clients/go/README.md | 2 +- clients/go/cmd/wavehouse-codegen/main.go | 44 ++++++++---- clients/go/cmd/wavehouse-codegen/main_test.go | 54 ++++++++++++-- clients/go/stream.go | 71 ++++++++++++------- clients/go/stream_test.go | 7 +- clients/go/table.go | 6 +- docs/src/content/docs/sdk/go/index.md | 40 +++++++---- docs/src/content/docs/sdk/go/queries.md | 17 +++-- docs/src/content/docs/sdk/go/reference.md | 23 +++--- docs/src/content/docs/sdk/go/streaming.md | 7 ++ docs/src/content/docs/sdk/queries.md | 2 +- 13 files changed, 193 insertions(+), 90 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 658706f8..262d3512 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -293,7 +293,7 @@ Then run the reviewers relevant to the PR's diff (the same set from `scripts/pre Documentation *prose* — accuracy against the code, runnable examples, clarity, completeness — **and code↔docs sync** (code that changed but whose docs didn't) are reviewed by the **`docs-reviewer`** subagent, not the code-focused `pre-push-reviewer`. The canonical rubric is `.github/prompts/docs-review.md`. It complements the deterministic prose tools — misspell, markdownlint, starlight-links-validator — reviewing only what they can't, and it never edits docs or posts PR comments. -**Scope** is the canonical docs-prose set from `scripts/docs-prose.sh` — a *denylist*: every tracked `.md`/`.mdx` EXCEPT `.claude/**`, `.github/**`, `CHANGELOG.md`, `AGENTS.md`, `CLAUDE.md`, `*.draft.md`/`*.old.md`, `PERF-CLAIMS-REVIEW.md`, `docs/posthog-setup-report.md`. So it covers the Starlight site under `docs/src/content/` **and** the governance docs (`README.md`, the SDK readme `clients/ts/README.md`, `CONTRIBUTING.md`, `SECURITY.md`, `CODE_OF_CONDUCT.md`, `SUPPORT.md`) — new docs are picked up automatically. `CODE_OF_CONDUCT.md`/`SUPPORT.md` are deep-reviewed only on change or material suspicion. +**Scope** is the canonical docs-prose set from `scripts/docs-prose.sh` — a *denylist*: every tracked `.md`/`.mdx` EXCEPT `.claude/**`, `.github/**`, `CHANGELOG.md`, `AGENTS.md`, `CLAUDE.md`, `*.draft.md`/`*.old.md`, `PERF-CLAIMS-REVIEW.md`, `docs/posthog-setup-report.md`. So it covers the Starlight site under `docs/src/content/` **and** the governance docs (`README.md`, the SDK readmes `clients/ts/README.md` / `clients/go/README.md`, `CONTRIBUTING.md`, `SECURITY.md`, `CODE_OF_CONDUCT.md`, `SUPPORT.md`) — new docs are picked up automatically. `CODE_OF_CONDUCT.md`/`SUPPORT.md` are deep-reviewed only on change or material suspicion. **It is a hard pre-push gate**, run in parallel with the other pre-push reviewers (see §Pre-push self-review). Invoked with the **default (branch) scope** it emits a `VERDICT:` line; on `ship_it` the `review-marker.sh` SubagentStop hook writes `tmp/docs-reviewer-passed-`, which the push gate requires — unconditionally, on every PR-branch push (even code-only ones). Run it via **`/docs-review`**; with **no arg** that's the gating review (branch scope), while an explicit **path/glob** or **`all`** is **advisory** (no `VERDICT:`, no marker) for ad-hoc audits. The whole dev team runs Claude Code and this command is tracked in-repo, so everyone runs it themselves; there is intentionally **no PR/cloud path** for docs review. @@ -304,7 +304,7 @@ Every code change should update the corresponding docs in the same PR. A code ch | Change | Files to update | | ------ | --------------- | | Add/modify API endpoint | `docs/src/content/docs/api.md`, `README.md` (if user-facing) | -| Add/modify config option | `docs/src/content/docs/configuration.md`, `config.yaml`, `deployments/compose/*` env blocks, `docs/src/content/docs/deployment.md` | +| Add/modify config option | `docs/src/content/docs/configuration.mdx`, `config.yaml`, `deployments/compose/*` env blocks, `docs/src/content/docs/deployment.md` | | Change architecture / add a package | `docs/src/content/docs/architecture.md`, `AGENTS.md` | | Change ingest / event format | `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md` (CH schema) | | Change deployment / Docker | `docs/src/content/docs/deployment.md`, compose files | @@ -313,7 +313,7 @@ Every code change should update the corresponding docs in the same PR. A code ch Source-of-truth pairs that must agree: -- Config struct tags in `internal/config/config.go` ↔ `docs/src/content/docs/configuration.md`, `config.yaml`, compose env blocks +- Config struct tags in `internal/config/config.go` ↔ `docs/src/content/docs/configuration.mdx`, `config.yaml`, compose env blocks - `EventMessage` JSON tags ↔ `docs/src/content/docs/api.md` event format, SSE examples, ClickHouse INSERT columns - Route registrations in `router.go` ↔ `docs/src/content/docs/api.md` endpoint list - Handler error responses ↔ `docs/src/content/docs/api.md` error tables @@ -363,7 +363,7 @@ Internal-only backend changes (middleware refactors, observability internals, de 1. Add the field to the appropriate struct in `internal/config/config.go` with `yaml`, `env`, and `env-default` tags. 2. Use the new config value in `cmd/wavehouse/main.go` or the relevant internal package. -3. Document in `docs/src/content/docs/configuration.md`. +3. Document in `docs/src/content/docs/configuration.mdx`. ### Adding a new internal package diff --git a/CHANGELOG.md b/CHANGELOG.md index 627487f8..d478b4a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Go SDK (`clients/go/`) — official Go client with full API-tree parity against the TypeScript SDK, zero third-party runtime dependencies, cross-language wire-format conformance tests +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), and a `wavehouse-codegen` CLI that generates row structs from `/v1/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. - **Non-JWT operator key for full-access admin + break-glass recovery** (`internal/auth/auth.go`, `internal/auth/context.go`, `internal/auth/auth_test.go`, `internal/api/router.go`, `internal/api/router_test.go`, `internal/config/config.go`, `internal/config/config_test.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `tests/e2e/fixtures/config.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/access-control.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/development.md`, `docs/src/content/docs/architecture.md`, `docs/src/content/docs/reverse-proxy.mdx`, `AGENTS.md`, `SECURITY.md`): closes #240; partially advances the auth-hardening epic [#228](https://github.com/Wave-RF/WaveHouse/issues/228) and the management/data-plane split [#359](https://github.com/Wave-RF/WaveHouse/issues/359). Adds an optional `auth.operator_key` (`WH_AUTH_OPERATOR_KEY`): a request presenting it — via an `Authorization: Operator ` header (forwarded verbatim by proxies, no collision with Bearer JWTs) or the `X-Operator-Key` alias — is authorized as a full-access platform operator — the whole data plane *and* the `/v1/admin/*` management surface — without minting a JWT and independently of the token verifier, giving the person running the deployment a role-free credential for bootstrap and break-glass. The middleware checks it before the Bearer token with a constant-time comparison (`crypto/subtle`) and stamps two things into the request context: the live `admin_role` (so the policy evaluator's admin bypass grants unrestricted data-plane access while a policy exists) and a platform-operator bit that `RequireAdmin` honors **even when the policy is `nil`/deleted** — the one HTTP path that can restore a wiped policy, which previously required SSH access and a reboot. Empty (the default) disables it, so existing deployments are unchanged; treat it as an admin secret (load from a secret store, serve only over TLS). A successful operator authentication is audit-logged at `INFO`; a request presenting a *non-matching* operator key is logged at `WARN` and counted by a new `wavehouse_auth_operator_key_failures_total` counter (a probing/brute-force signal on the most privileged credential in the system) before falling through to the normal token/default path — the middleware still never rejects. The `Authorization` auth-scheme is matched case-insensitively (RFC 7235) via a shared `authScheme` helper, which also makes the existing `Bearer` JWT scheme case-insensitive (previously it required the canonical `Bearer` casing). Explicitly out of scope, tracked in #359: scoping the operator credential to the management surface only, and capability-scoped admin permissions. - **Missing-dedupe-id observability + optional strict mode** (`internal/api/ingest.go`, `internal/api/ingest_test.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/architecture.md`): closes #219. With dedupe enabled, a row missing the configured `id_field` can't be deduped — previously it was published with idempotency silently disabled and *no* log or metric, so a producer bug that dropped the id turned off the guarantee for those rows unnoticed. Now every such row is logged at `WARN` and counted by a new `wavehouse_ingest_dedupe_missing_id_total` counter (labeled by `table`), making the loss observable server-side. A new opt-in `dedupe.require_id` (`WH_DEDUPE_REQUIRE_ID`, default `false`) turns that signal into enforcement: a row missing the id is rejected (`400` for a single insert; a per-record failure in a batch) instead of published — a tripwire for producers that must guarantee the id (complements the client-side [#202](https://github.com/Wave-RF/WaveHouse/issues/202)). Default behavior is unchanged. - **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes. diff --git a/clients/go/README.md b/clients/go/README.md index bd991359..cf2c1348 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -194,7 +194,7 @@ Generate Go structs from a running WaveHouse instance: ```bash export WAVEHOUSE_AUTH= # avoids leaking the token via argv -go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --url http://localhost:8080 \ --out ./db_types.go \ --package myapp diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 0ecfb885..f46998c1 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -185,27 +185,32 @@ func chTypeToGo(chType string) string { case chType == "Bool", chType == "Boolean": return "bool" } - // Numeric — map lookup. + // Numeric — map lookup. Generated structs target the structured-query and + // pipe paths (/v1/query, /v1/pipes/*), where the server scans ClickHouse + // values into Go types and re-marshals them — so 64-bit integers arrive + // as ordinary UNQUOTED JSON numbers and map to int64/uint64 exactly. + // (Only /v1/admin/query forwards ClickHouse's own JSON, which quotes + // 64-bit ints; use map[string]any with SQL[Row] there.) if mapped, ok := map[string]string{ - "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", - "Int8": "int8", "Int16": "int16", "Int32": "int32", + "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", + "Int8": "int8", "Int16": "int16", "Int32": "int32", "Int64": "int64", "Float32": "float32", "Float64": "float64", "BFloat16": "float32", }[chType]; ok { return mapped } switch { - case strings.HasPrefix(chType, "Decimal"), - strings.HasPrefix(chType, "UInt64"), - strings.HasPrefix(chType, "UInt128"), + case strings.HasPrefix(chType, "Decimal"): + // Decimals are marshaled as quoted strings on the structured path + // (shopspring decimal.MarshalJSON quotes by default). + return "string" + case strings.HasPrefix(chType, "UInt128"), strings.HasPrefix(chType, "UInt256"), - strings.HasPrefix(chType, "Int64"), strings.HasPrefix(chType, "Int128"), strings.HasPrefix(chType, "Int256"): - // 64-bit and bigger integers (and decimals) are strings on the wire: - // ClickHouse's JSON output quotes them by default - // (output_format_json_quote_64bit_integers=1) and the server forwards - // the ClickHouse `data` array verbatim. - return "string" + // 128/256-bit ints scan into *big.Int server-side and marshal as + // unquoted JSON numbers of arbitrary width — json.Number preserves + // them exactly where int64/uint64 would overflow. + return "json.Number" } // Array. if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { @@ -286,10 +291,23 @@ func sortedKeys(m map[string]tableSchema) []string { func generate(schemas map[string]tableSchema, pkg string) (string, error) { var sb strings.Builder - fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) names := sortedKeys(schemas) + // json.Number fields (128/256-bit integer columns) need the import. + needsJSON := false + for _, name := range names { + for _, col := range schemas[name].Columns { + if strings.Contains(chTypeToGo(col.Type), "json.Number") { + needsJSON = true + } + } + } + fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) + if needsJSON { + sb.WriteString("import \"encoding/json\"\n\n") + } + // pascalCase is not injective ("user_id" and "userId" both yield // "UserId"), and format.Source only parses — it doesn't type-check — so // a duplicate identifier would be written as a non-compiling file with a diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go index 2fe23a0e..47b2ce3e 100644 --- a/clients/go/cmd/wavehouse-codegen/main_test.go +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -1,6 +1,8 @@ package main import ( + "encoding/json" + "go/format" "strings" "testing" ) @@ -28,14 +30,14 @@ func TestChTypeToGo(t *testing.T) { {"Float32", "float32"}, {"BFloat16", "float32"}, {"Float64", "float64"}, - // 64-bit and wider integers are quoted in ClickHouse JSON output. - {"UInt64", "string"}, - {"Int64", "string"}, - {"UInt128", "string"}, - {"Int256", "string"}, + // /v1/query re-marshals server-side: 64-bit ints arrive unquoted. + {"UInt64", "uint64"}, + {"Int64", "int64"}, + {"UInt128", "json.Number"}, + {"Int256", "json.Number"}, {"Decimal(18, 4)", "string"}, {"Nullable(Int32)", "*int32"}, - {"Nullable(Int64)", "*string"}, + {"Nullable(Int64)", "*int64"}, {"LowCardinality(String)", "string"}, {"LowCardinality(Nullable(String))", "*string"}, {"SimpleAggregateFunction(sum, UInt32)", "uint32"}, @@ -137,3 +139,43 @@ func TestGenerate_TypeCollisionFails(t *testing.T) { t.Fatalf("want type-collision error naming X2faRow, got %v", err) } } + +// TestGeneratedShapeDecodesStructuredQueryPayload asserts the mapping choices +// actually decode what /v1/query emits: the server scans ClickHouse values +// into Go types and re-marshals, so 64-bit ints are unquoted numbers, +// 128/256-bit ints are unquoted arbitrary-width numbers, and Decimals are +// quoted strings. +func TestGeneratedShapeDecodesStructuredQueryPayload(t *testing.T) { + type row struct { + ID uint64 `json:"id"` + Delta int64 `json:"delta"` + Big json.Number `json:"big"` + Price string `json:"price"` + } + payload := `[{"id":18446744073709551615,"delta":-9007199254740993,"big":170141183460469231731687303715884105727,"price":"12.3400"}]` + var rows []row + if err := json.Unmarshal([]byte(payload), &rows); err != nil { + t.Fatalf("generated shape failed to decode /v1/query payload: %v", err) + } + if rows[0].ID != 18446744073709551615 || rows[0].Delta != -9007199254740993 { + t.Fatalf("64-bit values corrupted: %+v", rows[0]) + } + if rows[0].Big.String() != "170141183460469231731687303715884105727" { + t.Fatalf("128-bit value corrupted: %s", rows[0].Big) + } +} + +func TestGenerate_JSONNumberImport(t *testing.T) { + out, err := generate(map[string]tableSchema{ + "t": {Name: "t", Columns: []column{{Name: "big", Type: "UInt128"}}}, + }, "main") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(out, `import "encoding/json"`) { + t.Fatalf("json.Number field without encoding/json import:\n%s", out) + } + if _, err := format.Source([]byte(out)); err != nil { + t.Fatalf("generated output is not valid Go: %v", err) + } +} diff --git a/clients/go/stream.go b/clients/go/stream.go index 418b11ec..6da259be 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -396,6 +396,7 @@ func (sc *StreamController) handleSSEData(data, eventID string) { // newFilteredStreamController wraps a StreamController with client-side // filtering and column projection. func newFilteredStreamController(inner *StreamController, filters []QueryFilter, columns []string) *StreamController { + compiled := compileFilters(filters) ctx, cancel := context.WithCancel(context.Background()) sc := &StreamController{ status: inner.Status(), @@ -416,7 +417,7 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, unsub := inner.Subscribe(&StreamSubscriber{ Next: func(event StreamEvent) { - if !matchesFilters(event.Data, filters) { + if !matchesFilters(event.Data, compiled) { return } if len(columns) > 0 { @@ -443,18 +444,54 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, return sc } +// compiledFilter pairs a filter with its precompiled LIKE regex (nil for +// every other operator, or when the pattern isn't a string / doesn't compile). +type compiledFilter struct { + QueryFilter + re *regexp.Regexp +} + +// compileFilters precompiles LIKE/NOT LIKE patterns once per stream. A +// controller's filters never change after construction, so this replaces a +// per-event compile (and avoids any process-global pattern cache). +func compileFilters(filters []QueryFilter) []compiledFilter { + out := make([]compiledFilter, len(filters)) + for i, f := range filters { + out[i] = compiledFilter{QueryFilter: f} + if f.Op == "like" || f.Op == "not_like" { + if pattern, ok := f.Value.(string); ok { + out[i].re = compileLike(pattern) + } + } + } + return out +} + +// compileLike converts a SQL LIKE pattern to a case-insensitive anchored +// regex (matching the TS SDK). Returns nil if the pattern doesn't compile. +func compileLike(pattern string) *regexp.Regexp { + escaped := regexp.QuoteMeta(pattern) + escaped = strings.ReplaceAll(escaped, "%", ".*") + escaped = strings.ReplaceAll(escaped, "_", ".") + re, err := regexp.Compile("(?i)^" + escaped + "$") + if err != nil { + return nil + } + return re +} + // matchesFilters evaluates all filters against a data row (AND). -func matchesFilters(row map[string]any, filters []QueryFilter) bool { +func matchesFilters(row map[string]any, filters []compiledFilter) bool { for _, f := range filters { val := row[f.Column] - if !evaluateFilter(val, f.Op, f.Value) { + if !evaluateFilter(val, f.Op, f.Value, f.re) { return false } } return true } -func evaluateFilter(actual any, op string, expected any) bool { +func evaluateFilter(actual any, op string, expected any, re *regexp.Regexp) bool { switch op { case "eq": return equalValues(actual, expected) @@ -475,12 +512,11 @@ func evaluateFilter(actual any, op string, expected any) bool { case "in": return evaluateIn(actual, expected) case "like", "not_like": - aStr, aOK := actual.(string) - eStr, eOK := expected.(string) - if !aOK || !eOK { + aStr, ok := actual.(string) + if !ok || re == nil { return false } - return (op == "like") == matchLike(aStr, eStr) + return (op == "like") == re.MatchString(aStr) default: return false } @@ -521,25 +557,6 @@ func evaluateIn(actual, expected any) bool { return false } -var likeRegexCache sync.Map // pattern string → *regexp.Regexp - -// matchLike converts a SQL LIKE pattern to a regex and tests it -// (case-insensitive, matching the TS SDK). -func matchLike(actual, pattern string) bool { - if cached, ok := likeRegexCache.Load(pattern); ok { - return cached.(*regexp.Regexp).MatchString(actual) - } - escaped := regexp.QuoteMeta(pattern) - escaped = strings.ReplaceAll(escaped, "%", ".*") - escaped = strings.ReplaceAll(escaped, "_", ".") - re, err := regexp.Compile("(?i)^" + escaped + "$") - if err != nil { - return false - } - likeRegexCache.Store(pattern, re) - return re.MatchString(actual) -} - // compareOrdered returns (-1, 0, or 1) and true for comparable ordered types, // or (0, false) when the types cannot be compared. func compareOrdered(actual, expected any) (int, bool) { diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 18216aba..3721df23 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -245,7 +245,8 @@ func TestEvaluateFilter(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - if got := evaluateFilter(tt.actual, tt.op, tt.expected); got != tt.want { + cf := compileFilters([]QueryFilter{{Column: "c", Op: tt.op, Value: tt.expected}})[0] + if got := evaluateFilter(tt.actual, tt.op, tt.expected, cf.re); got != tt.want { t.Errorf("evaluateFilter(%v, %q, %v) = %v, want %v", tt.actual, tt.op, tt.expected, got, tt.want) } }) @@ -258,11 +259,11 @@ func TestMatchesFilters_AllMustMatch(t *testing.T) { {Column: "page", Op: "eq", Value: "/home"}, {Column: "score", Op: "gt", Value: 5}, } - if !matchesFilters(row, both) { + if !matchesFilters(row, compileFilters(both)) { t.Fatal("want match when every filter passes") } oneFails := append(append([]QueryFilter(nil), both...), QueryFilter{Column: "score", Op: "gt", Value: 99}) - if matchesFilters(row, oneFails) { + if matchesFilters(row, compileFilters(oneFails)) { t.Fatal("want no match when any filter fails") } if !matchesFilters(row, nil) { diff --git a/clients/go/table.go b/clients/go/table.go index 02bf4a0a..204a3b26 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -9,9 +9,9 @@ import ( "strings" ) -// TableRef is a reference to a table. Use it for queries, inserts, schema, and -// streams. NOT safe to use concurrently from multiple goroutines for mutations; -// reads (Fetch, Select, etc.) are safe. +// TableRef is a reference to a table. Use it for queries, inserts, schema, +// and streams. Safe for concurrent use: it holds no mutable state, and every +// builder method returns a fresh value. type TableRef struct { ctx httpContext table string diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 0cc2f4c3..24a640c5 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -35,26 +35,33 @@ every example on these pages assumes it. ## Quick Start ```go +package main + import ( "context" "fmt" + "log" wavehouse "github.com/Wave-RF/WaveHouse/clients/go" ) -wh := wavehouse.NewClient(wavehouse.Config{ - BaseURL: "http://localhost:8080", - Auth: wavehouse.StaticToken("your-jwt"), -}) - -page, err := wh.From("clicks"). - Select("page", "button"). - Where("page", wavehouse.OpEq, "/home"). - Limit(10). - FetchUntyped(context.Background()) -if err != nil { /* handle */ } -for _, row := range page.Data { - fmt.Println(row["page"], row["button"]) +func main() { + wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), + }) + + page, err := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(context.Background()) + if err != nil { + log.Fatal(err) + } + for _, row := range page.Data { + fmt.Println(row["page"], row["button"]) + } } ``` @@ -63,8 +70,6 @@ See the [README](https://github.com/Wave-RF/WaveHouse/blob/main/clients/go/READM ## Creating a Client ```go -import wavehouse "github.com/Wave-RF/WaveHouse/clients/go" - wh := wavehouse.NewClient(wavehouse.Config{ BaseURL: "https://wavehouse.example.com", Auth: func(ctx context.Context) (string, error) { @@ -91,6 +96,11 @@ wh := wavehouse.NewClient(wavehouse.Config{ |-------|------|---------|-------------| | `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, network failures) | +A `*Client` is safe for concurrent use by multiple goroutines — client state +is immutable after `NewClient`, and every builder chain copies. Supply a +concurrency-safe `Auth` func (it's called from any goroutine that issues a +request). + :::caution[`Options` opts you out of the default, not just in] The default of 2 retries only applies when `Config.Options` is `nil`. If you set `Options` to configure anything else in the future, an unset diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 6ef9380f..e790cdbb 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -136,9 +136,10 @@ page, err := clicks.Select("page", "button"). Start a query that selects **every column your role is allowed to read** — the explicit form of what `.Fetch()` does. Mutually exclusive with -`.Select(...)` and with aggregations (`.Count()`, `.Sum()`, etc.); the -server expands it to your allowed columns (never a raw `SELECT *`) and never -bypasses `deny_columns`/`allow_columns`. See +`.Select(...)` and with aggregations (`.Count()`, `.Sum()`, etc.); for a +column-restricted role the server expands it to exactly that role's allowed +columns rather than a bare `SELECT *` (unrestricted/admin roles do get +`SELECT *`), and it never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). ```go @@ -182,9 +183,10 @@ q := clicks.Select("page").Select("button") // SELECT page, button #### `.SelectAll()` -Select every column your role may read (the all-columns wildcard, expanded -server-side to your allowed columns). Mutually exclusive with `.Select(...)` -and with aggregations (`.Count()`, `.Sum()`, etc.). +Select every column your role may read (the all-columns wildcard; a +column-restricted role's projection is expanded server-side to its allowed +columns). Mutually exclusive with `.Select(...)` and with aggregations +(`.Count()`, `.Sum()`, etc.). ```go q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") @@ -345,7 +347,8 @@ rows decode into `map[string]any`, where JSON numbers become `float64`, so an integer cursor column loses exactness past 2^53 and pagination can repeat or skip a row at that scale — the same ceiling the TypeScript SDK has with JS numbers. `FetchTyped` with an `int64` field keeps the cursor exact, and -codegen structs are unaffected (their 64-bit integer columns are `string`). +codegen structs are unaffected (their 64-bit integer columns are `int64`/ +`uint64`, decoded exactly). ```go page, err := clicks.Select(). diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 271143b6..dd58c6bf 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -145,7 +145,7 @@ Generate Go structs from a running WaveHouse instance. The module ships a ```bash export WAVEHOUSE_AUTH= # avoids leaking the token via argv -go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen \ +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --url http://localhost:8080 \ --out ./db_types.go \ --package myapp @@ -215,23 +215,28 @@ value field with `omitempty` would silently drop. |------------------|---------| | `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Time`/`Time64`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | | `Bool` / `Boolean` | `bool` | -| `UInt8` / `UInt16` / `UInt32` | `uint8` / `uint16` / `uint32` | -| `Int8` / `Int16` / `Int32` | `int8` / `int16` / `int32` | +| `UInt8` / `UInt16` / `UInt32` / `UInt64` | `uint8` / `uint16` / `uint32` / `uint64` | +| `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | | `Float32`, `BFloat16` | `float32` | | `Float64` | `float64` | -| `UInt64`/`Int64`, `Decimal*`, `UInt128`/`UInt256`, `Int128`/`Int256` | `string` (ClickHouse quotes 64-bit-and-wider integers in JSON output — `output_format_json_quote_64bit_integers` — and the server forwards them verbatim) | +| `UInt128`/`UInt256`, `Int128`/`Int256` | `json.Number` (arbitrary-width unquoted numbers on the structured-query path) | +| `Decimal*` | `string` (marshaled as a quoted string on the structured-query path) | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | -| `Array(T)` | `[]T` | +| `Array(T)` | `[]T` (`Array(UInt8)` → `[]uint16`: `[]byte` would JSON-encode as base64, not an array) | | `Map(K, V)` | `map[K]V` (falls back to `map[string]any` if `K`/`V` can't be split) | | `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | This differs from the TypeScript SDK's mapping in one notable way: Go's -codegen preserves ClickHouse's integer **widths** up to 32 bits (`UInt32` → -`uint32`, not a generic `number`), since Go — unlike TypeScript — has -native fixed-width integer types. 64-bit integers stay `string` because -that is what actually arrives on the wire. +codegen preserves ClickHouse's integer **widths** (`UInt64` → `uint64`, not +a generic `number`), since Go — unlike TypeScript — has native fixed-width +integer types; 64-bit columns decode exactly where TS hits the JS-number +2^53 ceiling. Generated structs target the structured-query and pipe paths +(`/v1/query`, `/v1/pipes/*`), where the server re-marshals values as plain +JSON numbers. The raw-SQL path (`/v1/admin/query`) instead forwards +ClickHouse's own JSON, which **quotes** 64-bit-and-wider integers — use +`map[string]any` with `SQL[Row]` there rather than generated structs. ## Testing diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 90107898..093ff1ae 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -143,6 +143,13 @@ type StreamEvent struct { } ``` +:::note[`Events()` starts feeding on first call] +The channel only receives events emitted **after** the first `Events()` +call — a stream you set up but don't consume yet buffers nothing for the +channel. Call `Events()` immediately after `.Stream()` (or use +`.Subscribe`) if you can't start ranging right away. +::: + ### Transport Behavior | Transport | Reconnect | Protocol | diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index 7eead8ba..50e1e8e3 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -86,7 +86,7 @@ const { data } = await clicks.select('page', 'button').where('page', '=', '/home ### `.selectAll()` -Start a query that selects **every column your role is allowed to read** — the explicit form of what a bare `.fetch()` does. Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.); the server expands it to your allowed columns (never a raw `SELECT *`) and never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). +Start a query that selects **every column your role is allowed to read** — the explicit form of what a bare `.fetch()` does. Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.); for a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`), and it never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). ```ts const { data } = await clicks.selectAll().where('country', '=', 'US').limit(10); From 978451c24c9b4ad49b4a381159745be20992aeda Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 15:10:19 -0400 Subject: [PATCH 12/40] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=205=20=E2=80=94=20Array(UInt8)=20round-trip,=20SSE=20r?= =?UTF-8?q?esume=20test,=20docs=20accuracy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - codegen: Array(UInt8) maps to json.RawMessage — the wire is asymmetric (ingest requires a JSON array; /v1/query responses base64-encode the column because the server scans it into []byte and doesn't normalize it), so neither []uint8 nor the round-4 []uint16 could decode a query response; RawMessage round-trips both directions. Server-side normalization tracked in #436 - New reconnect-resume test pins the gap-fill contract: initial request carries StreamOptions.Since, the reconnect carries ?since= - docs: raw-SQL typed example decodes count() (UInt64, quoted on the admin path) via a `,string` tag instead of a plain int that fails; Insert's []byte rule stated the right way around; select_all "never a raw SELECT *" claim qualified for unrestricted/admin roles in api.md, access-control.mdx, architecture.md (matching the SDK pages); SDK health ping named in both languages in api.md, reverse-proxy.mdx, deployment.md; landing-page Go pointer no longer introduces TS-only tabs with a colon - make test-all runs test-conformance-ts, so "all suites" stays true --- Makefile | 1 + clients/go/cmd/wavehouse-codegen/main.go | 12 ++-- clients/go/cmd/wavehouse-codegen/main_test.go | 5 +- clients/go/stream_test.go | 58 +++++++++++++++++++ docs/src/content/docs/access-control.mdx | 2 +- docs/src/content/docs/api.md | 6 +- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/deployment.md | 2 +- docs/src/content/docs/index.mdx | 2 +- docs/src/content/docs/reverse-proxy.mdx | 2 +- docs/src/content/docs/sdk/go/queries.md | 14 +++-- docs/src/content/docs/sdk/go/reference.md | 2 +- 12 files changed, 88 insertions(+), 20 deletions(-) diff --git a/Makefile b/Makefile index 243ddcfa..f468fff6 100644 --- a/Makefile +++ b/Makefile @@ -756,6 +756,7 @@ test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVE test-all: ## Run all suites sequentially + one consolidated Go + TS coverage report + gates @$(MAKE) test-unit COV_DEFER=1 @$(MAKE) test-go-sdk + @$(MAKE) test-conformance-ts @$(MAKE) test-ts COV_DEFER=1 @$(MAKE) test-integration COV_DEFER=1 @$(MAKE) test-e2e COV_DEFER=1 diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index f46998c1..8861860e 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -215,11 +215,13 @@ func chTypeToGo(chType string) string { // Array. if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { inner := chTypeToGo(chType[6 : len(chType)-1]) - // []uint8 is []byte, which encoding/json base64-encodes as a string — - // the server requires a real JSON array for Array(...) columns, so - // widen the element type instead. + // Array(UInt8) is asymmetric on the wire: ingest requires a real JSON + // array, but /v1/query responses currently base64-encode it (the + // server scans into []byte and encoding/json base64s that — #436). + // json.RawMessage is the + // only shape that round-trips both directions without a decode error. if inner == "uint8" { - inner = "uint16" + return "json.RawMessage" } return "[]" + inner } @@ -298,7 +300,7 @@ func generate(schemas map[string]tableSchema, pkg string) (string, error) { needsJSON := false for _, name := range names { for _, col := range schemas[name].Columns { - if strings.Contains(chTypeToGo(col.Type), "json.Number") { + if strings.Contains(chTypeToGo(col.Type), "json.") { needsJSON = true } } diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go index 47b2ce3e..f6207ddc 100644 --- a/clients/go/cmd/wavehouse-codegen/main_test.go +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -44,8 +44,9 @@ func TestChTypeToGo(t *testing.T) { {"SimpleAggregateFunction(any)", "any"}, {"Array(String)", "[]string"}, {"Array(Nullable(Int32))", "[]*int32"}, - // []uint8 is []byte → base64 on marshal; must widen. - {"Array(UInt8)", "[]uint16"}, + // []uint8 is []byte → base64 on marshal; RawMessage round-trips both + // the ingest array form and the (currently base64) query response. + {"Array(UInt8)", "json.RawMessage"}, {"Map(String, UInt32)", "map[string]uint32"}, {"Map(String, Map(UInt32, String))", "map[string]map[uint32]string"}, {"Tuple(String, UInt8)", "any"}, diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 3721df23..21be46f2 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -7,6 +7,7 @@ import ( "io" "net/http" "net/http/httptest" + "sync" "sync/atomic" "testing" "time" @@ -341,3 +342,60 @@ func TestStream_NonRetryableConnectErrorIsTerminal(t *testing.T) { t.Fatal("Connected must fail on a terminally-closed stream") } } + +// TestStream_ReconnectResumesFromLastEventID: the gap-fill contract. The +// initial request carries StreamOptions.Since; after the connection drops, +// the reconnect carries ?since=. +func TestStream_ReconnectResumesFromLastEventID(t *testing.T) { + var mu sync.Mutex + var sinceParams []string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + sinceParams = append(sinceParams, r.URL.Query().Get("since")) + n := len(sinceParams) + mu.Unlock() + + w.Header().Set("Content-Type", "text/event-stream") + fl := w.(http.Flusher) + w.WriteHeader(200) + fl.Flush() + if n == 1 { + _, _ = io.WriteString(w, sseFrame("2026-01-01T00:00:01Z", "/home")) + fl.Flush() + return // server closes → client must reconnect with since= + } + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + + stream := streamClient(t, srv).From("clicks").Stream(&StreamOptions{Since: "seed-id"}) + defer stream.Close() + + got := make(chan StreamEvent, 4) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { got <- e }}) + recvEvent(t, got) + + // Reconnect happens after ~backoff(0) (≈1s with jitter). + deadline := time.Now().Add(10 * time.Second) + for { + mu.Lock() + n := len(sinceParams) + mu.Unlock() + if n >= 2 { + break + } + if time.Now().After(deadline) { + t.Fatal("stream never reconnected") + } + time.Sleep(20 * time.Millisecond) + } + + mu.Lock() + defer mu.Unlock() + if sinceParams[0] != "seed-id" { + t.Fatalf("initial request: want since=seed-id, got %q", sinceParams[0]) + } + if sinceParams[1] != "2026-01-01T00:00:01Z" { + t.Fatalf("reconnect: want since=, got %q", sinceParams[1]) + } +} diff --git a/docs/src/content/docs/access-control.mdx b/docs/src/content/docs/access-control.mdx index 69c21cb7..8a25cd1e 100644 --- a/docs/src/content/docs/access-control.mdx +++ b/docs/src/content/docs/access-control.mdx @@ -174,7 +174,7 @@ The rules, in order: 2. **An empty (or `["*"]`) `allow_columns` means "all columns"** — every column not in `deny_columns` is permitted. Use this with `deny_columns` for a blocklist posture: see everything *except* a few sensitive columns. 3. **A non-empty `allow_columns` is an allowlist** — only the named columns (and never the denied ones) are permitted. -On a structured query (`POST /v1/query?table={table}`) the allowlist is a **hard cap on every column the query references — in any clause**: the projection, an aggregation argument, `filters`, `group_by`, `order_by`, and `time_range`. Naming a disallowed column anywhere is rejected with `403 column "x" not allowed`. A full-row read is requested explicitly with `"select_all": true`, which expands to exactly the columns the role may read — never a raw `SELECT *` that could include a denied column; if the role is allowed *no* columns, the read is rejected (`403`) rather than returning empty rows. **Omitting `columns` (or sending `[]` / `""`) returns nothing** — a request for no data — so a hidden column can't leak by being left out, grouped on, or filtered on to infer its values. (Note: in a query, `["*"]` is the *literal column named `*`*, not a wildcard — use `select_all` for all columns. In `allow_columns`, `["*"]` is still the all-columns wildcard.) On insert (`POST /v1/ingest?table={table}`) the body is `403 column "x" not allowed for insert`. On live streams, denied columns are silently **stripped** from each event rather than rejecting the connection. The structured-query and live-stream paths defer to the **same** per-column decision (`IsColumnAllowed`), so the two read surfaces enforce identical column visibility and can't drift apart. +On a structured query (`POST /v1/query?table={table}`) the allowlist is a **hard cap on every column the query references — in any clause**: the projection, an aggregation argument, `filters`, `group_by`, `order_by`, and `time_range`. Naming a disallowed column anywhere is rejected with `403 column "x" not allowed`. A full-row read is requested explicitly with `"select_all": true`; for a column-restricted role it expands to exactly the columns the role may read rather than a bare `SELECT *` that could include a denied column (an unrestricted/admin role does get `SELECT *`); if the role is allowed *no* columns, the read is rejected (`403`) rather than returning empty rows. **Omitting `columns` (or sending `[]` / `""`) returns nothing** — a request for no data — so a hidden column can't leak by being left out, grouped on, or filtered on to infer its values. (Note: in a query, `["*"]` is the *literal column named `*`*, not a wildcard — use `select_all` for all columns. In `allow_columns`, `["*"]` is still the all-columns wildcard.) On insert (`POST /v1/ingest?table={table}`) the body is `403 column "x" not allowed for insert`. On live streams, denied columns are silently **stripped** from each event rather than rejecting the connection. The structured-query and live-stream paths defer to the **same** per-column decision (`IsColumnAllowed`), so the two read surfaces enforce identical column visibility and can't drift apart. ## Row-level security diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 3302b5af..91aa3e72 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -150,7 +150,7 @@ Status code: `503 Service Unavailable` Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. -This is what the SDK's `wh.sys.health()` calls, and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. +This is what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`), and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. --- @@ -416,7 +416,7 @@ curl -X POST http://localhost:8080/v1/admin/query \ Executes a type-safe structured query against a table. The query AST is validated against the schema and converted to parameterized SQL. Permissions from the access control policy are enforced (column filtering, row-level security, aggregation restrictions). :::note[The column allowlist is a hard cap on every clause] -Every column the query references — in `columns`, an aggregation argument, `filters`, `group_by`, `order_by`, or `time_range` — must be permitted by the role's `allow_columns`/`deny_columns`, or the request is rejected with `403 column "x" not allowed`. A full-row read is requested with `"select_all": true` (expanded to the columns the role may read — never a raw `SELECT *`); **omitting `columns` returns nothing**, so a hidden column never leaks by being left out, grouped on, or filtered on. See [Access control → Column permissions](/access-control#column-permissions). +Every column the query references — in `columns`, an aggregation argument, `filters`, `group_by`, `order_by`, or `time_range` — must be permitted by the role's `allow_columns`/`deny_columns`, or the request is rejected with `403 column "x" not allowed`. A full-row read is requested with `"select_all": true` (for a column-restricted role, expanded to exactly the columns the role may read rather than a bare `SELECT *`; unrestricted/admin roles do get `SELECT *`); **omitting `columns` returns nothing**, so a hidden column never leaks by being left out, grouped on, or filtered on. See [Access control → Column permissions](/access-control#column-permissions). ::: **Request:** @@ -444,7 +444,7 @@ Every column the query references — in `columns`, an aggregation argument, `fi | Field | Type | Required | Description | | ----- | ---- | -------- | ----------- | | `columns` | string \| string[] | No | Columns to SELECT — an array, or a single string for one column. A literal `"*"` is the column *named* `*`, **not** a wildcard. Omit (or send `[]` / `""`) to select nothing; use `select_all` for a full-row read. Mutually exclusive with `select_all`. | -| `select_all` | bool | No | Select every column the role may read (the all-columns wildcard, expanded server-side to the allow/deny set). Mutually exclusive with a non-empty `columns`, and with `aggregations`. | +| `select_all` | bool | No | Select every column the role may read (the all-columns wildcard; a column-restricted role's projection is expanded server-side to its allow/deny set, an unrestricted/admin role gets `SELECT *`). Mutually exclusive with a non-empty `columns`, and with `aggregations`. | | `aggregations` | object[] | No | Aggregation functions (`fn`, `column`, `alias`). | | `filters` | object[] | No | WHERE conditions (`column`, `op`, `value`). Ops: eq, neq, gt, gte, lt, lte, in, like. | | `group_by` | string[] | No | GROUP BY columns. | diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 8b8b6ff2..35eefd5f 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -149,7 +149,7 @@ The package's design invariants — stdout always 100%, WARN+ERROR always export ### `query/` — Structured Query Engine - **ast.go** — `StructuredQuery` AST types: columns, aggregations, filters, group by, order by, limit, time range. -- **builder.go** — `Build()` converts AST to parameterized SQL. It is the single chokepoint that validates every referenced identifier against the schema **and** authorizes every column reference — projection, aggregation args, filters, group_by, order_by, time_range — against the role's column allowlist (the [#223](https://github.com/Wave-RF/WaveHouse/issues/223) hard cap). A full-row read is requested with `select_all`, which expands to the role's allowed columns rather than emitting a raw `SELECT *`; an omitted projection selects nothing, and `*` in `columns` is a literal column name. Every identifier is backtick-quoted via `internal/chsql` (`QuoteIdent`) so any ClickHouse-legal name is accepted — a name containing `?` is refused fail-closed ([#279](https://github.com/Wave-RF/WaveHouse/issues/279)). `InjectPermissionFilters()` adds row-level security. `ApplyMaxRows()` enforces limits. Timestamp bucketing for cache optimization. +- **builder.go** — `Build()` converts AST to parameterized SQL. It is the single chokepoint that validates every referenced identifier against the schema **and** authorizes every column reference — projection, aggregation args, filters, group_by, order_by, time_range — against the role's column allowlist (the [#223](https://github.com/Wave-RF/WaveHouse/issues/223) hard cap). A full-row read is requested with `select_all`, which for a column-restricted role expands to the role's allowed columns rather than emitting a raw `SELECT *` (unrestricted/admin roles get `SELECT *`); an omitted projection selects nothing, and `*` in `columns` is a literal column name. Every identifier is backtick-quoted via `internal/chsql` (`QuoteIdent`) so any ClickHouse-legal name is accepted — a name containing `?` is refused fail-closed ([#279](https://github.com/Wave-RF/WaveHouse/issues/279)). `InjectPermissionFilters()` adds row-level security. `ApplyMaxRows()` enforces limits. Timestamp bucketing for cache optimization. ### `chsql/` — ClickHouse SQL Helpers diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 93e30086..bb72452a 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -264,7 +264,7 @@ API servers in standalone mode expose liveness and readiness endpoints under the Configure your load balancer or orchestrator to use these endpoints. -**Exposure.** Probes share the API server's port (`:8080`) — kubelet probes the container internally, so there's no separate-port convention for them (metrics are the signal that optionally gets its own `prometheus.port`). If you forward `:8080` to the public internet the probe paths become reachable. The **recommended** posture is to keep `/livez`/`/readyz`/`/healthz` to internal callers and expose only **`/v1/health`** publicly (the SDK's content-free liveness ping, which never touches ClickHouse). `/readyz` issues a ClickHouse `Ping` on every call, so a public `/readyz` lets an unauthenticated flood become per-request backend pings, and the bare probes leak boot/readiness state — keeping them internal is a [reverse-proxy/ingress concern](/reverse-proxy#health-probes), and your orchestrator reaches them the internal way (kubelet on the container, LB on the backend) regardless. +**Exposure.** Probes share the API server's port (`:8080`) — kubelet probes the container internally, so there's no separate-port convention for them (metrics are the signal that optionally gets its own `prometheus.port`). If you forward `:8080` to the public internet the probe paths become reachable. The **recommended** posture is to keep `/livez`/`/readyz`/`/healthz` to internal callers and expose only **`/v1/health`** publicly (the SDKs' content-free liveness ping — `wh.sys.health()` / `wh.Sys.Health(ctx)` — which never touches ClickHouse). `/readyz` issues a ClickHouse `Ping` on every call, so a public `/readyz` lets an unauthenticated flood become per-request backend pings, and the bare probes leak boot/readiness state — keeping them internal is a [reverse-proxy/ingress concern](/reverse-proxy#health-probes), and your orchestrator reaches them the internal way (kubelet on the container, LB on the backend) regardless. ### Boot-time degraded mode diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index 9308a38e..aaffef84 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -104,7 +104,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click ## Query it like a database. Subscribe to it like a socket. -The zero-dependency [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming. Writing Go? The official [Go SDK](/sdk/go) mirrors the same feature set: +The zero-dependency [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming (examples below are TypeScript). Writing Go? The official [Go SDK](/sdk/go) mirrors the same feature set — see the [Go quick start](/sdk/go#quick-start). diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index 34900c0e..e462e4ea 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -156,7 +156,7 @@ WaveHouse serves Kubernetes-convention probes on `:8080` (full behavior in [Depl - **`/v1/health`** — the SDK's content-free liveness ping; mirrors `/livez` (200 once booted) and never touches ClickHouse. :::caution[Recommended: keep the bare probe paths off the public vhost] -Expose only **`/v1/health`** (and your API) to the internet — route `/livez`, `/readyz`, and `/healthz` on a private/internal listener, not the public proxy. Your orchestrator still reaches them the internal way regardless: kubelet probes the container directly on `:8080`, and a load balancer health-checks the backend — neither goes through the public proxy. Two reasons to keep them internal: `/readyz` pings ClickHouse on every call, so a public `/readyz` lets an unauthenticated flood turn into a per-request backend ping; and the probes leak boot/readiness state. `/v1/health` is the safe public liveness endpoint because it answers the same "is this server up" question without touching ClickHouse — it's what the SDK's `wh.sys.health()` calls. +Expose only **`/v1/health`** (and your API) to the internet — route `/livez`, `/readyz`, and `/healthz` on a private/internal listener, not the public proxy. Your orchestrator still reaches them the internal way regardless: kubelet probes the container directly on `:8080`, and a load balancer health-checks the backend — neither goes through the public proxy. Two reasons to keep them internal: `/readyz` pings ClickHouse on every call, so a public `/readyz` lets an unauthenticated flood turn into a per-request backend ping; and the probes leak boot/readiness state. `/v1/health` is the safe public liveness endpoint because it answers the same "is this server up" question without touching ClickHouse — it's what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`). ::: ## Timeouts and slow links diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index e790cdbb..9b88fd77 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -51,8 +51,11 @@ see [Pagination](#pagination). Insert one row or many. What you pass determines the wire format: -- A single **map or struct** (anything that isn't a slice, and isn't - `[]byte`) is sent as JSON: `POST /v1/ingest?table={table}`. +- A single **map or struct** (anything that isn't a slice — plus `[]byte`, + which is treated as one opaque value rather than a batch of numbers, and + would reach the server as a base64 string it rejects; pass raw NDJSON + through `.InsertNDJSON(ctx, string(raw))` instead) is sent as JSON: + `POST /v1/ingest?table={table}`. - **Any slice** — `[]map[string]any`, a generated/user-defined row type like `[]ClickRow`, etc. — is serialized to NDJSON (one record per line, via reflection for non-`[]map[string]any` slices) and sent as a single @@ -385,10 +388,13 @@ authorizes `/v1/admin/*` without a JWT. Package-level generic function — use rows, err := wavehouse.SQL[map[string]any](ctx, wh, "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") -// Or decode into a struct that matches the projected columns/aliases: +// Or decode into a struct that matches the projected columns/aliases. +// NOTE: this path forwards ClickHouse's own JSON, which QUOTES 64-bit +// integers (count() is UInt64) — decode them with the `,string` tag, or +// use map[string]any. See Reference → Codegen CLI for the full story. type PageTotal struct { Page string `json:"page"` - Total int `json:"total"` + Total uint64 `json:"total,string"` } typed, err := wavehouse.SQL[PageTotal](ctx, wh, "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index dd58c6bf..0074b7f0 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -223,7 +223,7 @@ value field with `omitempty` would silently drop. | `Decimal*` | `string` (marshaled as a quoted string on the structured-query path) | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | -| `Array(T)` | `[]T` (`Array(UInt8)` → `[]uint16`: `[]byte` would JSON-encode as base64, not an array) | +| `Array(T)` | `[]T` — except `Array(UInt8)` → `json.RawMessage`: the wire is asymmetric (ingest takes a JSON array, but query responses currently base64-encode the column), and `RawMessage` is the one shape that decodes both; server-side normalization tracked in [#436](https://github.com/Wave-RF/WaveHouse/issues/436) | | `Map(K, V)` | `map[K]V` (falls back to `map[string]any` if `K`/`V` can't be split) | | `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | From 3a2640144ad7cf4be8dd898955bb13eafb78b846 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 15:26:04 -0400 Subject: [PATCH 13/40] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=206=20=E2=80=94=20e2e=20container=20gate,=20module=20f?= =?UTF-8?q?loor,=20test/docs=20precision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - e2e buildMarkerRow: container types (Array/Map/Tuple/Nested) are gated before the substring cases — "array(string)" contains "string", so the skip branch was unreachable and a scalar marker went into array columns - TestStream_HandleMalformedSSEData sets chanRequested so its no-emit assertion is real instead of vacuously passing on the opt-in guard - clients/go/go.mod floor lowered 1.26.5 → 1.24: the SDK needs nothing past Go 1.22 (range-over-int, math/rand/v2) and a patch-pinned floor exists for the server's stdlib CVEs, not for a zero-dep library consumers go get - development.md/CONTRIBUTING.md: gotestsum/ARGS/V=1/race claims scoped to the instrumented suites; nested-module gofumpt path documented; SDK test locations added - queries.md: operator-key note explains Config.Auth always sends Bearer — use a custom HTTPClient transport for X-Operator-Key --- CONTRIBUTING.md | 4 ++-- clients/go/e2e_test.go | 12 +++++++++--- clients/go/go.mod | 5 ++++- clients/go/stream_test.go | 4 +++- docs/src/content/docs/development.md | 9 +++++---- docs/src/content/docs/sdk/go/index.md | 2 +- docs/src/content/docs/sdk/go/queries.md | 7 +++++-- 7 files changed, 29 insertions(+), 14 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1b459bd7..d831fcff 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -39,7 +39,7 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t The pre-push hook (installed by `make tools`) blocks a push until the tree has been validated locally: a code change needs `make ci`, a docs/prose-only change needs only `make verify` (the same split CI makes). `make lint` / `make test` / `make build` are fast inner-loop subsets. -2. Write tests for new functionality. Unit tests go alongside the code in `internal/`. Integration tests go in `tests/` with the `//go:build integration` tag. +2. Write tests for new functionality. Unit tests go alongside the code in `internal/`; SDK tests live in `clients/ts/src/` and `clients/go/`. Integration tests go in `tests/` with the `//go:build integration` tag. 3. Update documentation if your change affects: - API endpoints → update `docs/src/content/docs/api.md` @@ -90,7 +90,7 @@ test(cache): add tiered cache stampede test ## Code Style -- **Formatting**: Code must be formatted with `gofumpt` (a strict superset of `gofmt`). `make fmt` checks it (CI runs the same target); `make fix` applies it. +- **Formatting**: Code must be formatted with `gofumpt` (a strict superset of `gofmt`). `make fmt` checks the root module; the nested `clients/go` module is checked by `make verify` (its `verify-go-sdk` leaf, which the pre-commit hook and CI run). `make fix` applies gofumpt to both. - **Linting**: All lint checks in `.golangci.yml` must pass (see `make lint`). - **Naming**: Follow [Go naming conventions](https://go.dev/doc/effective_go#names). - **Interfaces**: Define interfaces where they are consumed, not where they are implemented. diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go index 23f59221..3bce0341 100644 --- a/clients/go/e2e_test.go +++ b/clients/go/e2e_test.go @@ -372,6 +372,12 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) (map[string]any, st } ct := strings.ToLower(col.Type) switch { + case strings.Contains(ct, "array(") || strings.Contains(ct, "map(") || + strings.Contains(ct, "tuple(") || strings.Contains(ct, "nested("): + // Container types must be gated BEFORE the substring cases below — + // "array(string)" contains "string" and would otherwise get a + // scalar marker injected into an array column. + t.Skipf("e2e: table %q requires container column %q of type %q", ts.Name, col.Name, col.Type) case !markerSet && strings.Contains(ct, "string"): row[col.Name] = mk markerCol = col.Name @@ -387,9 +393,9 @@ func buildMarkerRow(t *testing.T, ts TableSchema, mk string) (map[string]any, st case strings.Contains(ct, "bool"): row[col.Name] = false default: - // No safe synthetic value for this type (Array, Map, Tuple, UUID, - // ...) — an empty string would make the insert fail with a type - // error that looks like an SDK defect. + // No safe synthetic value for this type (UUID, IPv6, ...) — an + // empty string would make the insert fail with a type error that + // looks like an SDK defect. t.Skipf("e2e: table %q requires column %q of unsupported type %q", ts.Name, col.Name, col.Type) } } diff --git a/clients/go/go.mod b/clients/go/go.mod index 84e065de..b78eaa50 100644 --- a/clients/go/go.mod +++ b/clients/go/go.mod @@ -1,3 +1,6 @@ module github.com/Wave-RF/WaveHouse/clients/go -go 1.26.5 +// Library floor, deliberately lower than the server's pinned toolchain: +// the newest things this module uses are range-over-int and math/rand/v2 +// (Go 1.22). Keep it a supported-releases floor, not a patch pin. +go 1.24 diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 21be46f2..bcf782dd 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -201,7 +201,9 @@ func TestStream_FilteredCloseUnderLoad(t *testing.T) { } func TestStream_HandleMalformedSSEData(t *testing.T) { - sc := &StreamController{eventCh: make(chan StreamEvent, 1)} + // chanRequested must be true or emitEvent skips the channel entirely and + // the no-emit assertion below would pass vacuously. + sc := &StreamController{eventCh: make(chan StreamEvent, 1), chanRequested: true} sc.handleSSEData("not json", "id1") // must not panic or emit select { case e := <-sc.eventCh: diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index ea642152..d3b2c7fc 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -288,9 +288,9 @@ go build -o bin/wavehouse ./cmd/wavehouse ### How It Works -All test commands use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. +The coverage-instrumented suite targets (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary, and accept `ARGS`/`V=1`. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. -All tests run with Go's **race detector** (`-race`) enabled by default. WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. +Go tests run with the **race detector** (`-race`) enabled by default (including `test-go-sdk` — the SDK's streaming subsystem is highly concurrent; `test-go-sdk-e2e` skips it since it drives a live server). WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. ### Quick Reference @@ -300,7 +300,8 @@ All tests run with Go's **race detector** (`-race`) enabled by default. WaveHous # Unit tests + Go SDK tests (compact output) — alias for `test-unit` + `test-go-sdk` make test -# Run specific test(s) +# Run specific root-module test(s) — ARGS reaches test-unit only; the +# test-go-sdk half of `make test` runs its full suite regardless make test ARGS="-run TestValidate" # Go integration tests (requires Docker) @@ -518,7 +519,7 @@ Run `make help` to see all targets. Key ones: | `make clean-tools` | Installed tools and pnpm deps (`.bin/`, `node_modules/`) | | `make clean-all` | Full reset: above + `data/` + Docker volumes | -All test targets accept `ARGS="..."` for pass-through `go test` flags. Build targets accept `TAGS="..."` for Go build tags. `V=1` switches to verbose `gotestsum` output. +The gotestsum-driven targets (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) accept `ARGS="..."` for pass-through `go test` flags and `V=1` for verbose output; `test-go-sdk`, `test-go-sdk-e2e`, and `test-conformance-ts` ignore both. Build targets accept `TAGS="..."` for Go build tags. ## Dependency Management diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 24a640c5..c292bc5c 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -20,7 +20,7 @@ either page mostly carries over. go get github.com/Wave-RF/WaveHouse/clients/go ``` -Requires Go 1.26.5 or later (the minimum pinned in the module's `go.mod`). +Requires Go 1.24 or later (the module's `go.mod` floor — deliberately a supported-releases floor rather than the server's patch-pinned toolchain). ## Import diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 9b88fd77..e1572c9f 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -381,8 +381,11 @@ the token must resolve to the policy admin role (`admin_role`, `"admin"` by default) — a JWT request with no token, or an invalid/expired one, falls back to the `default_role` and is rejected. Alternatively, a configured operator key (`Authorization: Operator ` or `X-Operator-Key`) -authorizes `/v1/admin/*` without a JWT. Package-level generic function — use -`map[string]any` for a dynamic/unknown schema. +authorizes `/v1/admin/*` without a JWT — but note `Config.Auth` always +sends its token as `Bearer `, so to use an operator key from this +SDK supply a `Config.HTTPClient` whose `Transport` sets the +`X-Operator-Key` header on each request. Package-level generic function — +use `map[string]any` for a dynamic/unknown schema. ```go rows, err := wavehouse.SQL[map[string]any](ctx, wh, From 10cb253800496e0963f625b12f7a41316e59a011 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 15:42:36 -0400 Subject: [PATCH 14/40] test(sdk): guard conformance request capture with a mutex The handler-goroutine write / test-goroutine read pattern was mutex-guarded everywhere else in the package after review; conformance_test.go was the one holdout. No happens-before edge exists via the TCP socket, so this was a latent race -race hadn't tripped yet. --- clients/go/conformance_test.go | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index e41c7b09..c30bb6ba 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -10,6 +10,7 @@ import ( "net/url" "reflect" "strings" + "sync" "testing" ) @@ -75,13 +76,18 @@ func TestConformance_WireFormat(t *testing.T) { for _, tc := range cases { t.Run(tc.Name, func(t *testing.T) { + // capt is written on the server goroutine and read on the test + // goroutine; the mutex is what makes that visible under -race. + var mu sync.Mutex var capt captured srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() capt.method = r.Method capt.path = r.URL.RequestURI() capt.contentType = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) capt.body = string(raw) + mu.Unlock() // Return valid JSON so the SDK doesn't error on decode. w.Header().Set("Content-Type", "application/json") @@ -214,6 +220,8 @@ func TestConformance_WireFormat(t *testing.T) { } // Verify method. + mu.Lock() + defer mu.Unlock() if tc.ExpectedMethod != "" && capt.method != tc.ExpectedMethod { t.Errorf("method: want %s, got %s", tc.ExpectedMethod, capt.method) } From cade9d7de6830b7422cf9e373a8d4b098a93a807 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 15:47:02 -0400 Subject: [PATCH 15/40] =?UTF-8?q?docs(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=207=20=E2=80=94=20channel=20error=20visibility,=20agg?= =?UTF-8?q?=20allowlist,=20titles?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - streaming.md: Events() carries events only — Error/Status arrive solely via Subscribe, so a channel-only consumer sees the range end silently on a terminal 401/403/404; documented with the pairing recommendation - Both SDK query pages document the server's aggregation-function allowlist (isValidAggFn) instead of implying any custom fn works - TS reference gains the SSE_CONNECT_ERROR row (auth provider threw / invalid baseURL — the more reachable of the two stream codes) - "the SDK's" → "the SDKs'" in reverse-proxy.mdx and architecture.md, matching the sweep already applied to api.md/deployment.md - Five TS topic pages retitled "TypeScript SDK …" now that two SDK doc trees sit side by side in search and tabs - query_builder.go comment no longer claims codegen 64-bit columns are strings (they're int64/uint64; 128/256-bit are json.Number) --- clients/go/query_builder.go | 4 ++-- docs/src/content/docs/architecture.md | 2 +- docs/src/content/docs/reverse-proxy.mdx | 2 +- docs/src/content/docs/sdk/admin.md | 2 +- docs/src/content/docs/sdk/go/queries.md | 9 ++++++++- docs/src/content/docs/sdk/go/streaming.md | 8 ++++++++ docs/src/content/docs/sdk/pipes.md | 2 +- docs/src/content/docs/sdk/queries.md | 11 +++++++++-- docs/src/content/docs/sdk/reference.md | 3 ++- docs/src/content/docs/sdk/streaming.md | 2 +- 10 files changed, 34 insertions(+), 11 deletions(-) diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 07c16783..a8453e16 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -281,8 +281,8 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro // protection: its rows were already decoded to float64 by // encoding/json, so precision above 2^53 is gone before we get here — // the same ceiling the TS SDK has with JS numbers. Use FetchTyped (or - // codegen structs, whose 64-bit int columns are strings) when paging - // on >2^53 integer cursors. + // codegen structs — their 64-bit int columns are int64/uint64, and + // 128/256-bit are json.Number) when paging on >2^53 integer cursors. raw, _ := json.Marshal(lastRow) m = make(map[string]any) dec := json.NewDecoder(bytes.NewReader(raw)) diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index 35eefd5f..617e26b9 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -79,7 +79,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request - **stream.go** — Real-time streaming via SSE. Callers select a table with the `?table=` query parameter. Each connection registers one `Subscriber` (the `stream/` package) with both the event `Hub` (under its `(topic, role)`) and the shared keepalive wheel, then drains both from a single byte-pump — so idle streams keep emitting `:` keepalive comments (surviving reverse-proxy idle timeouts) while live events arrive already projected and serialized. Per-event projection/serialization happens **once per role** in the `Hub`, not once per subscriber ([#294](https://github.com/Wave-RF/WaveHouse/issues/294)). Gap-fill replay from NATS JetStream (`DeliverByStartTime`) stays per-connection (low-volume, one-time on connect). - **schema.go** — Schema discovery API: list all schemas, get one table, trigger refresh. - **dlq.go** — DLQ stats endpoint and `EnsureDLQStream` helper for creating the `WAVEHOUSE_DLQ` NATS stream. -- **health.go** — Liveness (`/livez`), readiness (`/readyz`), and a content-free `Online` ping (`/v1/health`, the SDK's public liveness check); `/healthz` is a permanent alias of `/livez`, and `/health`/`/ready` are deprecated aliases. All three consult an optional `BootState` so they can return 503 while boot-time schema discovery is still failing in the retry loop (see `cmd/wavehouse/main.go`); once `BootState.Set(nil)` fires, `/livez` returns 200 and stays there. `/readyz` additionally pings ClickHouse each call; `/v1/health` deliberately does not. +- **health.go** — Liveness (`/livez`), readiness (`/readyz`), and a content-free `Online` ping (`/v1/health`, the SDKs' public liveness check); `/healthz` is a permanent alias of `/livez`, and `/health`/`/ready` are deprecated aliases. All three consult an optional `BootState` so they can return 503 while boot-time schema discovery is still failing in the retry loop (see `cmd/wavehouse/main.go`); once `BootState.Set(nil)` fires, `/livez` returns 200 and stays there. `/readyz` additionally pings ClickHouse each call; `/v1/health` deliberately does not. ### `stream/` — SSE keepalive & fan-out diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index e462e4ea..d2e647fc 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -153,7 +153,7 @@ WaveHouse serves Kubernetes-convention probes on `:8080` (full behavior in [Depl - **`/livez`** — liveness; sticky-200 after first successful boot. Does not touch ClickHouse. - **`/readyz`** — readiness; issues a ClickHouse `Ping` on **every** call. Point your load balancer's (internal) health check here so it routes around an instance whose ClickHouse is unreachable. - **`/healthz`** — permanent alias of `/livez`. -- **`/v1/health`** — the SDK's content-free liveness ping; mirrors `/livez` (200 once booted) and never touches ClickHouse. +- **`/v1/health`** — the SDKs' content-free liveness ping; mirrors `/livez` (200 once booted) and never touches ClickHouse. :::caution[Recommended: keep the bare probe paths off the public vhost] Expose only **`/v1/health`** (and your API) to the internet — route `/livez`, `/readyz`, and `/healthz` on a private/internal listener, not the public proxy. Your orchestrator still reaches them the internal way regardless: kubelet probes the container directly on `:8080`, and a load balancer health-checks the backend — neither goes through the public proxy. Two reasons to keep them internal: `/readyz` pings ClickHouse on every call, so a public `/readyz` lets an unauthenticated flood turn into a per-request backend ping; and the probes leak boot/readiness state. `/v1/health` is the safe public liveness endpoint because it answers the same "is this server up" question without touching ClickHouse — it's what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`). diff --git a/docs/src/content/docs/sdk/admin.md b/docs/src/content/docs/sdk/admin.md index 2a4971b5..453e7163 100644 --- a/docs/src/content/docs/sdk/admin.md +++ b/docs/src/content/docs/sdk/admin.md @@ -1,5 +1,5 @@ --- -title: "SDK Admin & System" +title: "TypeScript SDK Admin & System" description: "Schema introspection, access-control policy, DLQ stats, and health checks in @wavehouse/sdk." --- diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index e1572c9f..a7ff744c 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -227,9 +227,16 @@ clicks.Select("page"). Min("score", "min_score"). // MIN(score) Max("score", "max_score"). // MAX(score) CountDistinct("page", "unique_pages"). - Aggregate("uniqExact", "user_id", "unique_users") // custom fn + Aggregate("uniqExact", "user_id", "unique_users") // allowlisted fn ``` +Custom function names pass through `.Aggregate(fn, column, alias)` but are +validated server-side against a fixed allowlist (matched case-insensitively): +`count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, +`any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, +`stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Anything else is rejected +with `400 unsupported aggregation function`. + `Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias string)`; `Aggregate` takes `(fn, column, alias string)`. Empty-alias defaults: `Count` → `count` (and `column=""` becomes `*`); `Sum`/`Avg`/ diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 093ff1ae..f491795b 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -143,6 +143,14 @@ type StreamEvent struct { } ``` +:::note[`Events()` carries events only] +`Error` and `Status` are delivered exclusively through `.Subscribe(...)` — +the channel is typed `chan StreamEvent` and simply ends (closes) when the +stream closes, including on a terminal 401/403/404. Pair `Events()` with a +`Subscribe(&StreamSubscriber{Error: ..., Status: ...})` if you need to know +*why* a stream ended. +::: + :::note[`Events()` starts feeding on first call] The channel only receives events emitted **after** the first `Events()` call — a stream you set up but don't consume yet buffers nothing for the diff --git a/docs/src/content/docs/sdk/pipes.md b/docs/src/content/docs/sdk/pipes.md index 3e1cca2f..86c0d763 100644 --- a/docs/src/content/docs/sdk/pipes.md +++ b/docs/src/content/docs/sdk/pipes.md @@ -1,5 +1,5 @@ --- -title: "SDK Pipes" +title: "TypeScript SDK Pipes" description: "Execute and manage named query pipes with @wavehouse/sdk." --- diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index 50e1e8e3..a75ee3e5 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -1,5 +1,5 @@ --- -title: "SDK Queries" +title: "TypeScript SDK Queries" description: "Tables, the chainable query builder, pagination, and raw SQL in @wavehouse/sdk." --- @@ -162,9 +162,16 @@ clicks.select('page') .min('score', 'min_score') // MIN(score) .max('score', 'max_score') // MAX(score) .countDistinct('page', 'unique_pages') - .aggregate('uniqExact', 'user_id', 'unique_users') // custom fn + .aggregate('uniqExact', 'user_id', 'unique_users') // allowlisted fn ``` +Custom function names pass through `.aggregate(fn, column, alias)` but are +validated server-side against a fixed allowlist (matched case-insensitively): +`count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, +`any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, +`stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Anything else is rejected +with `400 unsupported aggregation function`. + Each aggregation method signature: `(column: string, alias?: string)`. `count()` defaults to `column='*'`, `alias='count'`. diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index 5d54ca4b..cef0e82d 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -1,5 +1,5 @@ --- -title: "SDK Reference & CLI" +title: "TypeScript SDK Reference & CLI" description: "Error codes, AbortController, the full API tree, the codegen CLI, and E2E testing with @wavehouse/sdk." --- @@ -38,6 +38,7 @@ The SDK **never throws**. All errors are returned in `Result.error`. | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `AbortSignal` | | 0 | `SSE_ERROR` | Yes | Stream connection failure, delivered to the stream's error callback; the stream reconnects automatically | +| 0 | `SSE_CONNECT_ERROR` | Yes | Stream could not be opened (auth provider threw, invalid `baseURL`) | --- diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index 2a9868a5..d0297878 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -1,5 +1,5 @@ --- -title: "SDK Streaming & Live Queries" +title: "TypeScript SDK Streaming & Live Queries" description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in @wavehouse/sdk." --- From 791c198b71fab92f2b71f12ab2dd40a582e05264 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 16:03:09 -0400 Subject: [PATCH 16/40] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=208=20=E2=80=94=20dead=20param,=20error=20wrapping,=20?= =?UTF-8?q?doc=20accuracy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Code (pre-push-reviewer): - Drop unused limit param from fetchNextTyped; FetchTyped recomputes it from the cloned builder's state. - Wrap errors in Schema.List/Refresh and DLQ.stats with operation context, matching every other namespace. Docs (docs-reviewer): - development.md: make verify row no longer claims tidy/vulncheck cover clients/go (nested module — tracked in #437). - sdk/go/index.md: caution on timeouts — context deadlines, not http.Client.Timeout (which kills SSE streams). - sdk/go/queries.md: OpNotLike does emit not_like on the wire. - api.md: pluralize three leftover singular-SDK references. - clients/go/README.md: drop trailing slashes (trailingSlash: "never"). --- clients/go/README.md | 4 ++-- clients/go/dlq.go | 3 ++- clients/go/query_builder.go | 4 ++-- clients/go/schema.go | 14 ++++++++++---- docs/src/content/docs/api.md | 6 +++--- docs/src/content/docs/development.md | 2 +- docs/src/content/docs/sdk/go/index.md | 11 ++++++++++- docs/src/content/docs/sdk/go/queries.md | 2 +- 8 files changed, 31 insertions(+), 15 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index cf2c1348..bd6109d3 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -4,7 +4,7 @@ Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a s **Zero third-party runtime dependencies** — stdlib only. -**[Full SDK documentation on wavehouse.dev](https://wavehouse.dev/sdk/go/)** +**[Full SDK documentation on wavehouse.dev](https://wavehouse.dev/sdk/go)** ## Install @@ -200,7 +200,7 @@ go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --package myapp ``` -See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference/#codegen-cli). +See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference#codegen-cli). ## Error Handling diff --git a/clients/go/dlq.go b/clients/go/dlq.go index e833a41a..0009e4e3 100644 --- a/clients/go/dlq.go +++ b/clients/go/dlq.go @@ -2,6 +2,7 @@ package wavehouse import ( "context" + "fmt" "net/url" ) @@ -28,7 +29,7 @@ func (d *DLQNamespace) stats(ctx context.Context, params url.Values) (*DLQStats, path: "/v1/dlq/stats", params: params, }, &stats); err != nil { - return nil, err + return nil, fmt.Errorf("get dlq stats: %w", err) } return &stats, nil } diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index a8453e16..06092bbc 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -179,7 +179,7 @@ func FetchTyped[Row any](ctx context.Context, q *QueryBuilder) (*Page[Row], erro // SDK's QueryBuilder.fetch()/_fetchNext(), which has the same limitation). if hasMore && len(q.state.orderBy) > 0 { page.Next = func(ctx context.Context) (*Page[Row], error) { - return fetchNextTyped[Row](ctx, q, rows, limit) + return fetchNextTyped[Row](ctx, q, rows) } } @@ -265,7 +265,7 @@ func (q *QueryBuilder) buildAST(effectiveLimit int) *StructuredQuery { return ast } -func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Row, limit int) (*Page[Row], error) { +func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Row) (*Page[Row], error) { if len(q.state.orderBy) == 0 { return &Page[Row]{}, nil } diff --git a/clients/go/schema.go b/clients/go/schema.go index 3c7f4307..3f5b4a88 100644 --- a/clients/go/schema.go +++ b/clients/go/schema.go @@ -1,6 +1,9 @@ package wavehouse -import "context" +import ( + "context" + "fmt" +) // SchemaNamespace provides admin-only schema introspection. type SchemaNamespace struct { @@ -15,7 +18,7 @@ func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { method: "GET", path: "/v1/schema", }, &raw); err != nil { - return nil, err + return nil, fmt.Errorf("list schemas: %w", err) } schemas := make(Schemas, len(raw)) for _, t := range raw { @@ -26,8 +29,11 @@ func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { // Refresh forces a schema re-discovery from ClickHouse. Admin-only. func (s *SchemaNamespace) Refresh(ctx context.Context) error { - return doRequest(ctx, s.ctx, requestOptions{ + if err := doRequest(ctx, s.ctx, requestOptions{ method: "POST", path: "/v1/schema/refresh", - }, nil) + }, nil); err != nil { + return fmt.Errorf("refresh schema: %w", err) + } + return nil } diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index 91aa3e72..16254300 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -150,7 +150,7 @@ Status code: `503 Service Unavailable` Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. -This is what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`), and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. +This is what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`), and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDKs rely on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. --- @@ -254,7 +254,7 @@ curl -X POST "http://localhost:8080/v1/ingest?table=clicks" \ #### Batch Ingest -A **JSON array** of objects (`[{…}, {…}]`) or an **NDJSON** body (`Content-Type: application/x-ndjson`, one JSON object per line) ingests a batch in a single request. Each record is validated, authorized, deduplicated, and published independently, so **one malformed or rejected record never blocks the rest of the batch**. (The SDK's `insert([...])` array helper uses the NDJSON form automatically; both forms return the same response.) +A **JSON array** of objects (`[{…}, {…}]`) or an **NDJSON** body (`Content-Type: application/x-ndjson`, one JSON object per line) ingests a batch in a single request. Each record is validated, authorized, deduplicated, and published independently, so **one malformed or rejected record never blocks the rest of the batch**. (Both SDKs' array/slice insert helpers use the NDJSON form automatically; both forms return the same response.) - **JSON array** — the most convenient form from most HTTP clients. A structural JSON syntax error fails the whole request (`400`), but a wrong-typed element (a non-object) is reported per-record like any other rejection. An explicit empty array (`[]`) is a valid, record-less batch (`200`, `total: 0`). - **NDJSON** — the streaming-friendly form for very large uploads. Blank lines are skipped, and a single malformed *line* is reported and skipped (the newline reframes the next record). @@ -316,7 +316,7 @@ A `200` is returned whenever the body was read and the records were processed | 503 | `{"error":"service unavailable"}` | NATS JetStream full (backpressure) mid-batch; includes `Retry-After: 30` | :::caution[At-least-once on retry] -A batch aborted partway (a `503`/`500`, or a JSON-array syntax error, after some leading records were already published) re-publishes those leading records when the whole batch is retried. Enable deduplication if duplicate suppression matters — this is the same at-least-once property the single-object path already has (the SDK retries both on `503`). +A batch aborted partway (a `503`/`500`, or a JSON-array syntax error, after some leading records were already published) re-publishes those leading records when the whole batch is retried. Enable deduplication if duplicate suppression matters — this is the same at-least-once property the single-object path already has (the SDKs retry both on `503`). ::: **curl example (JSON array):** diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index d3b2c7fc..cd9c6d46 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -488,7 +488,7 @@ Run `make help` to see all targets. Key ones: | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | | `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) | | `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | -| `make verify` | Repo-wide static checks: Go incl. `clients/go` (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | +| `make verify` | Repo-wide static checks: root Go (tidy + fmt + vulncheck + lint), `clients/go` (fmt + vet + lint — no tidy/vulncheck: it's a nested module, invisible to the root-scoped `tidy`/`vulncheck` targets) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | | `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`) and TS (Biome `--write`) | | **Build** | | | `make build` | Compile `wavehouse` → `bin/wavehouse` (debug symbols kept) | diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index c292bc5c..8e0b2329 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -88,7 +88,16 @@ wh := wavehouse.NewClient(wavehouse.Config{ | `BaseURL` | `string` | — | WaveHouse server URL (required) | | `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider, called before each request. `nil` means unauthenticated access | | `Options` | `*ClientOptions` | `nil` | Transport tuning (see below) | -| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports | +| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports (see caution below) | + +:::caution[Timeouts: use contexts, not `http.Client.Timeout`] +The default client sets no `Timeout` — a `context.Context` deadline is the +only bound on a request, so pass one for anything that mustn't hang on a +stalled server. If you supply your own `HTTPClient`, leave `Timeout` unset: +it covers body reads too, so it would kill every long-lived SSE stream at +the timeout and force a reconnect loop. Use `Transport`-level dial / +TLS / response-header timeouts instead. +::: ### `ClientOptions` diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index a7ff744c..894f05a1 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -215,7 +215,7 @@ clicks.Select("page"). | `wavehouse.OpLte` | `lte` | Less than or equal | | `wavehouse.OpIn` | `in` | Value in array — accepts a Go slice of any element type (`[]string`, `[]int`, `[]any`, ...) | | `wavehouse.OpLike` | `like` | SQL LIKE pattern | -| `wavehouse.OpNotLike` | — | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | +| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects the token | #### Aggregations From 478b215d54cb49bc4c9ead39ea0eeb540e631e6a Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 16:18:45 -0400 Subject: [PATCH 17/40] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=209=20=E2=80=94=20SSE=20error=20delivery,=20test=20rac?= =?UTF-8?q?es,=20parity=20gate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Code (pre-push-reviewer): - handleSSEData delivers malformed-frame errors via the subscriber Error callback instead of unbounded process-global logging; drop its unused eventID param. - table_test.go: mutex-guard all five handler-goroutine captures, matching the package-wide convention from 10cb253. - conformance_test.go: unhandled fixture endpoint is now t.Fatalf, matching the TS runner's skipped-cases-break-parity stance. - Convert ponytail: comment prefixes to TODO:/plain notes (repo uses TODO). - Move test-only errIs helper into http_test.go. Docs (docs-reviewer): - Both streaming pages: live-query dedup caution — projection must include received_timestamp or overlap-window events deliver twice; Go page also corrects the bound to max-across-rows (desc order puts oldest last). --- clients/go/conformance_test.go | 4 +++- clients/go/http.go | 10 -------- clients/go/http_test.go | 10 ++++++++ clients/go/query_builder.go | 6 ++--- clients/go/stream.go | 16 +++++++------ clients/go/stream_test.go | 8 ++++++- clients/go/table_test.go | 28 +++++++++++++++++++++++ docs/src/content/docs/sdk/go/streaming.md | 12 ++++++++-- docs/src/content/docs/sdk/streaming.md | 4 ++++ 9 files changed, 74 insertions(+), 24 deletions(-) diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index c30bb6ba..26fbac07 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -216,7 +216,9 @@ func TestConformance_WireFormat(t *testing.T) { logCallErr(t, c.Pipes.Delete(ctx, tc.PipeName)) default: - t.Skipf("unhandled endpoint: %s", tc.Endpoint) + // Hard failure, matching the TS runner: skipped cases break + // cross-SDK parity. + t.Fatalf("unhandled endpoint %q — wire it up in the dispatch switch", tc.Endpoint) } // Verify method. diff --git a/clients/go/http.go b/clients/go/http.go index 0fc59d9e..74dcbdad 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -4,7 +4,6 @@ import ( "bytes" "context" "encoding/json" - "errors" "fmt" "io" "math" @@ -207,12 +206,3 @@ func sleepWithContext(ctx context.Context, d time.Duration) error { return ctx.Err() } } - -// errIs checks if err wraps a *Error with the given code. -func errIs(err error, code string) bool { - var e *Error - if errors.As(err, &e) { - return e.Code == code - } - return false -} diff --git a/clients/go/http_test.go b/clients/go/http_test.go index aef9aa0f..df9045ff 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -3,6 +3,7 @@ package wavehouse import ( "context" "encoding/json" + "errors" "io" "net/http" "net/http/httptest" @@ -11,6 +12,15 @@ import ( "time" ) +// errIs checks if err wraps a *Error with the given code. +func errIs(err error, code string) bool { + var e *Error + if errors.As(err, &e) { + return e.Code == code + } + return false +} + func testCtx(t *testing.T, handler http.Handler) httpContext { t.Helper() srv := httptest.NewServer(handler) diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 06092bbc..6991371f 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -22,7 +22,7 @@ type queryState struct { orderBy []OrderClause limit *int timeRange *TimeRange - cacheTTL *int // ponytail: client-side only, not sent to server (#280) + cacheTTL *int // client-side only, not sent to server (#280) } // QueryBuilder builds structured queries. Immutable — every chain method @@ -179,7 +179,7 @@ func FetchTyped[Row any](ctx context.Context, q *QueryBuilder) (*Page[Row], erro // SDK's QueryBuilder.fetch()/_fetchNext(), which has the same limitation). if hasMore && len(q.state.orderBy) > 0 { page.Next = func(ctx context.Context) (*Page[Row], error) { - return fetchNextTyped[Row](ctx, q, rows) + return fetchNextTyped(ctx, q, rows) } } @@ -275,7 +275,7 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro lastRow := any(prevRows[len(prevRows)-1]) m, ok := lastRow.(map[string]any) if !ok { - // ponytail: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. + // TODO: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. // UseNumber keeps typed int64 cursor values exact past 2^53. The // untyped path (FetchUntyped / TableRef.Fetch) doesn't get this // protection: its rows were already decoded to float64 by diff --git a/clients/go/stream.go b/clients/go/stream.go index 6da259be..d1c4c206 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -22,7 +22,7 @@ type StreamController struct { mu sync.Mutex status StreamStatus subscribers []*StreamSubscriber - eventCh chan StreamEvent // ponytail: single buffered channel for Go-native consumption + eventCh chan StreamEvent // single buffered channel for Go-native consumption chanRequested bool // set by Events(); until then emitEvent skips the channel dropLogOnce sync.Once cancel context.CancelFunc @@ -103,7 +103,7 @@ func (sc *StreamController) Connected(ctx context.Context) error { sc.mu.Unlock() // Poll — simple and correct. - // ponytail: condition variable if polling shows up in profiles. + // TODO: switch to a condition variable if polling shows up in profiles. ticker := time.NewTicker(50 * time.Millisecond) defer ticker.Stop() for { @@ -338,7 +338,7 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if line == "" { // Empty line = end of event frame. if dataLine != "" { - sc.handleSSEData(dataLine, eventID) + sc.handleSSEData(dataLine) // Track last event ID for reconnect gap-fill. if eventID != "" { lastID = eventID @@ -376,12 +376,14 @@ type sseMessage struct { Data map[string]any `json:"data"` } -func (sc *StreamController) handleSSEData(data, eventID string) { +func (sc *StreamController) handleSSEData(data string) { var msg sseMessage if err := json.Unmarshal([]byte(data), &msg); err != nil { - // Deliberately omits the payload: event data can carry tenant/PII - // fields and this goes to the process-global logger. - log.Printf("[wavehouse] SSE received malformed message (%d bytes): %v", len(data), err) + // Delivered via the subscriber Error callback rather than the + // process-global logger, so consumers control visibility and a + // malformed-frame flood can't spam host-application logs. Payload + // deliberately omitted: event data can carry tenant/PII fields. + sc.emitError(fmt.Errorf("malformed SSE message (%d bytes): %w", len(data), err)) return } diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index bcf782dd..c201fa2f 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -7,6 +7,7 @@ import ( "io" "net/http" "net/http/httptest" + "strings" "sync" "sync/atomic" "testing" @@ -204,12 +205,17 @@ func TestStream_HandleMalformedSSEData(t *testing.T) { // chanRequested must be true or emitEvent skips the channel entirely and // the no-emit assertion below would pass vacuously. sc := &StreamController{eventCh: make(chan StreamEvent, 1), chanRequested: true} - sc.handleSSEData("not json", "id1") // must not panic or emit + var gotErr error + sc.subscribers = []*StreamSubscriber{{Error: func(err error) { gotErr = err }}} + sc.handleSSEData("not json") // must not panic or emit select { case e := <-sc.eventCh: t.Fatalf("malformed data emitted event: %+v", e) default: } + if gotErr == nil || !strings.Contains(gotErr.Error(), "malformed SSE message") { + t.Fatalf("want malformed-SSE error via subscriber, got %v", gotErr) + } } // --------------------------------------------------------------------------- diff --git a/clients/go/table_test.go b/clients/go/table_test.go index 5116d53f..e935657f 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -6,13 +6,19 @@ import ( "io" "net/http" "net/http/httptest" + "sync" "testing" ) func TestTableRef_InsertSingle(t *testing.T) { + // mu guards handler captures throughout this file: the handler runs on the + // server goroutine and no happens-before edge exists via the TCP socket. + var mu sync.Mutex var gotBody map[string]any var gotPath string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() gotPath = r.URL.Path _ = json.NewDecoder(r.Body).Decode(&gotBody) _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) @@ -26,6 +32,8 @@ func TestTableRef_InsertSingle(t *testing.T) { if !result.OK { t.Fatal("want ok=true") } + mu.Lock() + defer mu.Unlock() if gotPath != "/v1/ingest" { t.Fatalf("want /v1/ingest, got %s", gotPath) } @@ -35,9 +43,12 @@ func TestTableRef_InsertSingle(t *testing.T) { } func TestTableRef_InsertBatch(t *testing.T) { + var mu sync.Mutex var gotCT string var gotBody string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() gotCT = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) gotBody = string(raw) @@ -57,6 +68,8 @@ func TestTableRef_InsertBatch(t *testing.T) { if !result.OK { t.Fatal("want ok=true") } + mu.Lock() + defer mu.Unlock() if gotCT != "application/x-ndjson" { t.Fatalf("want ndjson content type, got %s", gotCT) } @@ -75,9 +88,12 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { Page string `json:"page"` } + var mu sync.Mutex var gotCT string var gotBody string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() gotCT = r.Header.Get("Content-Type") raw, _ := io.ReadAll(r.Body) gotBody = string(raw) @@ -94,6 +110,8 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { if err != nil { t.Fatal(err) } + mu.Lock() + defer mu.Unlock() if gotCT != "application/x-ndjson" { t.Fatalf("want ndjson content type, got %s", gotCT) } @@ -114,8 +132,11 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { // TestTableRef_InsertByteSliceNotBatch ensures []byte keeps going through // insertSingle rather than being (mis)treated as a slice of per-byte rows. func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { + var mu sync.Mutex var gotPath string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() gotPath = r.URL.Path _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) @@ -128,6 +149,8 @@ func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { if !result.OK { t.Fatal("want ok=true") } + mu.Lock() + defer mu.Unlock() if gotPath != "/v1/ingest" { t.Fatalf("want /v1/ingest, got %s", gotPath) } @@ -152,8 +175,11 @@ func TestTableRef_InsertEmptyBatch(t *testing.T) { } func TestTableRef_InsertNDJSON(t *testing.T) { + var mu sync.Mutex var gotBody string srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() raw, _ := io.ReadAll(r.Body) gotBody = string(raw) _ = json.NewEncoder(w).Encode(map[string]any{ @@ -170,6 +196,8 @@ func TestTableRef_InsertNDJSON(t *testing.T) { if result.Total == nil || *result.Total != 2 { t.Fatalf("want total=2, got %v", result.Total) } + mu.Lock() + defer mu.Unlock() if gotBody != ndjson { t.Fatalf("want raw NDJSON, got %s", gotBody) } diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index f491795b..011c3c6e 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -257,14 +257,22 @@ Decode into your own type inside the callback if you need one. 1. Subscribes to the stream **immediately** and buffers incoming events. 2. Runs the `.FetchUntyped(ctx)` query for historical data, calls `sub.Initial(rows, err)` with the result. -3. Deduplicates buffered events by comparing timestamps against the latest - historical row's `received_timestamp`. +3. Deduplicates buffered events against the **newest** `received_timestamp` + in the backfill — the maximum across all rows, not the last row's (an + `OrderBy(..., "desc")` puts the *oldest* row last). 4. Flushes remaining buffered events (re-checking for anything that arrived mid-flush) and switches to live mode. This "stream-first" approach ensures no events are lost between the fetch and stream start. +:::caution[Dedup needs `received_timestamp` in the projection] +The dedup bound comes from the backfill rows' `received_timestamp` values. +`.SelectAll()` (or no projection) includes it; a `.Select(...)` projection +that omits it disables dedup, and events in the fetch/stream overlap window +are delivered twice — once in `Initial`, again via `Next`. +::: + ### `.Close()` Shuts down the live query and its underlying stream. Safe to call more than diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index d0297878..7bc4b7fb 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -158,3 +158,7 @@ interface StreamSubscriber { 4. Flushes remaining buffered events and switches to live mode. This "stream-first" approach ensures no events are lost between the fetch and stream start. + +:::caution[Dedup needs `received_timestamp` in the projection] +The dedup boundary comes from the fetched rows' `received_timestamp` values. A `.select(...)` projection that omits that column disables dedup, and events in the fetch/stream overlap window are delivered twice — once in `initial()`, again via `next()`. +::: From a841f1a8cb464411e0859601bf856a576f87aa23 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 16:34:37 -0400 Subject: [PATCH 18/40] =?UTF-8?q?fix(sdk):=20address=20pre-push=20review?= =?UTF-8?q?=20round=2010=20=E2=80=94=20terminal=20StatusClosed,=20doc=20pr?= =?UTF-8?q?ecision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - stream.go: StatusClosed is terminal in setStatus; a filtered wrapper's stale inner Status callback can land after Close and must not resurrect the status. - conformance_ts.mjs: comment no longer claims the Go harness skips unhandled endpoints (it hard-fails since round 9). - sdk/go/queries.md: scope the ctx/(T, error) claim to request-response operations, matching the sibling pages. - sdk/go/streaming.md: note the one-time drop log line; document that Auth provider errors during (re)connect retry forever (MaxRetries bounds request retries only). PR #434 body refreshed separately (stale vulncheck note, commit list, test counts). --- clients/go/stream.go | 5 ++++- docs/src/content/docs/sdk/go/queries.md | 5 +++-- docs/src/content/docs/sdk/go/streaming.md | 11 ++++++++--- tests/conformance/conformance_ts.mjs | 5 +++-- 4 files changed, 18 insertions(+), 8 deletions(-) diff --git a/clients/go/stream.go b/clients/go/stream.go index d1c4c206..3bab2ac2 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -144,7 +144,10 @@ func (sc *StreamController) Close() { func (sc *StreamController) setStatus(s StreamStatus) { sc.mu.Lock() - if s == sc.status { + // StatusClosed is terminal: a filtered wrapper's inner controller can + // have copied its subscriber slice before unsub, so a stale Status + // callback may land after Close — it must not resurrect the status. + if s == sc.status || sc.status == StatusClosed { sc.mu.Unlock() return } diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 894f05a1..d63c6715 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -5,8 +5,9 @@ description: "Tables, the chainable query builder, pagination, and raw SQL in th Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: table references, the chainable query builder, cursor pagination, and the -admin-only raw-SQL escape hatch. Every call takes a `context.Context` as its -first argument and returns `(T, error)` — see +admin-only raw-SQL escape hatch. Every request-response operation takes a +`context.Context` as its first argument and returns `(T, error)`; the +chainable builder methods and `.Stream(opts)` are the exceptions — see [Error Handling](/sdk/go#error-handling). Compare with the TypeScript SDK's [Queries](/sdk/queries) page, which covers the same surface with a `Result`-returning, `PromiseLike` builder. diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 011c3c6e..30c46a3c 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -85,8 +85,9 @@ exit path) regardless of which consumption style you use. The channel is buffered (256 events); a slow consumer that never drains it causes the SDK to **drop** new events for that channel rather than block the -stream's read loop (`.Subscribe` callbacks still fire per event -regardless of channel backpressure). +stream's read loop — the first drop logs one line via the standard `log` +package, further drops are silent (`.Subscribe` callbacks still fire per +event regardless of channel backpressure). ### `.Close()` @@ -168,7 +169,11 @@ Reconnect covers transport failures and retryable (5xx) responses. A non-retryable response (401/403/404) is terminal: the error is delivered to the subscriber's `Error` callback, status goes to `StatusClosed`, and the stream does not reconnect — fix the cause (refresh the token, correct the -table) and open a new stream. +table) and open a new stream. An `Auth` provider error during a (re)connect +is treated as retryable (`SSE_ERROR`) and the stream keeps reconnecting — +`ClientOptions.MaxRetries` bounds request retries only, not stream +reconnects — so call `.Close()` if your token provider is failing +permanently. Auth is sent as an `Authorization: Bearer` header on every stream (re)connection — see diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index d430bcfe..4996b594 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -233,8 +233,9 @@ for (const tc of cases) { await wh.pipes.delete(tc.pipe_name); break; default: - // Not a pass — the Go harness skips these too. Fixture cases with a - // new endpoint value must be wired up here before they count. + // Not a pass — the Go harness hard-fails on these; we count and exit + // non-zero below. Fixture cases with a new endpoint value must be + // wired up here before they count. skipped++; skippedNames.push(`${tc.name} (endpoint: ${tc.endpoint})`); continue; From 6a4e189ed60fbb784760870ed3d9263648469e71 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 20:45:52 -0400 Subject: [PATCH 19/40] =?UTF-8?q?fix(sdk):=20address=20verified=20Codex=20?= =?UTF-8?q?review=20findings=20=E2=80=94=20Events()=20buffering,=20filter?= =?UTF-8?q?=20kinds,=20make=20fix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit External Codex review of PR #434 produced 8 findings; each was verified against source and git history before acting. Fixed here (4): - stream.go: Events() channel buffers from construction again, restoring TS parity (controller.ts buffers unconditionally). The chanRequested opt-in gate (round 1) silently discarded events emitted before the first Events() call; the dropLogOnce once-only drop log it was added alongside is kept. New test pins the buffering behavior. - stream.go: toFloat64 handles all int/uint kinds; a codegen-typed narrow/unsigned operand in an ordered .Where() filter silently dropped matching live events. Test table extended. - stream.go: scanner-cap comment cited the wrong bound — the binding per-event ceiling is the embedded NATS 1 MiB MaxPayload, not the 16 MiB HTTP ingest cap. - types.go: MaxRowsToRead *int -> *int64, matching the server's wire type (internal/policy: int64, can exceed 2^31). - Makefile: fix-go ran gofumpt/goimports via `go tool` inside the nested module, which has no tool directives — `make fix` failed with "go: no such tool" (regression from round 1). Formatters now run from the root module against clients/go; tidy + golangci-lint keep their nested cd. Remaining findings triaged to issues (drafted separately): failed-backfill buffer flush (Go+TS), LIKE case semantics (Go+TS+server), Nullable+DEFAULT explicit-null collapse (codegen). --- Makefile | 2 +- clients/go/stream.go | 58 +++++++++++++++-------- clients/go/stream_test.go | 31 ++++++++++-- clients/go/types.go | 2 +- docs/src/content/docs/sdk/go/streaming.md | 12 +++-- 5 files changed, 74 insertions(+), 31 deletions(-) diff --git a/Makefile b/Makefile index f468fff6..2985b92b 100644 --- a/Makefile +++ b/Makefile @@ -452,7 +452,7 @@ fix-go: $(GOLANGCI_LINT) @$(GOIMPORTS) -w $(GO_DIRS) @$(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners @echo "$(CYAN)==> Applying Go auto-fixes (Go SDK — nested module, outside GO_DIRS)...$(RESET)" - @cd clients/go && go mod tidy && $(GOFUMPT) -w . && $(GOIMPORTS) -w . && $(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners + @$(GOFUMPT) -w clients/go && $(GOIMPORTS) -w clients/go && cd clients/go && go mod tidy && $(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners .PHONY: fix-ts fix-ts: pnpm-install diff --git a/clients/go/stream.go b/clients/go/stream.go index 3bab2ac2..b6402489 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -19,15 +19,14 @@ import ( // StreamController manages a live SSE event stream. Use Subscribe for // callback-based consumption or Events for channel-based consumption. type StreamController struct { - mu sync.Mutex - status StreamStatus - subscribers []*StreamSubscriber - eventCh chan StreamEvent // single buffered channel for Go-native consumption - chanRequested bool // set by Events(); until then emitEvent skips the channel - dropLogOnce sync.Once - cancel context.CancelFunc - done chan struct{} - closed bool + mu sync.Mutex + status StreamStatus + subscribers []*StreamSubscriber + eventCh chan StreamEvent // single buffered channel for Go-native consumption + dropLogOnce sync.Once + cancel context.CancelFunc + done chan struct{} + closed bool } // newStreamController opens an SSE connection for the given table. @@ -78,13 +77,12 @@ func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { } // Events returns a read-only channel that receives stream events. -// The channel is closed when the stream closes. Events are fed to the -// channel only from the first Events() call onward — a Subscribe-only -// consumer never fills (and overflows) a channel it isn't reading. +// The channel is closed when the stream closes. Events buffer into it from +// stream construction (matching the TS SDK), so events that arrive before +// the first Events() call are not lost. A Subscribe-only consumer that never +// calls Events() at most fills the 256-slot buffer and trips the one-time +// drop log. func (sc *StreamController) Events() <-chan StreamEvent { - sc.mu.Lock() - sc.chanRequested = true - sc.mu.Unlock() return sc.eventCh } @@ -173,12 +171,13 @@ func (sc *StreamController) emitEvent(event StreamEvent) { } } - // Non-blocking send to the channel. Guarded by mu so the send and - // closeEventCh serialize — a late event can never hit a closed channel — - // and skipped entirely until Events() opts in. + // Non-blocking send to the channel, which buffers from construction (TS + // parity) so events emitted before the first Events() call survive. + // Guarded by mu so the send and closeEventCh serialize — a late event can + // never hit a closed channel. sc.mu.Lock() defer sc.mu.Unlock() - if sc.closed || !sc.chanRequested { + if sc.closed { return } select { @@ -327,7 +326,10 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table // Parse SSE frames. scanner := bufio.NewScanner(resp.Body) - scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) // 16 MiB max, matching server ingest cap + // 16 MiB max line: generous headroom over the ~1 MiB NATS MaxPayload + // ceiling on a single event envelope (oversized records are rejected at + // ingest publish and never reach the stream). + scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) var eventID, dataLine string lastID := since @@ -600,8 +602,24 @@ func toFloat64(v any) (float64, bool) { return float64(n), true case int: return float64(n), true + case int8: + return float64(n), true + case int16: + return float64(n), true + case int32: + return float64(n), true case int64: return float64(n), true + case uint: + return float64(n), true + case uint8: + return float64(n), true + case uint16: + return float64(n), true + case uint32: + return float64(n), true + case uint64: + return float64(n), true case json.Number: f, err := n.Float64() return f, err == nil diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index c201fa2f..1d9a9f69 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -2,6 +2,7 @@ package wavehouse import ( "context" + "encoding/json" "errors" "fmt" "io" @@ -202,9 +203,7 @@ func TestStream_FilteredCloseUnderLoad(t *testing.T) { } func TestStream_HandleMalformedSSEData(t *testing.T) { - // chanRequested must be true or emitEvent skips the channel entirely and - // the no-emit assertion below would pass vacuously. - sc := &StreamController{eventCh: make(chan StreamEvent, 1), chanRequested: true} + sc := &StreamController{eventCh: make(chan StreamEvent, 1)} var gotErr error sc.subscribers = []*StreamSubscriber{{Error: func(err error) { gotErr = err }}} sc.handleSSEData("not json") // must not panic or emit @@ -218,6 +217,23 @@ func TestStream_HandleMalformedSSEData(t *testing.T) { } } +// TestStream_EventsBufferBeforeFirstEventsCall pins TS parity: the channel +// buffers from construction, so events emitted before the first Events() call +// are still delivered once the consumer starts reading. +func TestStream_EventsBufferBeforeFirstEventsCall(t *testing.T) { + sc := &StreamController{eventCh: make(chan StreamEvent, 256)} + sc.emitEvent(StreamEvent{Table: "clicks", Data: map[string]any{"page": "/"}}) + + select { + case e := <-sc.Events(): + if e.Table != "clicks" { + t.Fatalf("want event for clicks, got %+v", e) + } + default: + t.Fatal("event emitted before Events() was not buffered") + } +} + // --------------------------------------------------------------------------- // Client-side filter engine // --------------------------------------------------------------------------- @@ -240,6 +256,8 @@ func TestEvaluateFilter(t *testing.T) { {"Lt", float64(9), "lt", 10, true}, {"LteString", "a", "lte", "b", true}, {"GtIncomparable", "a", "gt", 10, false}, + // Narrow/unsigned codegen-struct fields must compare, not silently drop. + {"GtUnsignedOperand", float64(10), "gt", uint32(5), true}, {"InAnySlice", "b", "in", []any{"a", "b"}, true}, {"InTypedSlice", float64(2), "in", []int{1, 2}, true}, {"InMiss", "c", "in", []any{"a", "b"}, false}, @@ -293,7 +311,12 @@ func TestCompareOrdered(t *testing.T) { } func TestToFloat64(t *testing.T) { - for _, v := range []any{float64(1), float32(1), int(1), int64(1)} { + for _, v := range []any{ + float64(1), float32(1), + int(1), int8(1), int16(1), int32(1), int64(1), + uint(1), uint8(1), uint16(1), uint32(1), uint64(1), + json.Number("1"), + } { if f, ok := toFloat64(v); !ok || f != 1 { t.Fatalf("toFloat64(%T) = (%v, %v)", v, f, ok) } diff --git a/clients/go/types.go b/clients/go/types.go index 0d17699f..1257e446 100644 --- a/clients/go/types.go +++ b/clients/go/types.go @@ -175,7 +175,7 @@ type RolePermissions struct { DeniedAggregations []string `json:"denied_aggregations,omitempty"` MaxRows *int `json:"max_rows,omitempty"` MaxExecutionTime any `json:"max_execution_time,omitempty"` - MaxRowsToRead *int `json:"max_rows_to_read,omitempty"` + MaxRowsToRead *int64 `json:"max_rows_to_read,omitempty"` MaxMemoryUsage any `json:"max_memory_usage,omitempty"` } diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 30c46a3c..70f44a37 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -152,11 +152,13 @@ stream closes, including on a terminal 401/403/404. Pair `Events()` with a *why* a stream ended. ::: -:::note[`Events()` starts feeding on first call] -The channel only receives events emitted **after** the first `Events()` -call — a stream you set up but don't consume yet buffers nothing for the -channel. Call `Events()` immediately after `.Stream()` (or use -`.Subscribe`) if you can't start ranging right away. +:::note[The channel buffers from stream construction] +Events buffer into the channel (up to 256) from the moment `.Stream()` +constructs the stream, matching the TypeScript SDK — events arriving before +your first `Events()` call are **not** lost, so you don't have to call +`Events()` immediately. A consumer that never drains the channel still +drops everything past the 256th buffered event (with the one-time log line +described above). ::: ### Transport Behavior From c88d1b1072599720f2a8227ca317da05fa02ebc4 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 7 Aug 2026 21:12:24 -0400 Subject: [PATCH 20/40] fix(sdk): apply verified cavecrew/ponytail wave findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Correctness (cavecrew, all verified): - http.go: cap backoff after jitter so the documented 30s max holds (was cap-then-jitter, allowing 36s). - stream.go: filtered wrapper unsubscribes from the inner controller in the inner-closed branch too, not just on wrapper Close. - codegen: a flag missing its value now errors (exit 2) instead of silently using the default; flagValue helper replaces four copies. - conformance: ingest/ingest_batch hard-fail on fixture cases without an insert operation (both runners); toStringMap fails on non-object rows. - wire_cases.json: all 23 query cases now assert expected_path and expected_method (previously body-only). Complexity (ponytail, safe cuts only): - stream.go: snapshotSubs helper replaces two copy-pasted subscriber snapshots (setStatus keeps its inline copy — its snapshot must share the critical section with the status write); evaluateIn drops the redundant []any special case (reflect covers it); toFloat64 collapses ten integer cases to reflect CanInt/CanUint. - wavehouse.go: inline single-caller trimTrailingSlashes. - codegen: drop dead IsNullable field; avoid rune-slice alloc in pascalCase digit check. - tests: nsClient (byte-identical duplicate of queryTestCtx) deleted, table_test boilerplate collapsed onto queryTestCtx (9 + 8 sites). - Makefile: drop test-go-sdk from ci-parallel (already reached via test). Rejected after verification: 1000<= len(os.Args) { + fmt.Fprintf(os.Stderr, "Error: missing value for %s (use --help)\n", flag) + os.Exit(2) + } + return os.Args[*i] +} + func parseArgs() cliArgs { args := cliArgs{url: "http://localhost:8080", out: "./wavehouse_types.go", pkg: "main"} for i := 1; i < len(os.Args); i++ { switch os.Args[i] { case "--url", "-u": - i++ - if i < len(os.Args) { - args.url = os.Args[i] - } + args.url = flagValue(&i) case "--out", "-o": - i++ - if i < len(os.Args) { - args.out = os.Args[i] - } + args.out = flagValue(&i) case "--auth", "-a": - i++ - if i < len(os.Args) { - args.auth = os.Args[i] - } + args.auth = flagValue(&i) case "--package", "-p": - i++ - if i < len(os.Args) { - args.pkg = os.Args[i] - } + args.pkg = flagValue(&i) case "--help", "-h": fmt.Println(`wavehouse-codegen — Generate Go types from WaveHouse schema @@ -76,7 +76,6 @@ Options: type column struct { Name string `json:"name"` Type string `json:"type"` - IsNullable bool `json:"is_nullable"` HasDefault bool `json:"has_default"` } @@ -276,7 +275,7 @@ func pascalCase(s string) string { // Go identifiers can't start with a digit (e.g. a table named // "2fa_events" would otherwise produce the invalid identifier // "2faEvents"). Prefix with "X" to keep it a valid, exported name. - if unicode.IsDigit([]rune(result)[0]) { + if unicode.IsDigit(rune(result[0])) { // digits are ASCII; no rune-slice needed result = "X" + result } return result diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index 26fbac07..e90f8412 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -125,31 +125,33 @@ func TestConformance_WireFormat(t *testing.T) { logCallErr(t, err) case "ingest": - if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { - if len(tc.Operations[0].Args) == 0 { - t.Fatal("insert needs 1 arg, got 0") - } - data := tc.Operations[0].Args[0] - _, err := c.From(tc.Table).Insert(ctx, data) - logCallErr(t, err) + if len(tc.Operations) == 0 || tc.Operations[0].Method != "insert" { + t.Fatalf("ingest case %q has no insert operation", tc.Name) } + if len(tc.Operations[0].Args) == 0 { + t.Fatal("insert needs 1 arg, got 0") + } + data := tc.Operations[0].Args[0] + _, err := c.From(tc.Table).Insert(ctx, data) + logCallErr(t, err) case "ingest_batch": - if len(tc.Operations) > 0 && tc.Operations[0].Method == "insert" { - if len(tc.Operations[0].Args) == 0 { - t.Fatal("insert needs 1 arg, got 0") - } - rawArr, ok := tc.Operations[0].Args[0].([]any) - if !ok { - t.Fatalf("batch insert args[0] is not an array") - } - rows := make([]map[string]any, len(rawArr)) - for i, r := range rawArr { - rows[i] = toStringMap(r) - } - _, err := c.From(tc.Table).Insert(ctx, rows) - logCallErr(t, err) + if len(tc.Operations) == 0 || tc.Operations[0].Method != "insert" { + t.Fatalf("ingest_batch case %q has no insert operation", tc.Name) + } + if len(tc.Operations[0].Args) == 0 { + t.Fatal("insert needs 1 arg, got 0") + } + rawArr, ok := tc.Operations[0].Args[0].([]any) + if !ok { + t.Fatalf("batch insert args[0] is not an array") } + rows := make([]map[string]any, len(rawArr)) + for i, r := range rawArr { + rows[i] = toStringMap(t, r) + } + _, batchErr := c.From(tc.Table).Insert(ctx, rows) + logCallErr(t, batchErr) case "pipe": p := c.Pipe(tc.PipeName, tc.PipeParams) @@ -377,12 +379,13 @@ func toStringSlice(args []any) []string { return out } -func toStringMap(v any) map[string]any { +func toStringMap(t *testing.T, v any) map[string]any { + t.Helper() m, ok := v.(map[string]any) - if ok { - return m + if !ok { + t.Fatalf("fixture row is not an object: %T", v) } - return nil + return m } // deepEqualJSON compares two JSON-decoded values, treating float64 ints as equal diff --git a/clients/go/http.go b/clients/go/http.go index 74dcbdad..d8a8b37b 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -185,12 +185,10 @@ func retryAfterDelay(ra string, attempt int) time.Duration { func backoff(attempt int) time.Duration { ms := 1000 * math.Pow(2, float64(attempt)) - if ms > 30000 { - ms = 30000 - } - // ±20% jitter so clients failing at the same moment don't retry in lockstep. + // ±20% jitter so clients failing at the same moment don't retry in + // lockstep; capped after jitter so the documented 30s max holds. ms *= 0.8 + 0.4*rand.Float64() //nolint:gosec // retry jitter, not cryptographic - return time.Duration(ms) * time.Millisecond + return time.Duration(min(ms, 30000)) * time.Millisecond } func sleepWithContext(ctx context.Context, d time.Duration) error { diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index 76acc176..e7340b6f 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -4,24 +4,12 @@ import ( "context" "encoding/json" "net/http" - "net/http/httptest" "sync" "testing" ) -func nsClient(t *testing.T, handler http.Handler) *Client { - t.Helper() - srv := httptest.NewServer(handler) - t.Cleanup(srv.Close) - return NewClient(Config{ - BaseURL: srv.URL, - HTTPClient: srv.Client(), - Options: &ClientOptions{MaxRetries: 0}, - }) -} - func TestSysNamespace_Health(t *testing.T) { - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/v1/health" { t.Errorf("want /v1/health, got %s", r.URL.Path) } @@ -34,7 +22,7 @@ func TestSysNamespace_Health(t *testing.T) { } func TestSchemaNamespace_List(t *testing.T) { - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/v1/schema" { t.Errorf("want /v1/schema, got %s", r.URL.Path) } @@ -54,7 +42,7 @@ func TestSchemaNamespace_List(t *testing.T) { func TestSchemaNamespace_Refresh(t *testing.T) { var mu sync.Mutex var gotMethod string - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() gotMethod = r.Method mu.Unlock() @@ -72,7 +60,7 @@ func TestSchemaNamespace_Refresh(t *testing.T) { } func TestPolicyNamespace_GetSetValidate(t *testing.T) { - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) @@ -107,7 +95,7 @@ func TestPolicyNamespace_GetSetValidate(t *testing.T) { func TestDLQNamespace(t *testing.T) { t.Run("List", func(t *testing.T) { - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) })) stats, err := c.DLQ.List(context.Background()) @@ -122,7 +110,7 @@ func TestDLQNamespace(t *testing.T) { t.Run("Table", func(t *testing.T) { var mu sync.Mutex var gotParam string - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() gotParam = r.URL.Query().Get("table") mu.Unlock() @@ -141,7 +129,7 @@ func TestDLQNamespace(t *testing.T) { } func TestPipesNamespace_CRUD(t *testing.T) { - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": if r.URL.Path == "/v1/admin/pipes" { @@ -187,7 +175,7 @@ func TestPipeRef_Fetch(t *testing.T) { var mu sync.Mutex var gotPath, gotMethod string var gotBody map[string]any - c := nsClient(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() gotPath = r.URL.Path gotMethod = r.Method diff --git a/clients/go/stream.go b/clients/go/stream.go index b6402489..8307e2fb 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -160,12 +160,17 @@ func (sc *StreamController) setStatus(s StreamStatus) { } } -func (sc *StreamController) emitEvent(event StreamEvent) { +// snapshotSubs copies the subscriber list under mu so callbacks run unlocked. +// setStatus keeps its own inline copy: there the snapshot must share the +// critical section with the status write to keep callback order consistent. +func (sc *StreamController) snapshotSubs() []*StreamSubscriber { sc.mu.Lock() - subs := append([]*StreamSubscriber(nil), sc.subscribers...) - sc.mu.Unlock() + defer sc.mu.Unlock() + return append([]*StreamSubscriber(nil), sc.subscribers...) +} - for _, sub := range subs { +func (sc *StreamController) emitEvent(event StreamEvent) { + for _, sub := range sc.snapshotSubs() { if sub.Next != nil { sub.Next(event) } @@ -190,11 +195,7 @@ func (sc *StreamController) emitEvent(event StreamEvent) { } func (sc *StreamController) emitError(err error) { - sc.mu.Lock() - subs := append([]*StreamSubscriber(nil), sc.subscribers...) - sc.mu.Unlock() - - for _, sub := range subs { + for _, sub := range sc.snapshotSubs() { if sub.Error != nil { sub.Error(err) } @@ -445,6 +446,9 @@ func newFilteredStreamController(inner *StreamController, filters []QueryFilter, unsub() inner.Close() case <-inner.done: + // Inner closed on its own — still unsubscribe so the closed + // controller doesn't retain a reference to this wrapper. + unsub() } }() @@ -542,23 +546,15 @@ func equalValues(a, b any) bool { } // evaluateIn checks whether actual is contained in the expected slice. -// Handles both []any and typed slices (e.g., []string, []int). +// Reflection handles []any and typed slices (e.g., []string, []int) alike. func evaluateIn(actual, expected any) bool { - if arr, ok := expected.([]any); ok { - for _, v := range arr { - if equalValues(actual, v) { - return true - } - } + rv := reflect.ValueOf(expected) + if rv.Kind() != reflect.Slice { return false } - // Handle typed slices via reflection. - rv := reflect.ValueOf(expected) - if rv.Kind() == reflect.Slice { - for i := range rv.Len() { - if equalValues(actual, rv.Index(i).Interface()) { - return true - } + for i := range rv.Len() { + if equalValues(actual, rv.Index(i).Interface()) { + return true } } return false @@ -600,32 +596,19 @@ func toFloat64(v any) (float64, bool) { return n, true case float32: return float64(n), true - case int: - return float64(n), true - case int8: - return float64(n), true - case int16: - return float64(n), true - case int32: - return float64(n), true - case int64: - return float64(n), true - case uint: - return float64(n), true - case uint8: - return float64(n), true - case uint16: - return float64(n), true - case uint32: - return float64(n), true - case uint64: - return float64(n), true case json.Number: f, err := n.Float64() return f, err == nil - default: - return 0, false } + // All int/uint widths in two cases (codegen structs use the narrow ones). + rv := reflect.ValueOf(v) + switch { + case rv.CanInt(): + return float64(rv.Int()), true + case rv.CanUint(): + return float64(rv.Uint()), true + } + return 0, false } func projectColumns(row map[string]any, columns []string) map[string]any { diff --git a/clients/go/table_test.go b/clients/go/table_test.go index e935657f..2dfd0cff 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -5,7 +5,6 @@ import ( "encoding/json" "io" "net/http" - "net/http/httptest" "sync" "testing" ) @@ -16,15 +15,13 @@ func TestTableRef_InsertSingle(t *testing.T) { var mu sync.Mutex var gotBody map[string]any var gotPath string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() gotPath = r.URL.Path _ = json.NewDecoder(r.Body).Decode(&gotBody) _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/home"}) if err != nil { t.Fatal(err) @@ -46,7 +43,7 @@ func TestTableRef_InsertBatch(t *testing.T) { var mu sync.Mutex var gotCT string var gotBody string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() gotCT = r.Header.Get("Content-Type") @@ -56,8 +53,6 @@ func TestTableRef_InsertBatch(t *testing.T) { "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []map[string]any{ {"page": "/a"}, {"page": "/b"}, @@ -91,7 +86,7 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { var mu sync.Mutex var gotCT string var gotBody string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() gotCT = r.Header.Get("Content-Type") @@ -101,8 +96,6 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { "total": 2, "succeeded": 1, "failed": 1, "duplicates": 0, }) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []ClickRow{ {Page: "/a"}, {Page: "/b"}, @@ -134,14 +127,12 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { var mu sync.Mutex var gotPath string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() gotPath = r.URL.Path _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) if err != nil { t.Fatal(err) @@ -157,11 +148,9 @@ func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { } func TestTableRef_InsertEmptyBatch(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { t.Fatal("should not make a request for empty batch") })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), []map[string]any{}) if err != nil { t.Fatal(err) @@ -177,7 +166,7 @@ func TestTableRef_InsertEmptyBatch(t *testing.T) { func TestTableRef_InsertNDJSON(t *testing.T) { var mu sync.Mutex var gotBody string - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() raw, _ := io.ReadAll(r.Body) @@ -186,8 +175,6 @@ func TestTableRef_InsertNDJSON(t *testing.T) { "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, }) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) ndjson := `{"page":"/a"}` + "\n" + `{"page":"/b"}` result, err := c.From("clicks").InsertNDJSON(context.Background(), ndjson) if err != nil { @@ -204,7 +191,7 @@ func TestTableRef_InsertNDJSON(t *testing.T) { } func TestTableRef_Schema(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Query().Get("table") != "clicks" { t.Errorf("want table=clicks") } @@ -215,8 +202,6 @@ func TestTableRef_Schema(t *testing.T) { }, }) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) schema, err := c.From("clicks").Schema(context.Background()) if err != nil { t.Fatal(err) @@ -230,11 +215,9 @@ func TestTableRef_Schema(t *testing.T) { } func TestTableRef_InsertDuplicate(t *testing.T) { - srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _ = json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) })) - t.Cleanup(srv.Close) - c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/dup"}) if err != nil { t.Fatal(err) diff --git a/clients/go/testdata/wire_cases.json b/clients/go/testdata/wire_cases.json index 2c27b46a..e721afb6 100644 --- a/clients/go/testdata/wire_cases.json +++ b/clients/go/testdata/wire_cases.json @@ -47,6 +47,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "=", "/home"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "page", "op": "eq", "value": "/home" }], @@ -61,6 +63,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "!=", "/home"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "page", "op": "neq", "value": "/home" }], @@ -75,6 +79,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", ">", 10] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "gt", "value": 10 }], @@ -89,6 +95,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", ">=", 10] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "gte", "value": 10 }], @@ -103,6 +111,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", "<", 5] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "lt", "value": 5 }], @@ -117,6 +127,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["score", "<=", 5] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "lte", "value": 5 }], @@ -131,6 +143,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "in", ["/home", "/about"]] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "page", "op": "in", "value": ["/home", "/about"] }], @@ -145,6 +159,8 @@ { "method": "select", "args": ["page"] }, { "method": "where", "args": ["page", "like", "/home%"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "page", "op": "like", "value": "/home%" }], @@ -158,6 +174,8 @@ "operations": [ { "method": "count", "args": ["*", "total"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], "limit": 1000 @@ -170,6 +188,8 @@ "operations": [ { "method": "sum", "args": ["score", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "sum", "column": "score", "alias": "sum_score" }], "limit": 1000 @@ -182,6 +202,8 @@ "operations": [ { "method": "avg", "args": ["score", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "avg", "column": "score", "alias": "avg_score" }], "limit": 1000 @@ -194,6 +216,8 @@ "operations": [ { "method": "min", "args": ["score", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "min", "column": "score", "alias": "min_score" }], "limit": 1000 @@ -206,6 +230,8 @@ "operations": [ { "method": "max", "args": ["score", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "max", "column": "score", "alias": "max_score" }], "limit": 1000 @@ -218,6 +244,8 @@ "operations": [ { "method": "countDistinct", "args": ["page", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "countDistinct", "column": "page", "alias": "count_distinct_page" }], "limit": 1000 @@ -230,6 +258,8 @@ "operations": [ { "method": "aggregate", "args": ["uniqExact", "user_id", "unique_users"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "aggregations": [{ "fn": "uniqExact", "column": "user_id", "alias": "unique_users" }], "limit": 1000 @@ -243,6 +273,8 @@ { "method": "select", "args": ["page"] }, { "method": "groupBy", "args": ["page"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "group_by": ["page"], @@ -257,6 +289,8 @@ { "method": "select", "args": ["page"] }, { "method": "orderBy", "args": ["page", "asc"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "order_by": [{ "column": "page", "dir": "asc" }], @@ -271,6 +305,8 @@ { "method": "select", "args": ["page"] }, { "method": "orderBy", "args": ["score", "desc"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "order_by": [{ "column": "score", "dir": "desc" }], @@ -285,6 +321,8 @@ { "method": "select", "args": ["page"] }, { "method": "limit", "args": [50] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "limit": 50 @@ -298,6 +336,8 @@ { "method": "select", "args": ["page"] }, { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "time_range": { "column": "received_timestamp", "since": "1h" }, @@ -312,6 +352,8 @@ { "method": "select", "args": ["page"] }, { "method": "timeRange", "args": ["ts", "2026-01-01", "2026-02-01"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "time_range": { "column": "ts", "since": "2026-01-01", "until": "2026-02-01" }, @@ -327,6 +369,8 @@ { "method": "where", "args": ["score", ">", 10] }, { "method": "where", "args": ["page", "=", "/home"] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [ @@ -349,6 +393,8 @@ { "method": "limit", "args": [50] }, { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", "expected_body": { "columns": ["page"], "filters": [{ "column": "score", "op": "gt", "value": 10 }], diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index a4d338e8..5acfc46b 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -81,7 +81,7 @@ func NewClient(cfg Config) *Client { c := &Client{ ctx: httpContext{ - baseURL: trimTrailingSlashes(cfg.BaseURL), + baseURL: strings.TrimRight(cfg.BaseURL, "/"), auth: cfg.Auth, maxRetries: maxRetries, httpClient: hc, @@ -137,7 +137,3 @@ func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { func (c *Client) createStream(table string, opts *StreamOptions) *StreamController { return newStreamController(c.ctx, table, opts) } - -func trimTrailingSlashes(s string) string { - return strings.TrimRight(s, "/") -} diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 4996b594..0a0f3ef8 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -181,14 +181,12 @@ for (const tc of cases) { break; } case "ingest": - if (tc.operations?.[0]?.method === "insert") { - await wh.from(tc.table).insert(tc.operations[0].args[0]); - } - break; case "ingest_batch": - if (tc.operations?.[0]?.method === "insert") { - await wh.from(tc.table).insert(tc.operations[0].args[0]); + if (tc.operations?.[0]?.method !== "insert") { + // Hard failure, matching the Go harness. + throw new Error(`${tc.name}: ingest case has no insert operation`); } + await wh.from(tc.table).insert(tc.operations[0].args[0]); break; case "pipe": await wh.pipe(tc.pipe_name, tc.pipe_params ?? undefined).fetch(); From c2c1021d3d5551564a6da7e3f3e4ac462f3e84ab Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 12:47:09 -0400 Subject: [PATCH 21/40] =?UTF-8?q?docs:=20address=20pre-push=20review=20rou?= =?UTF-8?q?nd=2011=20=E2=80=94=20stale=20changelog=20entry,=20ARGS/V=20acc?= =?UTF-8?q?uracy?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drops the CHANGELOG dependency-bump entry: after merging origin/main, go.mod and go.sum are byte-identical to main and no longer appear in this PR's delta, and the bumps it claimed credit for landed on main via Dependabot (#438). The matching bullet in the PR description was removed too. Corrects the test-target flag documentation in development.md, which claimed all test targets accept ARGS. Per the Makefile: gotestsum drives test-unit and test-integration only; ARGS reaches test-unit, test-integration, and test-ts; V=1 reaches test-unit, test-integration, and test-e2e (the orchestrator reads it directly). Adds the Go SDK unit test and wire-conformance case entries to "Adding New Tests", which the SDK sync rule requires for every new endpoint. Narrows the "every operation returns (T, error)" claim in the Go SDK index, reference, and README: the void operations (Pipes.Set/Delete, Policy.Set, Schema.Refresh, Sys.Health) return a bare error, as reference.md's own API tree already showed. Lists the FilterOp constants rather than raw symbols in the streaming filter docs, matching how every example calls .Where(), and fixes "the SDK readme" to "readmes" now that docs-prose.sh resolves both. --- CHANGELOG.md | 2 -- clients/go/README.md | 2 +- docs/src/content/docs/claude-code.md | 2 +- docs/src/content/docs/development.md | 10 ++++++---- docs/src/content/docs/sdk/go/index.md | 2 +- docs/src/content/docs/sdk/go/reference.md | 4 +++- docs/src/content/docs/sdk/go/streaming.md | 6 ++++-- 7 files changed, 16 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ade0f381..5915fe88 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,8 +34,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Go toolchain requirement bumped to 1.26.5** (`go.mod`): the `go` directive moves from `1.26.4` to `1.26.5` so local builds (via `GOTOOLCHAIN=auto`) and CI's `setup-go` (which reads `go.mod` via `go-version-file`) install a Go whose standard library clears the `govulncheck` findings GO-2026-5856 (`crypto/tls`) and GO-2026-4970 (`os`), both fixed in 1.26.5 — those findings were failing `make verify`'s `vulncheck` leaf (and with it the pre-commit hook) on every tree. Patch-level toolchain bump only — no source changes — and the released binaries pick up the patched stdlib too. -- **Dependency bumps clearing new govulncheck findings** (`go.mod`, `go.sum`): `google.golang.org/grpc` 1.81.1 → 1.82.1 (GO-2026-6061), `golang.org/x/text` 0.37.0 → 0.39.0 (GO-2026-5970), `github.com/klauspost/compress` 1.18.6 → 1.18.7 (GO-2026-5841) — all three were failing `make verify`'s `vulncheck` leaf after the advisories published. No API changes. - ### Security - **Policy `_in` is now enforced on both the row-`filter` and insert-`check` paths, closing a fail-open row-security gap** (`internal/policy/policy.go`, `internal/api/ingest.go`, `docs/src/content/docs/access-control.mdx`, plus tests in `internal/policy/policy_test.go`, `internal/api/ingest_test.go`): closes #224. The `Filter` schema accepted `_in` but the engine never read it: on the row-`filter`/SELECT path `resolveFilters` had no `_in` branch, so a row-security filter like `tenant_id: { _in: … }` produced **no `WHERE` predicate** and the role saw every row instead of its tenant subset (a fail-open, same family as #223); on the `check`/INSERT path only `_eq` was honored, silently dropping any other operator. `_in` now takes a single claim that resolves to a JSON **array** (the multi-tenant case — a token's `tenant_ids` list) and emits `col IN (?, …)` with one bound param per element; a scalar claim is a one-element set, and an empty/absent claim matches **no rows** (fail-closed) rather than widening to all of them. On the insert path an `_in` check requires the column be present and one of the set — there is no single value to auto-inject as `_eq` does, so an omitted column is rejected (`403 check failed`). The comparison operators are enforced on `filter` (`_eq`/`_neq`/`_gt`/`_lt`/`_in` all produce predicates now, so nothing is rejected there) and, on `check`, `_neq`/`_gt`/`_lt` become a loud config-load rejection (no insert-time semantics; `check` honors `_eq` + `_in`). The `_in` value stays a single templated string in the wire schema (Go `Filter.In`, SDK `PolicyFilter._in`), matching the established "set = array" shape of the caller-query `in` operator. diff --git a/clients/go/README.md b/clients/go/README.md index bd6109d3..5db4eaac 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -204,7 +204,7 @@ See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference#c ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors originating from the HTTP exchange are `*wavehouse.Error` — unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`, and `Connected`) deliver errors through callbacks or plain errors instead: +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare `error` for operations with no result body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors originating from the HTTP exchange are `*wavehouse.Error` — unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`, and `Connected`) deliver errors through callbacks or plain errors instead: ```go page, err := client.From("clicks").Fetch(ctx) diff --git a/docs/src/content/docs/claude-code.md b/docs/src/content/docs/claude-code.md index c418517f..c57abc4c 100644 --- a/docs/src/content/docs/claude-code.md +++ b/docs/src/content/docs/claude-code.md @@ -80,7 +80,7 @@ To add a command: drop a `.md` file in `.claude/commands/`. Filename becomes the | Subagent | When to use | | -------- | ----------- | | `pre-push-reviewer` | **Mandatory before pushing to a PR branch** (enforced by `.claude/hooks/agent-bash-gate.sh`), run in parallel with the other reviewers in `scripts/pre-push-reviewers.sh` — all must reach `ship_it`. Also used for auditing someone else's PR after `wt switch pr:`. Runs the canonical `.github/prompts/pr-review.md` workflow against the local branch in fresh context. Fetches PR comments + CI status + linked-issue acceptance criteria when on a PR branch. Returns `[MUST]`/`[SHOULD]`/`[MAY]` findings + a parseable `VERDICT: ship_it\|iterate\|block` line that drives the `tmp/pre-push-reviewer-passed-` marker. | -| `docs-reviewer` | **Mandatory before pushing to a PR branch** — runs in parallel with the other pre-push reviewers (all enforced by `.claude/hooks/agent-bash-gate.sh`). Reviews docs **prose** (accuracy-vs-code, runnable examples, clarity, completeness) **and code↔docs sync** (code that changed but whose docs didn't), using `.github/prompts/docs-review.md` over the `scripts/docs-prose.sh` denylist set (Starlight site + governance docs incl. the SDK readme). Default (branch) scope emits `VERDICT: ship_it\|iterate\|block` → writes `tmp/docs-reviewer-passed-`; a path/`all` is advisory (no marker). Posts no PR comments, never edits docs. Complements misspell / markdownlint / starlight-links-validator, never duplicates them. | +| `docs-reviewer` | **Mandatory before pushing to a PR branch** — runs in parallel with the other pre-push reviewers (all enforced by `.claude/hooks/agent-bash-gate.sh`). Reviews docs **prose** (accuracy-vs-code, runnable examples, clarity, completeness) **and code↔docs sync** (code that changed but whose docs didn't), using `.github/prompts/docs-review.md` over the `scripts/docs-prose.sh` denylist set (Starlight site + governance docs incl. the SDK readmes). Default (branch) scope emits `VERDICT: ship_it\|iterate\|block` → writes `tmp/docs-reviewer-passed-`; a path/`all` is advisory (no marker). Posts no PR comments, never edits docs. Complements misspell / markdownlint / starlight-links-validator, never duplicates them. | Invoke via the `Agent` tool with `subagent_type: pre-push-reviewer`, or via `/agents`. diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index cd9c6d46..d4718565 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -288,7 +288,7 @@ go build -o bin/wavehouse ./cmd/wavehouse ### How It Works -The coverage-instrumented suite targets (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary, and accept `ARGS`/`V=1`. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. +The Go suite targets (`test-unit`, `test-integration`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. `test-e2e` runs the orchestrator + vitest and `test-ts` runs vitest directly, so neither uses gotestsum. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. Go tests run with the **race detector** (`-race`) enabled by default (including `test-go-sdk` — the SDK's streaming subsystem is highly concurrent; `test-go-sdk-e2e` skips it since it drives a live server). WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. @@ -327,9 +327,9 @@ make cov Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. -**Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). +**Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output on `test-unit` / `test-integration`, and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-ts` and the Go SDK / conformance targets ignore it. -**Extra flags**: All test targets accept `ARGS="..."` for additional `go test` flags (e.g., `-run`, `-count`, `-timeout`). +**Extra flags**: `test-unit`, `test-integration`, and `test-ts` accept `ARGS="..."` for pass-through flags (e.g., `-run`, `-count`, `-timeout` for the Go targets; vitest flags for `test-ts`). `test-e2e` and the Go SDK / conformance targets ignore it. **Note on timing**: gotestsum's `DONE ... in X.XXXs` reports pure test execution time. The total wall time includes Go compiling all packages — the first run compiles everything (~15s), subsequent runs use the build cache (~1s). @@ -355,6 +355,8 @@ Shared test utilities live in `internal/testutil/` (e.g., `testutil.NopLogger()` - **Unit test for `internal/foo/`** → create `internal/foo/foo_test.go` (same package). - **Integration test needing Docker** → add a subtest under `tests/integration/` (e.g. a new file with `//go:build integration`). - **E2E test via SDK** → add a `tests/e2e/sdk/*.test.ts` file. These tests exercise the full pipeline (ingest → ClickHouse → query) through the TypeScript SDK. Run with `make test-e2e`. +- **Go SDK unit test** → add to `clients/go/*_test.go` (nested module — outside `test-unit`'s scope). Run with `make test-go-sdk`. +- **Wire-format parity case** → when you add or change an endpoint, add an entry to `clients/go/testdata/wire_cases.json` plus its dispatch in both runners (`clients/go/conformance_test.go` and `tests/conformance/conformance_ts.mjs`). Required by the SDK sync rule in `AGENTS.md` / `CONTRIBUTING.md`. - **Test helpers** → add to `internal/testutil/` (Go) or `tests/e2e/sdk/helpers.ts` (E2E). ### E2E Tests via SDK @@ -519,7 +521,7 @@ Run `make help` to see all targets. Key ones: | `make clean-tools` | Installed tools and pnpm deps (`.bin/`, `node_modules/`) | | `make clean-all` | Full reset: above + `data/` + Docker volumes | -The gotestsum-driven targets (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) accept `ARGS="..."` for pass-through `go test` flags and `V=1` for verbose output; `test-go-sdk`, `test-go-sdk-e2e`, and `test-conformance-ts` ignore both. Build targets accept `TAGS="..."` for Go build tags. +`test-unit`, `test-integration`, and `test-ts` accept `ARGS="..."` for pass-through flags; `test-unit`, `test-integration`, and `test-e2e` accept `V=1` for verbose output. `test-go-sdk`, `test-go-sdk-e2e`, and `test-conformance-ts` ignore both. Build targets accept `TAGS="..."` for Go build tags. ## Dependency Management diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 8e0b2329..4ece842f 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -174,7 +174,7 @@ parameter. ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors originating from the HTTP exchange are `*wavehouse.Error`; unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close`, `Connected` — deliver errors through callbacks or plain errors instead; see [Streaming](/sdk/go/streaming).) +Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare `error` for operations with no result body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors originating from the HTTP exchange are `*wavehouse.Error`; unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close`, `Connected` — deliver errors through callbacks or plain errors instead; see [Streaming](/sdk/go/streaming).) ```go page, err := wh.From("clicks").Fetch(ctx) diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 0074b7f0..8eb16881 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -39,7 +39,9 @@ background goroutine, torn down explicitly via `.Close()`. See ## Error Handling The SDK never panics on API or network failures — every request-response -operation (queries, ingest, pipes, admin) returns `(T, error)`. Errors +operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare +`error` for operations with no result body (`Pipes.Set`/`Delete`, +`Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors originating from the HTTP exchange are `*wavehouse.Error` (unwrap with `errors.As`); client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 70f44a37..01484290 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -197,8 +197,10 @@ stream := wh.From("clicks"). // Only events where page == "/home" are emitted, with only page + button fields ``` -Supported operators: `=`, `!=`, `>`, `>=`, `<`, `<=`, `in`, `like`, -`not_like` — the same `FilterOp` set `.Where()` takes everywhere. `like` / +Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, +`OpIn`, `OpLike`, `OpNotLike` — the same `FilterOp` set `.Where()` takes +everywhere (the SDK maps them to wire tokens such as `eq`/`neq` +internally). `like` / `not_like` match SQL LIKE semantics (`%` → any run of characters, `_` → any single character), case-insensitively. `in` accepts any Go slice type on the right-hand side (`[]string`, `[]int`, `[]any`, ...), not just `[]any`. From 0ee17344e9d0f2ade266a34094214481e37ceb1a Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 13:01:33 -0400 Subject: [PATCH 22/40] =?UTF-8?q?docs:=20address=20pre-push=20review=20rou?= =?UTF-8?q?nd=2012=20=E2=80=94=20E2E=20lifecycle,=20lint=20install,=20429?= =?UTF-8?q?=20sync?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The E2E section of development.md still described the pre-orchestrator flow. tests/e2e/sdk/setup.ts starts nothing: it probes the orchestrator-supplied CLICKHOUSE_URL/WAVEHOUSE_URL and throws if either is unreachable, and the orchestrator has no :8080 handling at all — it launches wavehouse-cov on a random free port, so a running `make dev` is neither needed nor reused. golangci-lint is auto-installed pinned to v2.11.4 in .bin/_/ by the Makefile, not looked up on PATH. The manual install list steered contributors toward an unpinned version the build never uses, whose findings diverge from CI's. Also narrows the V=1 comment, which claimed every test target honors it. Documents 429 as retryable in the Go SDK reference and streaming pages, which clients/go/errors.go has classified as retryable since the review rounds, and notes the 30s Retry-After clamp on both retryable rows. Finishes the round-11 FilterOp constant switch, whose follow-on sentences still used wire tokens. The TS live query derives its dedup boundary from the last fetched row rather than the newest, so a desc-ordered fetch re-delivers overlap-window events; the Go client takes the max instead. Prose now describes actual behavior and links #449. TS not retrying 429 while Go does is tracked in #450. Extends the Go SDK changelog entry's file list to all 33 touched files and records the TS SDK doc corrections under Fixed, per the exhaustive-list convention the surrounding entries follow. --- CHANGELOG.md | 4 +++- docs/src/content/docs/development.md | 15 +++++---------- docs/src/content/docs/sdk/go/reference.md | 3 ++- docs/src/content/docs/sdk/go/streaming.md | 12 ++++++------ docs/src/content/docs/sdk/streaming.md | 2 +- 5 files changed, 17 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5915fe88..6eb553f4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,7 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), and a `wavehouse-codegen` CLI that generates row structs from `/v1/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{queries,streaming,pipes,admin,reference}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), and a `wavehouse-codegen` CLI that generates row structs from `/v1/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. - **Non-JWT operator key for full-access admin + break-glass recovery** (`internal/auth/auth.go`, `internal/auth/context.go`, `internal/auth/auth_test.go`, `internal/api/router.go`, `internal/api/router_test.go`, `internal/config/config.go`, `internal/config/config_test.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `tests/e2e/fixtures/config.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/access-control.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/development.md`, `docs/src/content/docs/architecture.md`, `docs/src/content/docs/reverse-proxy.mdx`, `AGENTS.md`, `SECURITY.md`): closes #240; partially advances the auth-hardening epic [#228](https://github.com/Wave-RF/WaveHouse/issues/228) and the management/data-plane split [#359](https://github.com/Wave-RF/WaveHouse/issues/359). Adds an optional `auth.operator_key` (`WH_AUTH_OPERATOR_KEY`): a request presenting it — via an `Authorization: Operator ` header (forwarded verbatim by proxies, no collision with Bearer JWTs) or the `X-Operator-Key` alias — is authorized as a full-access platform operator — the whole data plane *and* the `/v1/admin/*` management surface — without minting a JWT and independently of the token verifier, giving the person running the deployment a role-free credential for bootstrap and break-glass. The middleware checks it before the Bearer token with a constant-time comparison (`crypto/subtle`) and stamps two things into the request context: the live `admin_role` (so the policy evaluator's admin bypass grants unrestricted data-plane access while a policy exists) and a platform-operator bit that `RequireAdmin` honors **even when the policy is `nil`/deleted** — the one HTTP path that can restore a wiped policy, which previously required SSH access and a reboot. Empty (the default) disables it, so existing deployments are unchanged; treat it as an admin secret (load from a secret store, serve only over TLS). A successful operator authentication is audit-logged at `INFO`; a request presenting a *non-matching* operator key is logged at `WARN` and counted by a new `wavehouse_auth_operator_key_failures_total` counter (a probing/brute-force signal on the most privileged credential in the system) before falling through to the normal token/default path — the middleware still never rejects. The `Authorization` auth-scheme is matched case-insensitively (RFC 7235) via a shared `authScheme` helper, which also makes the existing `Bearer` JWT scheme case-insensitive (previously it required the canonical `Bearer` casing). Explicitly out of scope, tracked in #359: scoping the operator credential to the management surface only, and capability-scoped admin permissions. - **Missing-dedupe-id observability + optional strict mode** (`internal/api/ingest.go`, `internal/api/ingest_test.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/architecture.md`): closes #219. With dedupe enabled, a row missing the configured `id_field` can't be deduped — previously it was published with idempotency silently disabled and *no* log or metric, so a producer bug that dropped the id turned off the guarantee for those rows unnoticed. Now every such row is logged at `WARN` and counted by a new `wavehouse_ingest_dedupe_missing_id_total` counter (labeled by `table`), making the loss observable server-side. A new opt-in `dedupe.require_id` (`WH_DEDUPE_REQUIRE_ID`, default `false`) turns that signal into enforcement: a row missing the id is rejected (`400` for a single insert; a per-record failure in a batch) instead of published — a tripwire for producers that must guarantee the id (complements the client-side [#202](https://github.com/Wave-RF/WaveHouse/issues/202)). Default behavior is unchanged. - **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes. @@ -51,6 +51,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **TypeScript SDK documentation corrections found while writing the Go SDK's parity docs** (`docs/src/content/docs/sdk/{reference,queries,streaming}.md`): the error-code table was missing `SSE_ERROR` and `SSE_CONNECT_ERROR` entirely, and described `401` as "missing or invalid JWT" when a *missing* token actually resolves to `default_role` and is denied with `403` (`internal/auth/auth.go`) — only a present-but-invalid or expired token yields `401`. `.aggregate()` was documented as accepting a "custom fn", but the server enforces an allowlist (`internal/query/builder.go`); the allowed set is now listed. Live queries gained a caution for a real footgun: the backfill dedup boundary comes from the fetched rows' `received_timestamp`, so a `.select(...)` projection omitting that column silently disables dedup and delivers overlap-window events twice. No SDK code changed — these were pre-existing gaps between the TS docs and the server's behavior. + - **Go module cache stored once instead of once per compile flavor** (`.github/actions/setup-env/action.yml`, `.github/workflows/README.md`, `.github/workflows/publish-dev.yml`, `.github/workflows/release.yml`, `Makefile`): closes [#443](https://github.com/Wave-RF/WaveHouse/issues/443). `setup-env` cached `~/go/pkg/mod` together with `~/.cache/go-build` under a key partitioned by `go-cache-suffix`, but the module cache is a pure function of `go.mod` + `go.sum` and byte-identical for every flavor — so that tree was stored five times over (`-lint`, `-unit`, `-integration`, `-e2e-cov`, `-cov`) — five entries of ~0.9-1.2 GB stored each, ~5.2 GB per generation (the tree is ~1.6 GB on disk; 0.48 GB as a stored archive on a cold save, drifting up as superseded versions accumulate). Two live generations is the steady state (a bump mints a new set while the previous is still warm), so the repo sat near GitHub's hard 10 GB cache cap; the 24-module go-deps bump ([#438](https://github.com/Wave-RF/WaveHouse/pull/438)) tipped it to 10.53 GB and GitHub began LRU-evicting warm entries mid-run. The one cache is now two: `gomod-v1--` on `~/go/pkg/mod`, **unsuffixed** and shared by every `ci.yml` Go job that goes through `setup-env`, and `gobuild-v3--go-` on `~/.cache/go-build` only, still per flavor. Measured after the split: **1.05 GB per generation** (0.48 GB module + 0.57 GB across the five build entries), down from 5.18 GB — a 4.9x reduction. All sizes are stored-archive bytes / 2^30, the unit the README's usage check prints. The `v3` bump is load-bearing — saves fire only on an exact-key miss, so without it the old `v2` entry (still carrying the module cache) would exact-hit forever and the smaller content would never be saved — and `gobuild-v3` drops the bare-prefix restore-key, which existed solely to borrow another flavor's copy of the module cache. Separately, `publish-dev.yml` and `release.yml` now pass `cache: false` to `actions/setup-go` (matching `goreleaser-validate.yml`), which was holding a sixth ~1 GB entry — the module tree `gomod-v1` already keeps once, plus that job's own 8-target cross-compile objects — re-saved on every cache miss. (That entry is keyed on the root `go.mod`: setup-go hashed `go.sum` through v6.2.0 and `go.mod` from v6.3.0, [actions/setup-go#705](https://github.com/actions/setup-go/pull/705).) `publish-dev.yml` re-caches only the half that pays for itself, under `gobuild-v3--go-release-` (~0.5 GB — the bundled entry minus `gomod-v1`'s share): across its last 20 runs GoReleaser takes 36–246 s with the cross-compile objects warm and 401–446 s cold (measured on `setup-go`'s bundled cache, which carried the same `~/.cache/go-build` tree), so dropping the cross-compile objects outright would have cost roughly 2.5–7 minutes on every push to main (mean delta ≈4.8 min). The `-release` suffix keeps those 8-target objects from being restored by CI's native-only flavors and vice versa. Because those timings were taken with setup-go's bundled entry (which also held `~/go/pkg/mod`), `publish-dev` additionally *restores* `gomod-v1` from `main`'s scope via `actions/cache/restore` — read-only, so it costs no budget and cannot write a partial tree to the key every `ci.yml` Go job shares. Without that restore the job would re-download ~112 MB of modules per push and land above the warm range quoted above. Both Go keys now hash `go.mod` alongside `go.sum`, for a different reason each: the GOTOOLCHAIN=auto toolchain lives in `~/go/pkg/mod` and `go.sum` records no entry for it, so a `go`-directive bump would otherwise exact-hit a toolchain-less archive and — saves firing only on an exact-key miss — re-download it every run; and the compiler's build ID keys every build object, so the same bump invalidates `~/.cache/go-build` too, where the failure mode is a permanent cold recompile rather than a re-download. Two guards come with the shared entry: `setup-env` now fails a `go: true` job that passes no `go-cache-suffix` (an empty one yields a restore-key prefix-matching every other flavor), and `make cov` gains the `go-mod-download` prerequisite its siblings already had — CI's coverage job shares the unsuffixed `gomod-v1` and races to save it, but ran only `go run ./scripts/cov report`, so winning that race would have stored a partial `~/go/pkg/mod` that then exact-hit for every other job until the next rotation. The workflows README gains a sizing policy — the 10 GB cap, the two-generations rule, how to check the current footprint, and the rule that lockfile-derived content is keyed once and shared — plus the narrowing-rotation exception to the key-versioning policy. - **Live demo hero feed renders in `event_ts` order instead of SSE arrival order** (`docs/src/components/LiveDemo.astro`): the landing-page live activity feed prepended each streamed row to the top in the order the SSE stream delivered it, but a producer's webhook burst (a single merge-queue cycle fires ~20 events) arrives in no guaranteed order and the stream relays it in ingest order — so a late or out-of-order delivery landed above newer rows (e.g. a `pushed 12m ago` sitting on top of `reviewed a pull request 9m ago`). `addRow` now keeps the feed sorted by `event_ts` descending — it slots each row in before the first strictly-older sibling rather than blind-prepending — so the live tail matches the already-sorted `gh_activity_recent` backfill. The zone-less-SSE-timestamp normalization the sort relies on (`normTs`) was already in place; equal-second rows keep arrival order (`gh_events.event_ts` is only second-granular for CI/checks, so there's no finer tiebreak), and dedup + the `MAX_ROWS` trim are unchanged. Surfaced in dogfooding on `wavehouse.dev`; the client-side analog of the ingest-order reality the SSE stream can't reorder. diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index d4718565..962d6ae5 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -295,7 +295,8 @@ Go tests run with the **race detector** (`-race`) enabled by default (including ### Quick Reference ```bash -# Prefix any test target with V=1 for verbose output, e.g. `V=1 make test` +# V=1 gives verbose output on test-unit / test-integration / test-e2e, +# e.g. `V=1 make test-unit` # Unit tests + Go SDK tests (compact output) — alias for `test-unit` + `test-go-sdk` make test @@ -366,7 +367,7 @@ The primary E2E integration test suite lives in `tests/e2e/sdk/`. It uses the Ty **Architecture**: - `scripts/orchestrator` — the E2E entrypoint behind `make test-e2e`: it starts a clean ClickHouse **testcontainer** per run, launches the `wavehouse-cov` binary on a random free port, runs the SDK suite against it, then SIGINTs the binary to flush coverage. No Compose file is involved. CI runs the exact same path. -- `tests/e2e/sdk/setup.ts` — Smart `globalSetup` that probes ports before starting Docker services, so tests work seamlessly whether you started services manually or let the setup do it. +- `tests/e2e/sdk/setup.ts` — `globalSetup` that probes the orchestrator-supplied `CLICKHOUSE_URL`/`WAVEHOUSE_URL`, creates the per-suite tables, refreshes the schema, and bootstraps a baseline policy. Lifecycle is owned by the orchestrator: if either URL isn't reachable it fails fast rather than starting anything itself. - `tests/e2e/sdk/helpers.ts` — JWT factories, typed client constructors, async wait helpers, direct ClickHouse query helper. **Running E2E tests**: @@ -378,7 +379,7 @@ make test-e2e `make test-e2e` builds `bin/wavehouse-cov` (coverage-instrumented) and runs the orchestrator under `scripts/orchestrator/` to wire ClickHouse + the cover binary into the suite. covdata flushes on SIGINT into `tmp/coverage/e2e/data/`. -**If you already have `make dev` running**, the setup detects the healthy WaveHouse on `:8080` and skips starting it via Docker — only ClickHouse is started if needed. +`make test-e2e` is self-contained — it always starts its own ClickHouse testcontainer and `wavehouse-cov` on a random free port, so it neither needs nor reuses a running `make dev`. **Test files** (`tests/e2e/sdk/*.test.ts`): `admin`, `auth`, `batching`, `cache`, `dlq`, `ingest`, `ndjson`, `query`, `streaming`, `stress`. @@ -388,13 +389,7 @@ make test-e2e make lint ``` -`golangci-lint` is installed separately (not in `go.mod` — its massive dependency tree causes conflicts). If not found, `make lint` prints install instructions. - -Install options: - -- **macOS**: `brew install golangci-lint` -- **Binary**: See [golangci-lint.run/welcome/install/](https://golangci-lint.run/welcome/install/) -- **Go install**: `go install github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest` +`golangci-lint` is pinned in the `Makefile` (v2.11.4) and auto-installed to `.bin/_/` on first `make lint` (or `make tools`) — no manual install needed. It's kept out of `go.mod` because its dependency tree conflicts with the main module. Install it globally and you'll get an unpinned version the build never uses, with findings that diverge from CI. The configuration is in `.golangci.yml` (v2 format with `default: none` for explicit control) — that file is the authoritative list of enabled linters. Highlights: diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 8eb16881..7471371d 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -57,8 +57,9 @@ SDK's "the SDK never throws" guarantee. | 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | +| 429 | `HTTP_429` | Yes | Rate limited (auto-retries, honoring `Retry-After`, capped at 30s) | | 500 | `HTTP_500` | Yes | Server error (retried per `ClientOptions.MaxRetries`) | -| 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`) | +| 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`, capped at 30s) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `context.Context` | | 0 | `SSE_ERROR` | Yes | Stream connection failure, delivered to the subscriber's `Error` callback; the stream reconnects automatically | diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 01484290..d64c9305 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -167,7 +167,7 @@ described above). | --------- | --------- | -------- | | SSE | Automatic, with exponential backoff (capped at 30s) and gap-fill replay via the last-seen event ID | HTTP/2 recommended | -Reconnect covers transport failures and retryable (5xx) responses. A +Reconnect covers transport failures and retryable (5xx/429) responses. A non-retryable response (401/403/404) is terminal: the error is delivered to the subscriber's `Error` callback, status goes to `StatusClosed`, and the stream does not reconnect — fix the cause (refresh the token, correct the @@ -199,11 +199,11 @@ stream := wh.From("clicks"). Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the same `FilterOp` set `.Where()` takes -everywhere (the SDK maps them to wire tokens such as `eq`/`neq` -internally). `like` / -`not_like` match SQL LIKE semantics (`%` → any run of characters, `_` → any -single character), case-insensitively. `in` accepts any Go slice type on -the right-hand side (`[]string`, `[]int`, `[]any`, ...), not just `[]any`. +everywhere (the SDK maps them to wire tokens such as `eq`/`neq` internally). +`OpLike` / `OpNotLike` match SQL LIKE semantics (`%` → any run of +characters, `_` → any single character), case-insensitively. `OpIn` accepts +any Go slice type on the right-hand side (`[]string`, `[]int`, `[]any`, +...), not just `[]any`. --- diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index 7bc4b7fb..9b569128 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -154,7 +154,7 @@ interface StreamSubscriber { 1. Opens the stream **immediately** and buffers incoming events. 2. Runs the `.fetch()` query for historical data, calls `subscriber.initial()` with the result. -3. Deduplicates buffered events by comparing timestamps against the latest historical timestamp. +3. Deduplicates buffered events against the **last** fetched row's `received_timestamp` (a string comparison). With the `.orderBy('received_timestamp', 'desc')` used above, the last row is the *oldest* one, so the bound is looser than intended and events in the overlap window can still be re-delivered — tracked in [#449](https://github.com/Wave-RF/WaveHouse/issues/449). The Go SDK takes the maximum instead. 4. Flushes remaining buffered events and switches to live mode. This "stream-first" approach ensures no events are lost between the fetch and stream start. From 51d8b4514ff2405ad8f2b0bef207395058a8070d Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 13:13:22 -0400 Subject: [PATCH 23/40] =?UTF-8?q?docs:=20address=20pre-push=20review=20rou?= =?UTF-8?q?nd=2013=20=E2=80=94=20429=20sweep,=20gotestsum=20format,=20LIKE?= =?UTF-8?q?=20case=20split?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The round-12 429 sweep updated the two site pages but missed the Go SDK's own README, which is the module's front page on GitHub and pkg.go.dev: it still claimed only 5xx/network are retried and only 503 honors Retry-After. clients/go/errors.go classifies 429 retryable and clients/go/http.go honors Retry-After on both, clamped to 30s. The MaxRetries row on the Go SDK index page had the same incomplete enumeration. development.md named testdox as the compact gotestsum format; the Makefile uses pkgname-and-test-fails, switching to standard-verbose under V=1 (gotestdox is only a transitive dependency). The sentence was rewritten last round to add per-target scope, so the stale format name passed through. Documents a real semantic split in live queries: client-side like/not_like compiles to a case-insensitive regex (clients/go/stream.go compileLike, and the TS query builder it deliberately matches) while the server compiles the same operator to ClickHouse LIKE, which is case-sensitive. A live query can therefore disagree with itself — the backfill excludes rows the live stream includes. Both streaming pages now carry a caution beside the existing dedup one. Pre-existing since the TS SDK; the alignment decision is tracked in #451. --- clients/go/README.md | 2 +- docs/src/content/docs/development.md | 2 +- docs/src/content/docs/sdk/go/index.md | 2 +- docs/src/content/docs/sdk/go/streaming.md | 9 +++++++++ docs/src/content/docs/sdk/streaming.md | 4 ++++ 5 files changed, 16 insertions(+), 3 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index 5db4eaac..a44069e0 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -216,7 +216,7 @@ if err != nil { } ``` -The HTTP layer retries 5xx and network errors with exponential backoff (default 2 retries). 503 with `Retry-After` is honored. Context cancellation returns immediately with code `ABORTED`. +The HTTP layer retries 5xx, 429, and network errors with exponential backoff (default 2 retries). `Retry-After` on a 503 or 429 is honored, capped at 30s. Context cancellation returns immediately with code `ABORTED`. ## License diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index 962d6ae5..bee9476d 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -328,7 +328,7 @@ make cov Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. -**Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output on `test-unit` / `test-integration`, and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-ts` and the Go SDK / conformance targets ignore it. +**Verbose output**: Use `V=1` to switch from the compact `pkgname-and-test-fails` format to `standard-verbose` on `test-unit` / `test-integration`, and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-ts` and the Go SDK / conformance targets ignore it. **Extra flags**: `test-unit`, `test-integration`, and `test-ts` accept `ARGS="..."` for pass-through flags (e.g., `-run`, `-count`, `-timeout` for the Go targets; vitest flags for `test-ts`). `test-e2e` and the Go SDK / conformance targets ignore it. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 4ece842f..a41000a7 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -103,7 +103,7 @@ TLS / response-header timeouts instead. | Field | Type | Default | Description | |-------|------|---------|-------------| -| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, network failures) | +| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures) | A `*Client` is safe for concurrent use by multiple goroutines — client state is immutable after `NewClient`, and every builder chain copies. Supply a diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index d64c9305..e5e33de8 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -282,6 +282,15 @@ that omits it disables dedup, and events in the fetch/stream overlap window are delivered twice — once in `Initial`, again via `Next`. ::: +:::caution[`OpLike` matching differs between backfill and live] +Client-side `OpLike` / `OpNotLike` matching is case-**insensitive**, but the +backfill runs server-side where the same operator compiles to ClickHouse +`LIKE`, which is case-**sensitive**. A live query filtering on `OpLike` can +therefore disagree with itself: the backfill excludes rows the live stream +includes. Tracked in +[#451](https://github.com/Wave-RF/WaveHouse/issues/451). +::: + ### `.Close()` Shuts down the live query and its underlying stream. Safe to call more than diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index 9b569128..5aa5b73d 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -162,3 +162,7 @@ This "stream-first" approach ensures no events are lost between the fetch and st :::caution[Dedup needs `received_timestamp` in the projection] The dedup boundary comes from the fetched rows' `received_timestamp` values. A `.select(...)` projection that omits that column disables dedup, and events in the fetch/stream overlap window are delivered twice — once in `initial()`, again via `next()`. ::: + +:::caution[`like` matching differs between backfill and live] +Client-side `like` / `not_like` matching is case-**insensitive**, but the backfill runs server-side where the same operator compiles to ClickHouse `LIKE`, which is case-**sensitive**. A live query filtering on `like` can therefore disagree with itself: the backfill excludes rows the live stream includes. Tracked in [#451](https://github.com/Wave-RF/WaveHouse/issues/451). +::: From 4ee4f1f4c9682abdddf09e46c7bd5a0cb8c20d21 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 16:56:30 -0400 Subject: [PATCH 24/40] =?UTF-8?q?docs:=20address=20pre-push=20review=20rou?= =?UTF-8?q?nd=2014=20=E2=80=94=20not=5Flike=20is=20rejected,=20not=20case-?= =?UTF-8?q?split?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The LIKE caution added last round was right about like and wrong about not_like. internal/query/builder.go's filterToSQL has cases for eq/neq/gt/gte/lt/lte/like/in only, so not_like falls through to default and /v1/query rejects it with 400 — a live query filtering on it fails its Initial callback rather than quietly disagreeing with itself. Both streaming pages now scope the case-sensitivity claim to like and describe not_like separately, matching what the two queries pages and api.md already said. Also corrects "the Makefile uses go run" to go tool (Makefile:144-150 defines every pinned tool as `go tool `; the same page already said so thirty lines earlier), and gives the TS selectAll() description the same restricted-vs-unrestricted split this branch added everywhere else — an unrestricted role gets a bare SELECT *, not an expansion. --- docs/src/content/docs/development.md | 2 +- docs/src/content/docs/sdk/go/streaming.md | 16 ++++++++++------ docs/src/content/docs/sdk/queries.md | 2 +- docs/src/content/docs/sdk/streaming.md | 4 +++- 4 files changed, 15 insertions(+), 9 deletions(-) diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index bee9476d..b62d7635 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -288,7 +288,7 @@ go build -o bin/wavehouse ./cmd/wavehouse ### How It Works -The Go suite targets (`test-unit`, `test-integration`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go run` so no global installation is needed. `test-e2e` runs the orchestrator + vitest and `test-ts` runs vitest directly, so neither uses gotestsum. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. +The Go suite targets (`test-unit`, `test-integration`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile uses `go tool` so no global installation is needed. `test-e2e` runs the orchestrator + vitest and `test-ts` runs vitest directly, so neither uses gotestsum. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. Go tests run with the **race detector** (`-race`) enabled by default (including `test-go-sdk` — the SDK's streaming subsystem is highly concurrent; `test-go-sdk-e2e` skips it since it drives a live server). WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index e5e33de8..c105f28c 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -283,12 +283,16 @@ are delivered twice — once in `Initial`, again via `Next`. ::: :::caution[`OpLike` matching differs between backfill and live] -Client-side `OpLike` / `OpNotLike` matching is case-**insensitive**, but the -backfill runs server-side where the same operator compiles to ClickHouse -`LIKE`, which is case-**sensitive**. A live query filtering on `OpLike` can -therefore disagree with itself: the backfill excludes rows the live stream -includes. Tracked in -[#451](https://github.com/Wave-RF/WaveHouse/issues/451). +Client-side `OpLike` matching is case-**insensitive**, but the backfill runs +server-side where the operator compiles to ClickHouse `LIKE`, which is +case-**sensitive**. A live query filtering on `OpLike` can therefore +disagree with itself: the backfill excludes rows the live stream includes. +Tracked in [#451](https://github.com/Wave-RF/WaveHouse/issues/451). + +`OpNotLike` never reaches the backfill at all — `/v1/query` rejects the +operator with a `400`, so a live query filtering on it fails its `Initial` +callback. See the operator table in +[Queries](/sdk/go/queries#wherecolumn-op-value). ::: ### `.Close()` diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index a75ee3e5..92c98062 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -126,7 +126,7 @@ const q = clicks.select('page').select('button'); // SELECT page, button #### `.selectAll()` -Select every column your role may read (the all-columns wildcard, expanded server-side to your allowed columns). Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.). +Select every column your role may read. For a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`). Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.). ```ts const q = clicks.selectAll().where('country', '=', 'US'); diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index 5aa5b73d..18ba2ef0 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -164,5 +164,7 @@ The dedup boundary comes from the fetched rows' `received_timestamp` values. A ` ::: :::caution[`like` matching differs between backfill and live] -Client-side `like` / `not_like` matching is case-**insensitive**, but the backfill runs server-side where the same operator compiles to ClickHouse `LIKE`, which is case-**sensitive**. A live query filtering on `like` can therefore disagree with itself: the backfill excludes rows the live stream includes. Tracked in [#451](https://github.com/Wave-RF/WaveHouse/issues/451). +Client-side `like` matching is case-**insensitive**, but the backfill runs server-side where the operator compiles to ClickHouse `LIKE`, which is case-**sensitive**. A live query filtering on `like` can therefore disagree with itself: the backfill excludes rows the live stream includes. Tracked in [#451](https://github.com/Wave-RF/WaveHouse/issues/451). + +`not_like` never reaches the backfill at all — `/v1/query` rejects the operator with a `400`, so a live query filtering on it fails its `initial()` callback. See the operator table in [Queries](/sdk/queries#wherecolumn-op-value). ::: From 25e36cccaf73b3b6cc51363d2c6f8583a11280d5 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 17:35:21 -0400 Subject: [PATCH 25/40] docs: restore branch edits lost in the merge, extend path-prefix docs to Go MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolving the sdk/reference.md conflict by taking main's page wholesale was too blunt. Main's SSE rows and their explanatory paragraph are correct and stay — the branch's "the stream reconnects automatically" was wrong for the TS SDK, which never re-dials a stream itself. But the same resolution also reverted two unrelated branch edits on that page: - The 401 row went back to "Missing or invalid JWT". internal/auth/auth.go returns no auth error for an empty token, so a missing token resolves to default_role and is denied 403; only a present-but-invalid or expired one yields 401. api.md and the Go reference page still said so, so the page contradicted both. - The title lost its "TypeScript SDK" prefix, leaving the only unqualified "SDK ..." title in a two-SDK tree — it reads as the shared reference in breadcrumbs and search results when it is TS-only. Main's new path-prefix guidance is TypeScript-only, which understates it now that the Go SDK is co-canonical and preserves a prefix on both transports (wavehouse.go trims the trailing slash; http.go and stream.go concatenate). reverse-proxy.mdx now shows both clients and scopes the "upgrade your SDK" caution to TypeScript, since the Go client never had #428. The Go README gains the sentence its TypeScript counterpart got in the merge. --- clients/go/README.md | 2 ++ docs/src/content/docs/reverse-proxy.mdx | 14 ++++++++++++-- docs/src/content/docs/sdk/reference.md | 4 ++-- 3 files changed, 16 insertions(+), 4 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index a44069e0..c1412c6d 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -80,6 +80,8 @@ client = wavehouse.NewClient(wavehouse.Config{ }) ``` +`BaseURL` may include a path prefix (`https://app.example.com/api/warehouse`) when WaveHouse is served under one. A trailing `/` is trimmed and every request path is appended to it, on both REST and SSE — see [Config](https://wavehouse.dev/sdk/go#config). + ## Typed Queries (Generics) ```go diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index 672d0374..c2ce9b60 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -80,10 +80,20 @@ handle_path /api/warehouse/* { The ingress controller forwards the full path unless you ask it to rewrite. Pair a capture-group path (`/api/warehouse(/|$)(.*)` with `pathType: ImplementationSpecific`) with the `nginx.ingress.kubernetes.io/rewrite-target: /$2` annotation, or the prefix arrives at WaveHouse unstripped. ::: -Point the SDK at the prefixed URL and it does the rest — `createClient({ baseURL: 'https://app.example.com/api/warehouse' })` sends both REST calls and SSE streams under the prefix ([SDK → Serving under a path prefix](/sdk#serving-under-a-path-prefix)). +Point either SDK at the prefixed URL and it does the rest — both send REST calls and SSE streams under the prefix: + +```ts +// TypeScript — see /sdk#serving-under-a-path-prefix +createClient({ baseURL: 'https://app.example.com/api/warehouse' }); +``` + +```go +// Go — see /sdk/go#config +wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/warehouse"}) +``` :::caution[Check the prefix actually survives to the wire] -A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: an SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). +A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: a **TypeScript** SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). The Go SDK preserves the prefix on every released version. ::: ## Request-body size limits diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index 39b42d52..db6fc36b 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -1,5 +1,5 @@ --- -title: "SDK Reference & CLI" +title: "TypeScript SDK Reference & CLI" description: "Error codes, AbortController, the full API tree, the codegen CLI, and E2E testing with @wavehouse/sdk." --- @@ -30,7 +30,7 @@ The SDK **never throws** for anything the server returns — all API errors come | Status | Code | Retryable | Description | |--------|------|-----------|-------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Missing or invalid JWT | +| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 500 | `HTTP_500` | Yes | Server error (retried per `maxRetries`) | From 1158ff31072f77b55c603096dd17711a20164a81 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 17:37:42 -0400 Subject: [PATCH 26/40] test(sdk): pin BaseURL path-prefix support; qualify the 401 rows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The merge documented path-prefix support as a promise on the Go SDK's Config row and README, but nothing tested it — every test server in clients/go was root-hosted. The TS side shipped url.test.ts and friends pinning exactly this after #428. Adds a guard per transport, since the SSE URL is built in stream.go independently of buildURL: a mux serving only the prefixed path, so a dropped prefix 404s (REST) or never arrives (SSE). Both verified failing against a deliberately broken buildURL and stream URL before being kept. Qualifies the 401 row on both reference pages: "denied with 403" was too absolute. A missing token is evaluated as default_role, which may well succeed — the point is that it never yields 401. Also trims the changelog's claim that this branch added SSE_ERROR/SSE_CONNECT_ERROR to the TS error table; post-merge those rows come from #448 in the same Unreleased section. --- CHANGELOG.md | 2 +- clients/go/http_test.go | 27 +++++++++++++++++ clients/go/stream_test.go | 36 +++++++++++++++++++++++ docs/src/content/docs/sdk/go/reference.md | 2 +- docs/src/content/docs/sdk/reference.md | 2 +- 5 files changed, 66 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 54bef1b9..848f2325 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -52,7 +52,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **TypeScript SDK documentation corrections found while writing the Go SDK's parity docs** (`docs/src/content/docs/sdk/{reference,queries,streaming}.md`): the error-code table was missing `SSE_ERROR` and `SSE_CONNECT_ERROR` entirely, and described `401` as "missing or invalid JWT" when a *missing* token actually resolves to `default_role` and is denied with `403` (`internal/auth/auth.go`) — only a present-but-invalid or expired token yields `401`. `.aggregate()` was documented as accepting a "custom fn", but the server enforces an allowlist (`internal/query/builder.go`); the allowed set is now listed. Live queries gained a caution for a real footgun: the backfill dedup boundary comes from the fetched rows' `received_timestamp`, so a `.select(...)` projection omitting that column silently disables dedup and delivers overlap-window events twice. No SDK code changed — these were pre-existing gaps between the TS docs and the server's behavior. +- **TypeScript SDK documentation corrections found while writing the Go SDK's parity docs** (`docs/src/content/docs/sdk/{reference,queries,streaming}.md`): the error-code table described `401` as "missing or invalid JWT" when a *missing* token is actually evaluated as `default_role` — succeeding or denied with `403`, never `401` (`internal/auth/auth.go`, `internal/api/errors.go`) — and only a present-but-invalid or expired token yields `401`. `.aggregate()` was documented as accepting a "custom fn", but the server enforces an allowlist (`internal/query/builder.go`); the allowed set is now listed. Live queries gained a caution for a real footgun: the backfill dedup boundary comes from the fetched rows' `received_timestamp`, so a `.select(...)` projection omitting that column silently disables dedup and delivers overlap-window events twice. No SDK code changed — these were pre-existing gaps between the TS docs and the server's behavior. - **Go module cache stored once instead of once per compile flavor** (`.github/actions/setup-env/action.yml`, `.github/workflows/README.md`, `.github/workflows/publish-dev.yml`, `.github/workflows/release.yml`, `Makefile`): closes [#443](https://github.com/Wave-RF/WaveHouse/issues/443). `setup-env` cached `~/go/pkg/mod` together with `~/.cache/go-build` under a key partitioned by `go-cache-suffix`, but the module cache is a pure function of `go.mod` + `go.sum` and byte-identical for every flavor — so that tree was stored five times over (`-lint`, `-unit`, `-integration`, `-e2e-cov`, `-cov`) — five entries of ~0.9-1.2 GB stored each, ~5.2 GB per generation (the tree is ~1.6 GB on disk; 0.48 GB as a stored archive on a cold save, drifting up as superseded versions accumulate). Two live generations is the steady state (a bump mints a new set while the previous is still warm), so the repo sat near GitHub's hard 10 GB cache cap; the 24-module go-deps bump ([#438](https://github.com/Wave-RF/WaveHouse/pull/438)) tipped it to 10.53 GB and GitHub began LRU-evicting warm entries mid-run. The one cache is now two: `gomod-v1--` on `~/go/pkg/mod`, **unsuffixed** and shared by every `ci.yml` Go job that goes through `setup-env`, and `gobuild-v3--go-` on `~/.cache/go-build` only, still per flavor. Measured after the split: **1.05 GB per generation** (0.48 GB module + 0.57 GB across the five build entries), down from 5.18 GB — a 4.9x reduction. All sizes are stored-archive bytes / 2^30, the unit the README's usage check prints. The `v3` bump is load-bearing — saves fire only on an exact-key miss, so without it the old `v2` entry (still carrying the module cache) would exact-hit forever and the smaller content would never be saved — and `gobuild-v3` drops the bare-prefix restore-key, which existed solely to borrow another flavor's copy of the module cache. Separately, `publish-dev.yml` and `release.yml` now pass `cache: false` to `actions/setup-go` (matching `goreleaser-validate.yml`), which was holding a sixth ~1 GB entry — the module tree `gomod-v1` already keeps once, plus that job's own 8-target cross-compile objects — re-saved on every cache miss. (That entry is keyed on the root `go.mod`: setup-go hashed `go.sum` through v6.2.0 and `go.mod` from v6.3.0, [actions/setup-go#705](https://github.com/actions/setup-go/pull/705).) `publish-dev.yml` re-caches only the half that pays for itself, under `gobuild-v3--go-release-` (~0.5 GB — the bundled entry minus `gomod-v1`'s share): across its last 20 runs GoReleaser takes 36–246 s with the cross-compile objects warm and 401–446 s cold (measured on `setup-go`'s bundled cache, which carried the same `~/.cache/go-build` tree), so dropping the cross-compile objects outright would have cost roughly 2.5–7 minutes on every push to main (mean delta ≈4.8 min). The `-release` suffix keeps those 8-target objects from being restored by CI's native-only flavors and vice versa. Because those timings were taken with setup-go's bundled entry (which also held `~/go/pkg/mod`), `publish-dev` additionally *restores* `gomod-v1` from `main`'s scope via `actions/cache/restore` — read-only, so it costs no budget and cannot write a partial tree to the key every `ci.yml` Go job shares. Without that restore the job would re-download ~112 MB of modules per push and land above the warm range quoted above. Both Go keys now hash `go.mod` alongside `go.sum`, for a different reason each: the GOTOOLCHAIN=auto toolchain lives in `~/go/pkg/mod` and `go.sum` records no entry for it, so a `go`-directive bump would otherwise exact-hit a toolchain-less archive and — saves firing only on an exact-key miss — re-download it every run; and the compiler's build ID keys every build object, so the same bump invalidates `~/.cache/go-build` too, where the failure mode is a permanent cold recompile rather than a re-download. Two guards come with the shared entry: `setup-env` now fails a `go: true` job that passes no `go-cache-suffix` (an empty one yields a restore-key prefix-matching every other flavor), and `make cov` gains the `go-mod-download` prerequisite its siblings already had — CI's coverage job shares the unsuffixed `gomod-v1` and races to save it, but ran only `go run ./scripts/cov report`, so winning that race would have stored a partial `~/go/pkg/mod` that then exact-hit for every other job until the next rotation. The workflows README gains a sizing policy — the 10 GB cap, the two-generations rule, how to check the current footprint, and the rule that lockfile-derived content is keyed once and shared — plus the narrowing-rotation exception to the key-versioning policy. - **A path prefix in the SDK's `baseURL` now survives instead of being silently discarded** (`clients/ts/src/url.ts` (new), `clients/ts/src/http.ts`, `clients/ts/src/stream/sse.ts`, `clients/ts/src/cli/codegen.ts`, `clients/ts/src/url.test.ts` (new), `clients/ts/src/stream/sse.test.ts` (new), `clients/ts/src/{http,client}.test.ts`, `docs/src/content/docs/sdk/index.mdx`, `docs/src/content/docs/reverse-proxy.mdx`): closes #428. Pointing the SDK at a WaveHouse served under a prefix — `createClient({ baseURL: 'https://app.example.com/api/warehouse' })`, the shape you get behind a BFF, an app-server route, or a path-routed ingress — dropped the prefix from every request. Both transports resolved *absolute* request paths against the base (`new URL('/v1/query', base)` in `http.ts`, `new URL('/v1/stream', baseURL)` in `stream/sse.ts`), and per the URL spec an absolute path replaces the base's path entirely, so calls went to the origin root. The failure mode was the bad kind: no error, a client that looks correctly configured, and every request quietly going somewhere else — with no workaround from outside the SDK, since `baseURL` was the only path input and it couldn't survive. Request paths are now joined **onto** the base by a single shared `resolveURL` helper that both transports and the codegen CLI call (previously three separate constructions, one of which — codegen's string concat — already handled prefixes, so they disagreed). The helper normalizes the base to a directory before resolving, so a bare last segment or a stray query/fragment on `baseURL` can't eat the prefix either, and a root-hosted base (`http://localhost:8080`, the overwhelmingly common case) resolves exactly as before. Tests pin a prefixed base end-to-end across both transports. The proxy in front must still strip the prefix before forwarding — WaveHouse has no configurable base path by design — which the reverse-proxy guide now covers with nginx/Caddy snippets. diff --git a/clients/go/http_test.go b/clients/go/http_test.go index df9045ff..acf857d1 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -293,3 +293,30 @@ func TestDoRequest_429RetriesWithRetryAfter(t *testing.T) { t.Fatalf("Retry-After: 1 not honored — retried after only %v", elapsed) } } + +// A BaseURL carrying a path prefix must survive on both transports — the bug +// #428 fixed in the TS client, which Go avoids by concatenating rather than +// resolving. Guards against a future switch to url.JoinPath/ResolveReference. +func TestBaseURLPathPrefixIsPreserved(t *testing.T) { + mux := http.NewServeMux() + mux.HandleFunc("/api/warehouse/v1/query", func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + }) + srv := httptest.NewServer(mux) // anything off-prefix 404s + t.Cleanup(srv.Close) + + for _, base := range []string{srv.URL + "/api/warehouse", srv.URL + "/api/warehouse/"} { + client := NewClient(Config{BaseURL: base, Options: &ClientOptions{}, HTTPClient: srv.Client()}) + + var result map[string]string + if err := doRequest(context.Background(), client.ctx, requestOptions{ + method: "POST", + path: "/v1/query", + }, &result); err != nil { + t.Fatalf("base %q: %v", base, err) + } + if result["status"] != "ok" { + t.Fatalf("base %q: want ok, got %v", base, result) + } + } +} diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index 1d9a9f69..efe10fd2 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -430,3 +430,39 @@ func TestStream_ReconnectResumesFromLastEventID(t *testing.T) { t.Fatalf("reconnect: want since=, got %q", sinceParams[1]) } } + +// The SSE transport builds its URL separately from buildURL (stream.go), so a +// BaseURL path prefix needs its own guard. See TestBaseURLPathPrefixIsPreserved. +func TestStreamBaseURLPathPrefixIsPreserved(t *testing.T) { + gotPath := make(chan string, 1) + mux := http.NewServeMux() + mux.HandleFunc("/api/warehouse/v1/stream", func(w http.ResponseWriter, r *http.Request) { + select { + case gotPath <- r.URL.Path: + default: + } + w.Header().Set("Content-Type", "text/event-stream") + w.WriteHeader(200) + w.(http.Flusher).Flush() + <-r.Context().Done() + }) + srv := httptest.NewServer(mux) // anything off-prefix 404s + t.Cleanup(srv.Close) + + client := NewClient(Config{ + BaseURL: srv.URL + "/api/warehouse", + Options: &ClientOptions{}, + HTTPClient: srv.Client(), + }) + sc := client.From("clicks").Stream(nil) + t.Cleanup(sc.Close) + + select { + case p := <-gotPath: + if p != "/api/warehouse/v1/stream" { + t.Fatalf("want prefixed stream path, got %q", p) + } + case <-time.After(3 * time.Second): + t.Fatal("stream never reached the prefixed path") + } +} diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 7471371d..35e0ba2f 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -54,7 +54,7 @@ SDK's "the SDK never throws" guarantee. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | +| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token is evaluated as `default_role`, so it succeeds or is denied with 403 — never 401) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 429 | `HTTP_429` | Yes | Rate limited (auto-retries, honoring `Retry-After`, capped at 30s) | diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index db6fc36b..92b63460 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -30,7 +30,7 @@ The SDK **never throws** for anything the server returns — all API errors come | Status | Code | Retryable | Description | |--------|------|-----------|-------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token resolves to `default_role` and is denied with 403) | +| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token is evaluated as `default_role`, so it succeeds or is denied with 403 — never 401) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 500 | `HTTP_500` | Yes | Server error (retried per `maxRetries`) | From 0a46251e30e3dbc9daf63c2ac7eadbe9880d20ac Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 17:53:03 -0400 Subject: [PATCH 27/40] fix(sdk): clamp Retry-After before overflow; unalias insert-result pointers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Triage of the 24 open review threads surfaced one real defect. retryAfterDelay computed time.Duration(secs) * time.Second before any range check, which wraps negative past ~9.2e9 seconds — and because the wrap goes negative, the min(delay, maxRetryAfter) clamp below selected it rather than capping. The retry timer then fired immediately, so a server answering 429/503 with a large Retry-After got hammered with zero-delay retries: the exact inverse of the header's purpose, and a silent breach of the documented 30s cap. Verified: "10000000000" yielded -2346317h47m53s, MaxInt64 yielded -1s. The HTTP-date branch is unaffected — time.Sub saturates rather than wrapping. emptyInsertResult aliased one &z across Total/Succeeded/Failed/Duplicates, so a caller writing through any one of those exported *int fields mutated all four. Separate vars now. fetchNextTyped discarded its json.Marshal error, truncating a result set to look like normal end-of-pagination. Propagated. The Decode error stays deliberate — a Row marshaling to a non-object ends pagination the same way an absent cursor column does — now commented as such, with the parity question tracked in #452. Test fixes for guards that could not fail: - InsertByteSliceNotBatch asserted only the path and OK, both of which the batch path also satisfies; it now pins Content-Type and body, and was confirmed to fail with the []byte carve-out removed. - backoff's CappedAt30s accepted 24-36s where the cap is applied after jitter and returns exactly 30s, so moving the cap before the jitter still passed. - retryAfterDelay's HTTPDateFuture window subsumed the ~1s parse-failure fallback; tightened, and two overflow cases added. - TestNewClient_HasNamespaces boxed typed pointers into map[string]any, where a nil typed pointer is never == nil — it could not fail. Compares concrete fields now. - TestClient_From discarded Fetch's error, so an early return left its handler assertions unreached. Docs: a pagination snippet that did not compile, three shell examples that were bash syntax errors (unquoted ), the cursor tie-breaker caveat (#452), and the raw-SQL claim that a tokenless request is rejected — untrue when default_role is the admin role, which AGENTS.md permits as dev-only. Fixed on both SDKs' pages. --- clients/go/README.md | 2 +- clients/go/client_test.go | 27 ++++++++++++++++++----- clients/go/http.go | 6 +++++ clients/go/http_test.go | 19 ++++++++++++++-- clients/go/query_builder.go | 12 +++++++++- clients/go/table.go | 6 +++-- clients/go/table_test.go | 17 +++++++++++++- docs/src/content/docs/sdk/go/queries.md | 18 +++++++++++++-- docs/src/content/docs/sdk/go/reference.md | 4 ++-- docs/src/content/docs/sdk/queries.md | 2 +- 10 files changed, 96 insertions(+), 17 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index c1412c6d..112aae2b 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -195,7 +195,7 @@ rows, _ := wavehouse.SQL[map[string]any](ctx, client, "SELECT count() FROM click Generate Go structs from a running WaveHouse instance: ```bash -export WAVEHOUSE_AUTH= # avoids leaking the token via argv +export WAVEHOUSE_AUTH='' # avoids leaking the token via argv go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --url http://localhost:8080 \ --out ./db_types.go \ diff --git a/clients/go/client_test.go b/clients/go/client_test.go index 22a45067..aca92d0d 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -37,10 +37,23 @@ func TestNewClient_CustomMaxRetries(t *testing.T) { func TestNewClient_HasNamespaces(t *testing.T) { c := NewClient(Config{BaseURL: "http://localhost:8080"}) - for name, ns := range map[string]any{"Sys": c.Sys, "Schema": c.Schema, "Policy": c.Policy, "Pipes": c.Pipes, "DLQ": c.DLQ} { - if ns == nil { - t.Fatalf("%s namespace is nil", name) - } + // Compared as concrete typed pointers, not boxed into map[string]any: a nil + // typed pointer in an interface is never == nil, so the map form passed + // even if NewClient stopped assigning a namespace entirely. + if c.Sys == nil { + t.Error("Sys namespace is nil") + } + if c.Schema == nil { + t.Error("Schema namespace is nil") + } + if c.Policy == nil { + t.Error("Policy namespace is nil") + } + if c.Pipes == nil { + t.Error("Pipes namespace is nil") + } + if c.DLQ == nil { + t.Error("DLQ namespace is nil") } } @@ -54,7 +67,11 @@ func TestClient_From(t *testing.T) { })) t.Cleanup(srv.Close) c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) - _, _ = c.From("events").Fetch(context.Background()) + // Checked, not discarded: if Fetch returns before issuing the request, the + // handler never runs and the table= assertion above proves nothing. + if _, err := c.From("events").Fetch(context.Background()); err != nil { + t.Fatal(err) + } } func TestClient_SQL(t *testing.T) { diff --git a/clients/go/http.go b/clients/go/http.go index d8a8b37b..9d149e90 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -174,6 +174,12 @@ func buildURL(base, path string, params url.Values) string { func retryAfterDelay(ra string, attempt int) time.Duration { delay := backoff(attempt) if secs, err := strconv.Atoi(ra); err == nil && secs > 0 { + // Compare before converting: time.Duration(secs) * time.Second wraps + // negative past ~9.2e9 seconds, and min() below would then pick the + // negative value, firing the retry timer instantly. + if secs > int(maxRetryAfter/time.Second) { + return maxRetryAfter + } delay = time.Duration(secs) * time.Second } else if parsed, err := http.ParseTime(ra); err == nil { if d := time.Until(parsed); d > 0 { diff --git a/clients/go/http_test.go b/clients/go/http_test.go index acf857d1..57109f3b 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -224,10 +224,19 @@ func TestBackoff(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { + got := backoff(tt.attempt) + if tt.name == "CappedAt30s" { + // The cap is applied *after* jitter, so this is exact — a ±20% + // window here would also accept capping before the jitter, + // which lets the documented 30s max drift to 36s. + if got != 30*time.Second { + t.Errorf("backoff(%d) = %v, want exactly 30s", tt.attempt, got) + } + return + } // backoff applies ±20% jitter around the exponential base. lo := time.Duration(float64(tt.base) * 0.8) hi := time.Duration(float64(tt.base) * 1.2) - got := backoff(tt.attempt) if got < lo || got > hi { t.Errorf("backoff(%d) = %v, want within [%v, %v]", tt.attempt, got, lo, hi) } @@ -243,6 +252,10 @@ func TestRetryAfterDelay(t *testing.T) { }{ {"DeltaSeconds", "5", 5 * time.Second}, {"ClampedToMax", "3600", maxRetryAfter}, + // time.Duration(secs) * time.Second wraps negative past ~9.2e9s; an + // unguarded min() then picks the negative and retries instantly. + {"OverflowClamped", "10000000000", maxRetryAfter}, + {"MaxIntClamped", "9223372036854775807", maxRetryAfter}, {"HTTPDateFuture", time.Now().Add(10 * time.Second).UTC().Format(http.TimeFormat), 0}, // range-checked below {"Garbage", "not-a-delay", 0}, // range-checked below } @@ -251,7 +264,9 @@ func TestRetryAfterDelay(t *testing.T) { got := retryAfterDelay(tt.ra, 0) switch tt.name { case "HTTPDateFuture": - if got <= 0 || got > 10*time.Second { + // Lower bound well clear of the backoff(0) fallback (~1s), so a + // broken HTTP-date branch can't pass by falling through to it. + if got < 5*time.Second || got > 10*time.Second { t.Fatalf("want ~10s, got %v", got) } case "Garbage": diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go index 6991371f..8d0e0212 100644 --- a/clients/go/query_builder.go +++ b/clients/go/query_builder.go @@ -4,6 +4,7 @@ import ( "bytes" "context" "encoding/json" + "fmt" "net/url" ) @@ -283,10 +284,19 @@ func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Ro // the same ceiling the TS SDK has with JS numbers. Use FetchTyped (or // codegen structs — their 64-bit int columns are int64/uint64, and // 128/256-bit are json.Number) when paging on >2^53 integer cursors. - raw, _ := json.Marshal(lastRow) + raw, err := json.Marshal(lastRow) + if err != nil { + // Row itself is unmarshalable (e.g. a func field absent from the + // response). Silently truncating the result set would look like + // normal end-of-pagination, so surface it. + return nil, fmt.Errorf("wavehouse: marshal cursor row: %w", err) + } m = make(map[string]any) dec := json.NewDecoder(bytes.NewReader(raw)) dec.UseNumber() + // Decode error is deliberate: a Row that marshals to a non-object + // (FetchTyped[[]any], a scalar row type) leaves m empty and ends + // pagination below, same as an absent cursor column. Tracked in #452. _ = dec.Decode(&m) } lastValue, exists := m[cursor.Column] diff --git a/clients/go/table.go b/clients/go/table.go index 204a3b26..e66f7f05 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -119,8 +119,10 @@ func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, e } func emptyInsertResult() *InsertResult { - z := 0 - return &InsertResult{OK: true, Total: &z, Succeeded: &z, Failed: &z, Duplicates: &z} + // Separate vars, not one aliased &z: the fields are exported *int, so a + // caller writing through one would otherwise mutate all four. + total, succeeded, failed, duplicates := 0, 0, 0, 0 + return &InsertResult{OK: true, Total: &total, Succeeded: &succeeded, Failed: &failed, Duplicates: &duplicates} } func marshalNDJSON(n int, elem func(int) any) (string, error) { diff --git a/clients/go/table_test.go b/clients/go/table_test.go index 2dfd0cff..652daf88 100644 --- a/clients/go/table_test.go +++ b/clients/go/table_test.go @@ -126,11 +126,14 @@ func TestTableRef_InsertTypedSlice(t *testing.T) { // insertSingle rather than being (mis)treated as a slice of per-byte rows. func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { var mu sync.Mutex - var gotPath string + var gotPath, gotCT, gotBody string c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { mu.Lock() defer mu.Unlock() gotPath = r.URL.Path + gotCT = r.Header.Get("Content-Type") + b, _ := io.ReadAll(r.Body) + gotBody = string(b) _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) })) result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) @@ -145,6 +148,18 @@ func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { if gotPath != "/v1/ingest" { t.Fatalf("want /v1/ingest, got %s", gotPath) } + // The batch path posts to the same URL and also yields ok=true, so the + // wire format is the only thing that distinguishes them: one opaque JSON + // value vs. NDJSON of 16 per-byte rows. + if gotCT != "application/json" { + t.Fatalf("want application/json (single insert), got %q", gotCT) + } + // encoding/json base64s a []byte — documented in queries.md as a value the + // server rejects (use InsertNDJSON for raw bytes). Pinned here because it + // proves the batch path wasn't taken. + if gotBody != `"eyJwYWdlIjoiL2hvbWUifQ=="` { + t.Fatalf("want single base64 value, got %q", gotBody) + } } func TestTableRef_InsertEmptyBatch(t *testing.T) { diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index d63c6715..8806e4cf 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -319,9 +319,15 @@ Execute the query and decode rows into `[]map[string]any`. The ordinary ```go page, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUntyped(ctx) +if err != nil { + return err +} if page.HasMore && page.Next != nil { - page2, err := page.Next(ctx) // cursor-based pagination — needs OrderBy + page, err = page.Next(ctx) // cursor-based pagination — needs OrderBy + if err != nil { + return err + } } ``` @@ -353,6 +359,13 @@ add an `.OrderBy()` to paginate. If the order column was left out of an explicit `.Select(...)` projection, `Next` quietly returns an empty page instead of erroring (there is no cursor value to read). +The cursor filter is strict (`gt`/`lt` against the last row's value) and uses +only that first `.OrderBy()` column, with no tie-breaker — so rows sharing the +boundary value with the last row of a page are skipped. Paginate on a column +that is unique per row (or made unique by a monotonic timestamp), or accept +that ties at a page edge can be dropped. The TypeScript SDK's `next()` has the +same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). + One precision caveat on the untyped path (`FetchUntyped` / `TableRef.Fetch`): rows decode into `map[string]any`, where JSON numbers become `float64`, so an integer cursor column loses exactness past 2^53 and pagination can repeat or @@ -387,7 +400,8 @@ for page.HasMore && page.Next != nil { Execute a raw SQL query. `/v1/admin/query` is admin-only: for JWT callers, the token must resolve to the policy admin role (`admin_role`, `"admin"` by default) — a JWT request with no token, or an invalid/expired one, falls -back to the `default_role` and is rejected. Alternatively, a configured +back to the `default_role`, and is rejected unless the deployment sets +`default_role` to the admin role (permitted, but dev-only). Alternatively, a configured operator key (`Authorization: Operator ` or `X-Operator-Key`) authorizes `/v1/admin/*` without a JWT — but note `Config.Auth` always sends its token as `Bearer `, so to use an operator key from this diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 35e0ba2f..6a8590d5 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -147,7 +147,7 @@ Generate Go structs from a running WaveHouse instance. The module ships a `wavehouse-codegen` command under `cmd/`: ```bash -export WAVEHOUSE_AUTH= # avoids leaking the token via argv +export WAVEHOUSE_AUTH='' # avoids leaking the token via argv go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --url http://localhost:8080 \ --out ./db_types.go \ @@ -263,7 +263,7 @@ E2E tests (build tag `e2e`) run against a live WaveHouse instance and have their own Make target, separate from the repo's `make test-e2e`: ```bash -WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH= make test-go-sdk-e2e +WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH='' make test-go-sdk-e2e ``` `WAVEHOUSE_URL` defaults to `http://localhost:8080`; `WAVEHOUSE_AUTH` is diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index 77aecbe1..5dd51b6c 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -259,7 +259,7 @@ while (result.hasMore && result.next) { ## Raw SQL — `wh.sql(query, opts?)` -Execute a raw SQL query. `/v1/admin/query` is admin-only: the caller's JWT must resolve to the policy admin role (`admin_role`, `"admin"` by default). A request with no token, or an invalid/expired one, falls back to the `default_role` and is rejected. +Execute a raw SQL query. `/v1/admin/query` is admin-only: the caller's JWT must resolve to the policy admin role (`admin_role`, `"admin"` by default). A request with no token, or an invalid/expired one, falls back to the `default_role`, and is rejected unless the deployment sets `default_role` to the admin role (permitted, but dev-only). ```ts const { data, error } = await wh.sql('SELECT page, count() FROM clicks GROUP BY page LIMIT 10'); From 9da52dd99604814a823c625f26901c95f2600e78 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 18:04:06 -0400 Subject: [PATCH 28/40] test(sdk): pin the fetchNextTyped marshal-error branch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The guard added in 0a46251 had no test — the same gap that commit existed to close. A Row that unmarshals cleanly but fails to marshal back (an exported func field, absent from the response) previously produced an empty page, indistinguishable from real exhaustion. Verified failing with the guard swallowing the error instead of returning it. --- clients/go/query_builder_test.go | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go index 02431746..7ccb9027 100644 --- a/clients/go/query_builder_test.go +++ b/clients/go/query_builder_test.go @@ -456,3 +456,26 @@ func TestQueryBuilder_Pagination_UntypedCursorFloat64Ceiling(t *testing.T) { t.Fatalf("untyped ceiling changed (update docs if intentional): %s", got) } } + +// The cursor round-trip re-marshals the last row to read its cursor value. A +// Row that unmarshals cleanly but can't be marshaled back (an exported func +// field, absent from the response) must surface an error rather than an empty +// page, which is indistinguishable from real exhaustion. +func TestQueryBuilder_Pagination_UnmarshalableRowErrors(t *testing.T) { + type row struct { + ID string `json:"id"` + Cb func() `json:"cb"` + } + c, _ := pagingServer(t, [][]map[string]any{{{"id": "a"}, {"id": "b"}}}) + page, err := FetchTyped[row](context.Background(), + c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2)) + if err != nil { + t.Fatal(err) + } + if page.Next == nil { + t.Fatal("want a Next cursor") + } + if _, err := page.Next(context.Background()); err == nil { + t.Fatal("want a marshal error, got a silently empty page") + } +} From 4cc229a36bacfb0ee232b0f82856c8342e20f8a7 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 18:08:13 -0400 Subject: [PATCH 29/40] docs: finish the TS side of the pagination parity sweep MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Go queries page documents two pagination footguns and names the TS SDK as sharing both, but the TS page carried neither — so a data-loss caveat lived only on the page TS readers don't open. Ports both: the strict single-column cursor that drops rows tying the page boundary (#452, verified against clients/ts/src/query-builder.ts), and the Number.MAX_SAFE_INTEGER ceiling on integer cursors. Also fills the not_like row's Backend column on the TS operator table — the SDK does send a wire token (query-builder.ts maps not_like to "not_like"), and a reader debugging the resulting 400 needs to know which token the server rejected. The Go table already said so. Two smaller corrections: reverse-proxy.mdx claimed the Go SDK "preserves the prefix on every released version", asserting a property of a set that development.md says is empty (no tagged Go releases yet); and the key-targets table under-reported make lint / verify / fix, all three of which fan out to markdown, prose, shell, workflow, and astro checks beyond what was listed. --- docs/src/content/docs/development.md | 6 +++--- docs/src/content/docs/reverse-proxy.mdx | 2 +- docs/src/content/docs/sdk/queries.md | 6 +++++- 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index b62d7635..88e1f262 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -483,10 +483,10 @@ Run `make help` to see all targets. Key ones: | **Static checks** | | | `make fmt` | Check formatting across root-module Go (`gofumpt`) + TS (Biome); the nested `clients/go` module's gofumpt check runs under `make verify` (`verify-go-sdk`). Run `make fix` to apply everywhere. | | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | -| `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) | +| `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) + Markdown (markdownlint) + docs prose (misspell) | | `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | -| `make verify` | Repo-wide static checks: root Go (tidy + fmt + vulncheck + lint), `clients/go` (fmt + vet + lint — no tidy/vulncheck: it's a nested module, invisible to the root-scoped `tidy`/`vulncheck` targets) + TS (Biome + `tsc` typecheck) (parallel-safe: `make -j verify`) | -| `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`) and TS (Biome `--write`) | +| `make verify` | Repo-wide static checks: root Go (tidy + fmt + vulncheck + lint), `clients/go` (fmt + vet + lint — no tidy/vulncheck: it's a nested module, invisible to the root-scoped `tidy`/`vulncheck` targets) + TS (Biome + `tsc` typecheck) + Markdown/prose, shell (shellcheck), workflows (actionlint), and `astro check` (parallel-safe: `make -j verify`) | +| `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`), TS (Biome `--write`), Markdown (markdownlint) + docs prose (misspell) | | **Build** | | | `make build` | Compile `wavehouse` → `bin/wavehouse` (debug symbols kept) | | `make build-release` | Stripped release-style build → `bin/wavehouse-release` | diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index c2ce9b60..94015a00 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -93,7 +93,7 @@ wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/wareh ``` :::caution[Check the prefix actually survives to the wire] -A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: a **TypeScript** SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). The Go SDK preserves the prefix on every released version. +A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: a **TypeScript** SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). The Go SDK has supported prefixed base URLs since its first release. ::: ## Request-body size limits diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index 5dd51b6c..0f6a8efd 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -152,7 +152,7 @@ clicks.select('page').where('score', '>', 10).where('page', 'like', '/home%') | `'<='` | `lte` | Less than or equal | | `'in'` | `in` | Value in array | | `'like'` | `like` | SQL LIKE pattern | -| `'not_like'` | — | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | +| `'not_like'` | `not_like` | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects the token | #### Aggregations @@ -255,6 +255,10 @@ while (result.hasMore && result.next) { } ``` +The cursor filter is strict (`gt`/`lt` against the last row's value) and uses only that one order column, with no tie-breaker — so rows sharing the boundary value with the last row of a page are skipped. Paginate on a column that is unique per row (or made unique by a monotonic timestamp), or accept that ties at a page edge can be dropped. The Go SDK's `Next` has the same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). + +Rows decode with JSON numbers as JS `number`s, so an integer cursor column past `Number.MAX_SAFE_INTEGER` (2^53) loses exactness and pagination can repeat or skip a row at that scale. The Go SDK's `FetchTyped` with an `int64` field avoids this; there is no JS equivalent short of a string or `bigint` column. + --- ## Raw SQL — `wh.sql(query, opts?)` From 6d29538357f0fe324d6adc576f40f8c12e4fc3fd Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Tue, 11 Aug 2026 18:19:52 -0400 Subject: [PATCH 30/40] docs: flag the TS codegen Decimal mismatch; fix the Go release framing The TypeScript codegen type table promised `number` for Decimal* columns, but /v1/query returns them as quoted strings: transformRow in internal/api/clickhouse_exec.go converts only UUID and time.Time, so a shopspring decimal.Decimal reaches json.Marshal and marshals quoted. A TS user with a Decimal price column gets a number-typed field holding "12.34" and silently wrong arithmetic, with no type error. The Go page already documented the real shape, which is what made the discrepancy visible. Footnoted both that row and Array(UInt8) (base64 on the same path, #436); the codegen fix itself is tracked in #453. Also corrects the path-prefix sentence I got wrong in both directions: it now says the prefix is preserved in every version `go get` can resolve, rather than asserting a release history that doesn't exist yet. --- docs/src/content/docs/reverse-proxy.mdx | 2 +- docs/src/content/docs/sdk/reference.md | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index 94015a00..6b1f58a7 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -93,7 +93,7 @@ wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/wareh ``` :::caution[Check the prefix actually survives to the wire] -A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: a **TypeScript** SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). The Go SDK has supported prefixed base URLs since its first release. +A prefix that the proxy forwards *unstripped* produces a clean `404` from WaveHouse — annoying, but loud. The quiet failure is the other side: a **TypeScript** SDK older than the prefix support silently dropped it and sent every request to the origin root. If requests are landing at `/v1/…` instead of `/api/warehouse/v1/…`, upgrade `@wavehouse/sdk` — the fix is unreleased, so it ships on the `@dev` tag until the next release ([#428](https://github.com/Wave-RF/WaveHouse/issues/428)). The Go SDK preserves the prefix in every version `go get` can resolve — it has behaved this way since its first commit (there are no tagged Go releases yet). ::: ## Request-body size limits diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index 92b63460..d7a6fdae 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -136,10 +136,11 @@ export interface ClicksRow { | ClickHouse Type | TypeScript Type | |----------------|-----------------| | `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum*`, `IPv4/6` | `string` | -| `UInt*`, `Int*`, `Float*`, `Decimal*` | `number` | +| `UInt*`, `Int*`, `Float*` | `number` | +| `Decimal*` | `number` *(generated)* — but `/v1/query` returns Decimals as **quoted strings**, so treat the field as `string` until codegen is fixed ([#453](https://github.com/Wave-RF/WaveHouse/issues/453)) | | `Bool` | `boolean` | | `Nullable(T)` | `T \| null` | -| `Array(T)` | `T[]` | +| `Array(T)` | `T[]` — except `Array(UInt8)`, which `/v1/query` base64-encodes, so the generated `number[]` is a `string` at runtime ([#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | | `Map(K, V)` | `Record` | | `LowCardinality(T)` | same as `T` | From 7b85e0687844d032aa439276ef582636a974b5db Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Wed, 12 Aug 2026 15:14:46 -0400 Subject: [PATCH 31/40] chore(docs): revising Go SDK documentation --- clients/go/README.md | 8 +- docs/src/content/docs/sdk/go/admin.md | 39 ++--- docs/src/content/docs/sdk/go/index.md | 111 ++++---------- docs/src/content/docs/sdk/go/pipes.md | 29 +--- docs/src/content/docs/sdk/go/queries.md | 173 +++++----------------- docs/src/content/docs/sdk/go/reference.md | 120 ++++----------- docs/src/content/docs/sdk/go/streaming.md | 141 +++++------------- 7 files changed, 143 insertions(+), 478 deletions(-) diff --git a/clients/go/README.md b/clients/go/README.md index 112aae2b..64e19cac 100644 --- a/clients/go/README.md +++ b/clients/go/README.md @@ -2,7 +2,7 @@ Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a schema-aware real-time API gateway for ClickHouse. -**Zero third-party runtime dependencies** — stdlib only. +**Zero third-party runtime dependencies** (stdlib only). **[Full SDK documentation on wavehouse.dev](https://wavehouse.dev/sdk/go)** @@ -80,7 +80,7 @@ client = wavehouse.NewClient(wavehouse.Config{ }) ``` -`BaseURL` may include a path prefix (`https://app.example.com/api/warehouse`) when WaveHouse is served under one. A trailing `/` is trimmed and every request path is appended to it, on both REST and SSE — see [Config](https://wavehouse.dev/sdk/go#config). +`BaseURL` may include a path prefix (`https://app.example.com/api/warehouse`). A trailing `/` is trimmed, and request paths are appended, for both REST and SSE alike ([Config](https://wavehouse.dev/sdk/go#config)). ## Typed Queries (Generics) @@ -206,7 +206,7 @@ See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference#c ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare `error` for operations with no result body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors originating from the HTTP exchange are `*wavehouse.Error` — unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. Streaming lifecycle methods (`Stream`, `Subscribe`, `Close`, and `Connected`) deliver errors through callbacks or plain errors instead: +Request-response ops return `(T, error)`, or bare `error` for body-less calls (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP errors are `*wavehouse.Error` (unwrap with `errors.As`); failures before the request goes out (`Auth` provider, body marshal) are plain wrapped errors, so handle `errors.As == false` too. Streaming lifecycle (`Stream`, `Subscribe`, `Close`, `Connected`) reports via callbacks or plain errors: ```go page, err := client.From("clicks").Fetch(ctx) @@ -218,7 +218,7 @@ if err != nil { } ``` -The HTTP layer retries 5xx, 429, and network errors with exponential backoff (default 2 retries). `Retry-After` on a 503 or 429 is honored, capped at 30s. Context cancellation returns immediately with code `ABORTED`. +The HTTP layer retries 5xx, 429, and network errors with exponential backoff (2 retries by default). `Retry-After` on 503/429 is honored, capped at 30s. Context cancellation returns `ABORTED` immediately. ## License diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md index 957713fd..65695dbf 100644 --- a/docs/src/content/docs/sdk/go/admin.md +++ b/docs/src/content/docs/sdk/go/admin.md @@ -3,15 +3,11 @@ title: "Go SDK Admin & System" description: "Schema introspection, access-control policy, DLQ stats, and health checks in the WaveHouse Go SDK." --- -Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. -Everything here except `client.Sys.Health` requires the admin role -(`policy.admin_role`) — see [Access Control](/access-control) for how roles -resolve. Compare with the TypeScript SDK's [Admin & System](/sdk/admin) -page. +Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. All except `client.Sys.Health` require the admin role (`policy.admin_role`)—see [Access Control](/access-control) and the TypeScript SDK's [Admin & System](/sdk/admin) page. ## Schema — `client.Schema` -Introspect ClickHouse table schemas. +Introspect ClickHouse table schemas. `Schema.List`, `Schema.Refresh`, and `From(t).Schema` hit the **admin-only** `/v1/schema*`; against any non-dev policy (anything but `default_role: admin`) build the client with an admin-role token or they return a `*wavehouse.Error` with `Status: 403`. ```go // List all table schemas. @@ -22,20 +18,13 @@ schemas, err := wh.Schema.List(ctx) err = wh.Schema.Refresh(ctx) ``` -Individual table schema is also available via `wh.From("clicks").Schema(ctx)`. - -> `wh.Schema.List`, `wh.Schema.Refresh`, and `wh.From(t).Schema` hit -> `/v1/schema*`, which are **admin-only** endpoints. Against any non-dev -> policy (anything but `default_role: admin`), construct the client with an -> admin-role token or these calls return a `*wavehouse.Error` with -> `Status: 403`. +Individual table schema: `wh.From("clicks").Schema(ctx)`. --- ## Policy — `client.Policy` -Manage Hasura-style access control policies. Requires the admin role -(`policy.admin_role`). +Manage Hasura-style access control policies (admin role required). ```go // Get current policy. @@ -66,10 +55,7 @@ result, err := wh.Policy.Validate(ctx, policyDraft) // result.Valid == true, or err wraps the validation failure details ``` -`PolicyFilter`'s fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string`, not -`string` — an intentional empty-string comparison round-trips distinctly -from an absent operator. Take the address of a local variable (as above) or -write a small helper if you find yourself doing this often: +`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string` to distinguish empty strings from absent operators. Use a helper: ```go func strPtr(s string) *string { return &s } @@ -79,7 +65,7 @@ func strPtr(s string) *string { return &s } ## DLQ — `client.DLQ` -Dead Letter Queue operations. Requires the admin role (`policy.admin_role`). +Dead Letter Queue operations (admin role required). ```go // Get DLQ statistics. @@ -91,17 +77,13 @@ stats, err := wh.DLQ.List(ctx) stats, err = wh.DLQ.Table(ctx, "clicks") ``` -`wh.DLQ.Stream(opts)` exists in the API but is **not yet functional**: -there is no server-side DLQ stream today (the SSE bridge only carries -`ingest.>` subjects), so it connects and receives no events — live DLQ -streaming is tracked in -[#197](https://github.com/Wave-RF/WaveHouse/issues/197). +`wh.DLQ.Stream(opts)` is **not yet functional**: no server-side DLQ stream exists (the SSE bridge carries only `ingest.>` subjects), so it connects and receives nothing. Tracked in [#197](https://github.com/Wave-RF/WaveHouse/issues/197). --- ## System — `client.Sys` -Content-free server-online check. +Server-online check. ```go // Health hits the public, content-free /v1/health route — 200 → nil error, @@ -113,7 +95,4 @@ if err := wh.Sys.Health(ctx); err != nil { } ``` -> Readiness (`/readyz`) is intentionally **not** exposed through the SDK — -> it runs a ClickHouse query per call and is a load-balancer / reverse-proxy -> concern, not the client's. Probe `/readyz` directly from your -> orchestrator if you need it. +> Readiness (`/readyz`) is intentionally **not** exposed through the SDK — it runs a ClickHouse query per call and is a load-balancer / reverse-proxy concern. Probe it directly from your orchestrator. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index c70e6fd9..f2a4fc7f 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -20,7 +20,7 @@ either page mostly carries over. go get github.com/Wave-RF/WaveHouse/clients/go ``` -Requires Go 1.24 or later (the module's `go.mod` floor — deliberately a supported-releases floor rather than the server's patch-pinned toolchain). +Requires Go 1.24+ (the `go.mod` floor, matching supported releases rather than server's patch-pinned toolchain). ## Import @@ -28,9 +28,7 @@ Requires Go 1.24 or later (the module's `go.mod` floor — deliberately a suppor import wavehouse "github.com/Wave-RF/WaveHouse/clients/go" ``` -The package name is `wavehouse`; aliasing the import isn't required, but -keeps call sites short (`wavehouse.NewClient(...)`, `wavehouse.OpEq`, ...) — -every example on these pages assumes it. +Aliasing `wavehouse` is optional but keeps call sites short; all examples here assume it. ## Quick Start @@ -65,7 +63,7 @@ func main() { } ``` -See the [README](https://github.com/Wave-RF/WaveHouse/blob/main/clients/go/README.md) for more quick-start examples. +Find more examples in the [README](https://github.com/Wave-RF/WaveHouse/blob/main/clients/go/README.md). ## Creating a Client @@ -85,42 +83,28 @@ wh := wavehouse.NewClient(wavehouse.Config{ | Field | Type | Default | Description | |-------|------|---------|-------------| -| `BaseURL` | `string` | — | WaveHouse server URL, optionally including a path prefix (required). A trailing `/` is trimmed; every request path is appended to it on both transports, so a WaveHouse served under `https://app.example.com/wavehouse` works as-is. | -| `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider, called before each request. `nil` means unauthenticated access | -| `Options` | `*ClientOptions` | `nil` | Transport tuning (see below) | -| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports (see caution below) | +| `BaseURL` | `string` | — | Required WaveHouse server URL, optionally with a path prefix. A trailing `/` is trimmed and every request path is appended on both transports, so a server under `https://app.example.com/wavehouse` works as-is. | +| `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider called before each request. `nil` means unauthenticated access. | +| `Options` | `*ClientOptions` | `nil` | Transport tuning (see below). | +| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports. | :::caution[Timeouts: use contexts, not `http.Client.Timeout`] -The default client sets no `Timeout` — a `context.Context` deadline is the -only bound on a request, so pass one for anything that mustn't hang on a -stalled server. If you supply your own `HTTPClient`, leave `Timeout` unset: -it covers body reads too, so it would kill every long-lived SSE stream at -the timeout and force a reconnect loop. Use `Transport`-level dial / -TLS / response-header timeouts instead. +The default client has no `Timeout`; use a `context.Context` deadline to prevent hangs. If supplying your own `HTTPClient`, leave `Timeout` unset, as it would kill long-lived SSE streams and force reconnect loops. Use `Transport`-level dial/TLS/response-header timeouts instead. ::: ### `ClientOptions` | Field | Type | Default | Description | |-------|------|---------|-------------| -| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures) | +| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures). | -A `*Client` is safe for concurrent use by multiple goroutines — client state -is immutable after `NewClient`, and every builder chain copies. Supply a -concurrency-safe `Auth` func (it's called from any goroutine that issues a -request). +`*Client` is safe for concurrent use; state is immutable after `NewClient` and builder chains copy. Ensure your `Auth` function is concurrency-safe. :::caution[`Options` opts you out of the default, not just in] -The default of 2 retries only applies when `Config.Options` is `nil`. If you -set `Options` to configure anything else in the future, an unset -`MaxRetries` field is Go's int zero value — `0` — which is a **valid, -explicit** "no retries" setting, not "use the default." Today `MaxRetries` -is the struct's only field, so this mostly matters if you pass -`&wavehouse.ClientOptions{}` and expect retry-by-default: you won't get it. +The 2-retry default only applies if `Config.Options` is `nil`. If `Options` is provided, an unset `MaxRetries` field defaults to Go's int zero value (`0`), which explicitly disables retries. Passing `&wavehouse.ClientOptions{}` removes the default retry behavior. ::: -For a static token that never rotates, use `wavehouse.StaticToken(token)` -instead of writing the closure yourself: +For static tokens, use `wavehouse.StaticToken(token)`: ```go wh := wavehouse.NewClient(wavehouse.Config{ @@ -130,24 +114,16 @@ wh := wavehouse.NewClient(wavehouse.Config{ ``` :::note[How the token is transmitted] -Unlike a browser's `EventSource`, Go's `net/http` client can set arbitrary -headers on any request — so the Go SDK sends `Authorization: Bearer ` -on **every** request, including SSE streams. There's no `?token=` query -parameter fallback to worry about (that's a TypeScript-SDK-in-the-browser -concern only; see its [equivalent note](/sdk#creating-a-client)). +Unlike a browser's `EventSource`, Go's `net/http` client sets arbitrary headers on any request, so the Go SDK sends `Authorization: Bearer ` on every request, including SSE streams. No `?token=` query fallback (a TypeScript-in-the-browser concern; see its [equivalent note](/sdk#creating-a-client)). ::: :::caution[Use HTTPS for authenticated non-local servers] -The SDK doesn't forbid `http://` base URLs — local development and -private-network deployments rely on them — but a bearer token sent over -plaintext HTTP is readable by anything on the path. Point authenticated -clients at `https://` endpoints outside a trusted network. +While the SDK allows `http://` for local development or private networks, bearer tokens over plaintext HTTP are insecure. Use `https://` for endpoints outside trusted networks. ::: ## Typed Rows (Generics) -Pass a row type as a type parameter to get results decoded straight into -your struct, instead of `map[string]any`: +Pass a row type parameter to decode results into your struct instead of `map[string]any`: ```go type ClickRow struct { @@ -162,19 +138,13 @@ page, err := wavehouse.FetchTyped[ClickRow](ctx, // page.Data is []ClickRow ``` -Generate row structs from a running server with the -[codegen CLI](/sdk/go/reference#codegen-cli). +Use the [codegen CLI](/sdk/go/reference#codegen-cli) to generate row structs from a running server. -`FetchTyped` is a package-level generic function, not a method — Go doesn't -support generic methods, so this (and `Fetch[Row]` for pipes, and -`SQL[Row]` for raw SQL) are top-level functions that take the client or -builder as an argument. Untyped equivalents (`.FetchUntyped(ctx)`, decoding -into `map[string]any`) are ordinary methods, since they need no type -parameter. +`FetchTyped`, `Fetch[Row]` (pipes), and `SQL[Row]` (raw SQL) are package-level generic functions because Go lacks generic methods. Untyped equivalents (`.FetchUntyped(ctx)`) are ordinary methods. ## Error Handling -Every request-response operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare `error` for operations with no result body (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors originating from the HTTP exchange are `*wavehouse.Error`; unwrap with `errors.As`. Client-side failures before a request goes out (an `Auth` provider error, a request-body marshal failure) are plain wrapped errors, so handle the `errors.As == false` case too. (Streaming lifecycle methods — `Stream`, `Subscribe`, `Close`, `Connected` — deliver errors through callbacks or plain errors instead; see [Streaming](/sdk/go/streaming).) +Request-response operations (queries, ingest, pipes, admin) return `(T, error)` or just `error` if no body exists (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP exchange errors are `*wavehouse.Error`; unwrap via `errors.As`. Client-side failures (e.g., `Auth` provider, marshal errors) are plain wrapped errors; handle the `errors.As == false` case. Streaming methods (`Stream`, `Subscribe`, `Close`, `Connected`) use callbacks or plain errors; see [Streaming](/sdk/go/streaming). ```go page, err := wh.From("clicks").Fetch(ctx) @@ -193,40 +163,19 @@ See [Reference → Error Handling](/sdk/go/reference#error-handling) for retry b ## Differences from the TypeScript SDK -The two SDKs share a wire format and mirror each other's feature set closely -(a shared `wire_cases.json` conformance fixture is replayed by a test runner -per SDK — both run in CI — asserting each produces the expected HTTP request -for equivalent builder calls), but the -languages pull the API shape in different directions: - -- **No `Result` union.** Go returns `(T, error)`; nothing is wrapped in - an `{ok, data, error}` object, and there's no `error: null` sentinel to - check — a non-nil `error` is the only signal. -- **`context.Context` instead of `AbortSignal`.** Every non-streaming call - takes a `ctx context.Context` as its first argument; cancel it (timeout or - `cancel()`) instead of building an `AbortController`. See - [Reference → Context Cancellation](/sdk/go/reference#context-cancellation). -- **Streams are closed explicitly, not via `ctx`.** `TableRef.Stream` / - `QueryBuilder.Stream` don't take a `context.Context` — the returned - `*StreamController` manages its own background goroutine and connection, - torn down by calling `.Close()` (deferred `stream.Close()` is the usual - pattern). See [Streaming](/sdk/go/streaming). -- **Generics live on package-level functions, not methods** (`FetchTyped[Row]`, - `Fetch[Row]`, `SQL[Row]`), because Go doesn't support type parameters on - methods. -- **No implicit "await."** A `QueryBuilder` isn't `PromiseLike` — call - `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` - explicitly; there's no bare `await builder` shortcut. -- **Any slice batches, not just `[]map[string]any`.** Go detects slice-ness - via reflection, so `[]ClickRow{...}` takes the same NDJSON batch path as - `[]map[string]any` (the TS SDK's `insert` likewise accepts arrays of typed - rows — this bullet is about the Go mechanics, not a TS gap) — see - [Queries → Insert](/sdk/go/queries#insertctx-data). +Both SDKs share a wire format and feature set, verified by a shared `wire_cases.json` fixture in CI to ensure equivalent HTTP requests for builder calls. However, API shapes differ: + +- **No `Result` union.** Go returns `(T, error)`. A non-nil `error` is the only failure signal; no `{ok, data, error}` objects or `error: null` sentinels are used. +- **`context.Context` instead of `AbortSignal`.** Non-streaming calls take `ctx context.Context` as the first argument. Use timeout or `cancel()` instead of `AbortController`. See [Reference → Context Cancellation](/sdk/go/reference#context-cancellation). +- **Streams closed explicitly.** `TableRef.Stream` and `QueryBuilder.Stream` omit `context.Context`. The returned `*StreamController` manages its own goroutine and connection, torn down by `.Close()` (deferred `stream.Close()` is usual). See [Streaming](/sdk/go/streaming). +- **Generics on package functions.** Go lacks type parameters on methods; use `FetchTyped[Row]`, `Fetch[Row]`, or `SQL[Row]`. +- **No implicit "await."** Call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly; `QueryBuilder` is not `PromiseLike`. +- **Any slice batches.** Reflection allows `[]ClickRow{...}` to use the same NDJSON batch path as `[]map[string]any`. See [Queries → Insert](/sdk/go/queries#insertctx-data). ## Explore the Go SDK -- [Queries](/sdk/go/queries) — Tables, the chainable query builder, pagination, and raw SQL. -- [Streaming & Live Queries](/sdk/go/streaming) — Real-time SSE streams, client-side filtering, and backfill-then-live queries. -- [Pipes](/sdk/go/pipes) — Execute and manage named query pipes. +- [Queries](/sdk/go/queries) — Tables, chainable query builder, pagination, and raw SQL. +- [Streaming & Live Queries](/sdk/go/streaming) — SSE streams, client-side filtering, and backfill-then-live queries. +- [Pipes](/sdk/go/pipes) — Manage named query pipes. - [Admin & System](/sdk/go/admin) — Schema introspection, access-control policy, DLQ stats, and health checks. -- [Reference & CLI](/sdk/go/reference) — Error codes, context cancellation, the full API tree, and the codegen CLI. +- [Reference & CLI](/sdk/go/reference) — Error codes, context cancellation, API tree, and codegen CLI. diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md index 44786601..69e426a1 100644 --- a/docs/src/content/docs/sdk/go/pipes.md +++ b/docs/src/content/docs/sdk/go/pipes.md @@ -3,16 +3,11 @@ title: "Go SDK Pipes" description: "Execute and manage named query pipes with the WaveHouse Go SDK." --- -Named pipes are server-defined, parameterized queries — the -[Named Pipes guide](/pipes) covers defining them. The SDK executes pipes for -any allowed role and manages their definitions under the admin role. -Compare with the TypeScript SDK's [Pipes](/sdk/pipes) page. +Named pipes are server-defined, parameterized queries ([Named Pipes guide](/pipes)). The SDK executes them for allowed roles and manages definitions under the admin role. Compare with the TypeScript SDK's [Pipes](/sdk/pipes) page. ## Named Pipes — `client.Pipe(name, params)` -Execute a pre-defined named query pipe. Returns a `*PipeRef`; unlike the -TypeScript SDK's `PipeRef` (which is `PromiseLike`), you always call -`.FetchUntyped(ctx)` or the package-level `wavehouse.Fetch[Row]` explicitly. +Execute a pre-defined named query pipe. Returns a `*PipeRef`. Unlike the TypeScript SDK's `PromiseLike` `PipeRef`, you must explicitly call `.FetchUntyped(ctx)` or the package-level `wavehouse.Fetch[Row]`. ```go rows, err := wavehouse.Fetch[map[string]any](ctx, @@ -22,9 +17,7 @@ rows, err := wavehouse.Fetch[map[string]any](ctx, ### `wavehouse.Fetch[Row](ctx, pipeRef)` -Execute and decode results into `[]Row`. Package-level generic function -(Go has no generic methods) — the same pattern as `FetchTyped` for queries -and `SQL` for raw SQL. +Execute and decode results into `[]Row`. Package-level generic function (Go has no generic methods) — same pattern as `FetchTyped` for queries and `SQL` for raw SQL. ```go type TopPage struct { @@ -37,24 +30,17 @@ rows, err := wavehouse.Fetch[TopPage](ctx, wh.Pipe("top_pages", map[string]any{" ### `.FetchUntyped(ctx)` -Execute and decode results into `[]map[string]any`. The ordinary -(non-generic) method form of `Fetch`. +Execute and decode results into `[]map[string]any`. The non-generic method form of `Fetch`. ```go rows, err := wh.Pipe("top_pages", nil).FetchUntyped(ctx) ``` -Pass `nil` for `params` when the pipe takes none, or the pipe requires only -parameters with server-side defaults. +Pass `nil` for `params` if the pipe takes none or only requires server-side defaults. ### `.Stream(opts)` -Open a live stream from the pipe's underlying query. See -[Streaming](/sdk/go/streaming). - -This streams by table name, using the pipe's own name as the table — it -only works when the pipe name is also a valid table name. This matches the -TypeScript SDK's `PipeRef.stream()`, which has the same limitation. +Open a live stream from the pipe's underlying query; see [Streaming](/sdk/go/streaming). Streams by table name using the pipe's own name, so it works only when that name is a valid table name — the same limitation as the TypeScript SDK's `PipeRef.stream()`. ```go stream := wh.Pipe("top_pages", nil).Stream(nil) @@ -87,8 +73,7 @@ err = wh.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ err = wh.Pipes.Delete(ctx, "old_pipe") ``` -`PipeDef` is `Pipe` minus the `Name` field — the name is already in the -`Set`/`Get`/`Delete` call's path argument: +`PipeDef` is `Pipe` minus `Name` — the name is the method's path argument: ```go type PipeDef struct { diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 8806e4cf..337cc042 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -14,8 +14,7 @@ chainable builder methods and `.Stream(opts)` are the exceptions — see ## Tables — `client.From(table)` -`From` returns a `*TableRef` — a reference to a table. It performs no -request by itself, so it's safe to store in a variable or pass around. +`From` returns a `*TableRef`. It performs no request, making it safe to store or pass around. ```go clicks := wh.From("clicks") @@ -23,17 +22,9 @@ clicks := wh.From("clicks") ### `.Fetch(ctx)` -Shortcut for "select every column", with a default limit of 1000 -(`wavehouse.DefaultLimit`). Internally it's -`t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)` — unlike the -TypeScript SDK's `.fetch(opts?)`, there's no options struct to override the -limit or attach anything per-call; chain `.SelectAll().Limit(n)` yourself -(see [Query Builder](#query-builder)) if you need a different limit. +Shortcut for "select every column" with a default limit of 1000 (`wavehouse.DefaultLimit`). Internally it is `t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)`. Unlike the TypeScript SDK's `.fetch(opts?)`, there is no options struct to override the limit or attach anything per-call; chain `.SelectAll().Limit(n)` yourself ([Query Builder](#query-builder)). -When an access-control policy restricts your role's columns, the server -returns only the columns your role is allowed to read — `.Fetch()` is never -a way around `deny_columns`/`allow_columns` (see -[Access control](/access-control#column-permissions)). +Access-control policies restrict returned columns; `.Fetch()` cannot bypass `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). ```go page, err := clicks.Fetch(ctx) @@ -45,23 +36,14 @@ for _, row := range page.Data { } ``` -To paginate, use the query builder with an explicit `.OrderBy()` instead — -see [Pagination](#pagination). +For pagination, use the query builder with `.OrderBy()` (see [Pagination](#pagination)). ### `.Insert(ctx, data)` -Insert one row or many. What you pass determines the wire format: +Inserts one or many rows based on the input type: -- A single **map or struct** (anything that isn't a slice — plus `[]byte`, - which is treated as one opaque value rather than a batch of numbers, and - would reach the server as a base64 string it rejects; pass raw NDJSON - through `.InsertNDJSON(ctx, string(raw))` instead) is sent as JSON: - `POST /v1/ingest?table={table}`. -- **Any slice** — `[]map[string]any`, a generated/user-defined row type like - `[]ClickRow`, etc. — is serialized to NDJSON (one record per line, via - reflection for non-`[]map[string]any` slices) and sent as a single - `application/x-ndjson` request, so a bad record doesn't fail or hide the - rest of the batch. Per-record outcomes come back in the result. +- **Map or struct** (excluding slices and `[]byte`): Sent as JSON via `POST /v1/ingest?table={table}`. For raw NDJSON, use `.InsertNDJSON`. +- **Any slice** (`[]map[string]any`, `[]ClickRow`, etc.): Serialized to NDJSON via reflection and sent as one `application/x-ndjson` request. Per-record outcomes are returned in the result. ```go // Single row → InsertResult{OK: true} (or Duplicate: &true when dedup skips it) @@ -85,23 +67,13 @@ res, err = clicks.Insert(ctx, []ClickRow{ }) ``` -For a batch insert, `res.OK` is `true` only when every record succeeded -(`*res.Failed == 0`). Inspect `res.Failed` and `res.Results` (each -`InsertRecordResult{Index, OK, Duplicate, Error}`, 1-based `Index`) for -partial failures — the returned `error` is reserved for whole-request -failures (network, `404` unknown table, `403` forbidden, `503` -backpressure). An empty slice is a no-op and sends no request. +For batches, `res.OK` is `true` only if all records succeeded (`*res.Failed == 0`). Check `res.Failed` and `res.Results` (each `InsertRecordResult{Index, OK, Duplicate, Error}`, 1-based `Index`) for partial failures. The returned `error` indicates whole-request failures (network, `404`, `403`, `503`). Empty slices are no-ops. -> The server itself is format-agnostic: `POST /v1/ingest` also accepts a raw -> JSON array or a single object directly (the `Content-Type` is only a -> hint), so non-SDK clients can send whichever shape is convenient. See the -> [API reference](/api#post-v1ingesttabletable--ingest-data). +> The server is format-agnostic: `POST /v1/ingest` also accepts a raw JSON array or a single object (`Content-Type` is only a hint). See [API reference](/api#post-v1ingesttabletable--ingest-data). ### `.InsertNDJSON(ctx, ndjson)` -Insert pre-formatted NDJSON you already have, as a plain `string` — a file -you've read, or a string you built yourself — without first parsing it into -Go values. Returns the same per-record summary as a slice `Insert`. +Inserts pre-formatted NDJSON as a `string` without parsing it into Go values. Returns the same summary as slice `Insert`. ```go // From a literal string. @@ -117,7 +89,7 @@ res, err = clicks.InsertNDJSON(ctx, string(raw)) ### `.Schema(ctx)` -Fetch the table's column definitions from ClickHouse. Admin-only. +Fetch table column definitions from ClickHouse. Admin-only. ```go schema, err := clicks.Schema(ctx) @@ -138,13 +110,7 @@ page, err := clicks.Select("page", "button"). ### `.SelectAll()` -Start a query that selects **every column your role is allowed to read** — -the explicit form of what `.Fetch()` does. Mutually exclusive with -`.Select(...)` and with aggregations (`.Count()`, `.Sum()`, etc.); for a -column-restricted role the server expands it to exactly that role's allowed -columns rather than a bare `SELECT *` (unrestricted/admin roles do get -`SELECT *`), and it never bypasses `deny_columns`/`allow_columns`. See -[Access control → Column permissions](/access-control#column-permissions). +Selects every column your role is allowed to read. This is the explicit version of `.Fetch()`. It is mutually exclusive with `.Select(...)` and aggregations (`.Count()`, `.Sum()`). For restricted roles, the server expands this to allowed columns rather than a bare `SELECT *`; it never bypasses `deny_columns`/`allow_columns` (see [Access control → Column permissions](/access-control#column-permissions)). ```go page, err := clicks.SelectAll().Where("country", wavehouse.OpEq, "US").Limit(10).FetchUntyped(ctx) @@ -158,15 +124,9 @@ Open a real-time event subscription. See [Streaming](/sdk/go/streaming). stream := clicks.Stream(&wavehouse.StreamOptions{Since: "2026-01-01T00:00:00Z"}) ``` ---- - ## Query Builder -Returned by `tableRef.Select()`. Immutable — every chain method returns a -new `*QueryBuilder`, so intermediate values can be reused safely. Unlike the -TypeScript SDK's `PromiseLike` builder, a Go `*QueryBuilder` doesn't -auto-execute — call `.FetchUntyped(ctx)` or the package-level -`wavehouse.FetchTyped[Row](ctx, builder)` explicitly: +Returned by `tableRef.Select()`. Immutable—every chain method returns a new `*QueryBuilder`. Unlike the TypeScript SDK, Go builders do not auto-execute; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly: ```go page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) @@ -174,12 +134,11 @@ page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) ### Chain Methods -All methods return a new `*QueryBuilder` — the original is unchanged. +All methods return a new `*QueryBuilder`; the original remains unchanged. #### `.Select(...columns)` -Append columns to the SELECT clause. A literal `"*"` is the column *named* -`*`, not a wildcard — use `.SelectAll()` for all columns. +Append columns to the SELECT clause. A literal `"*"` is treated as a column named `*`—use `.SelectAll()` for all columns. ```go q := clicks.Select("page").Select("button") // SELECT page, button @@ -187,10 +146,7 @@ q := clicks.Select("page").Select("button") // SELECT page, button #### `.SelectAll()` -Select every column your role may read (the all-columns wildcard; a -column-restricted role's projection is expanded server-side to its allowed -columns). Mutually exclusive with `.Select(...)` and with aggregations -(`.Count()`, `.Sum()`, etc.). +Selects every readable column (expanded server-side based on role). Mutually exclusive with `.Select(...)` and aggregations (`.Count()`, `.Sum()`, etc.). ```go q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") @@ -198,7 +154,7 @@ q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") #### `.Where(column, op, value)` -Add a filter condition, using the `FilterOp` constants: +Add a filter using `FilterOp` constants: ```go clicks.Select("page"). @@ -214,9 +170,9 @@ clicks.Select("page"). | `wavehouse.OpGte` | `gte` | Greater than or equal | | `wavehouse.OpLt` | `lt` | Less than | | `wavehouse.OpLte` | `lte` | Less than or equal | -| `wavehouse.OpIn` | `in` | Value in array — accepts a Go slice of any element type (`[]string`, `[]int`, `[]any`, ...) | +| `wavehouse.OpIn` | `in` | Value in array (accepts any Go slice) | | `wavehouse.OpLike` | `like` | SQL LIKE pattern | -| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects the token | +| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE — **client-side only**; `/v1/query` rejects this token | #### Aggregations @@ -231,19 +187,9 @@ clicks.Select("page"). Aggregate("uniqExact", "user_id", "unique_users") // allowlisted fn ``` -Custom function names pass through `.Aggregate(fn, column, alias)` but are -validated server-side against a fixed allowlist (matched case-insensitively): -`count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, -`any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, -`stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Anything else is rejected -with `400 unsupported aggregation function`. +Custom functions via `.Aggregate(fn, column, alias)` are validated server-side (case-insensitive). Allowlist: `count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, `any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, `stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Others return `400 unsupported aggregation function`. -`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias -string)`; `Aggregate` takes `(fn, column, alias string)`. Empty-alias -defaults: `Count` → `count` (and `column=""` becomes `*`); `Sum`/`Avg`/ -`Min`/`Max` → `sum_`/`avg_`/`min_`/`max_`; -`CountDistinct` → `count_distinct_`. `Aggregate` has **no** alias -default — pass one explicitly or the query is sent with `"alias": ""`. +`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias)`; `Aggregate` takes `(fn, column, alias)`. Empty-alias defaults: `Count` → `count` (and `column=""` becomes `*`); `Sum`/`Avg`/`Min`/`Max` → `sum_`/`avg_`/`min_`/`max_`; `CountDistinct` uses `count_distinct_`. `Aggregate` has no default; pass one or it is sent as `""`. #### `.GroupBy(...columns)` @@ -257,7 +203,7 @@ clicks.Select("page").Count("", "").GroupBy("page") clicks.Select("page").Count("", "total").OrderBy("total", "desc") ``` -`dir` defaults to `"asc"` when passed as `""`. +`dir` defaults to `"asc"` if `""`. #### `.Limit(n)` @@ -265,16 +211,11 @@ clicks.Select("page").Count("", "total").OrderBy("total", "desc") clicks.Select().Limit(100) ``` -If no limit is specified, `wavehouse.DefaultLimit` (1000) is applied -automatically to prevent unbounded result sets. The server also enforces the -configured maximum (`query.default_max_rows`, default 10,000 rows). +If unspecified, `wavehouse.DefaultLimit` (1000) is applied. The server also enforces a maximum (`query.default_max_rows`, default 10,000). #### `.TimeRange(column, since, until)` -Filter by a time window. `since` and `until` accept RFC3339 timestamps or -relative durations (`"1h"`, `"30m"`, `"7d"`, `"2w"` — day and week suffixes -expand to hours, so `"7d"` is `"168h"`). Pass `""` for `until` to leave it -open-ended. +Filter by time window. `since`/`until` accept RFC3339 timestamps or relative durations (`"1h"`, `"30m"`, `"7d"`, `"2w"`; day/week suffixes expand to hours, so `"7d"` is `"168h"`). Pass `""` for `until` for open-ended ranges. ```go clicks.Select("page").TimeRange("received_timestamp", "1h", "") @@ -285,11 +226,7 @@ clicks.Select("page").TimeRange( #### `.CacheTTL(seconds)` -Records a desired result-cache TTL on the builder. **Currently client-side -state only** — the value is never sent to the server, which derives each -result's cache TTL adaptively from query execution time. Wiring it through -the wire format is tracked in -[#280](https://github.com/Wave-RF/WaveHouse/issues/280). +Sets a desired result-cache TTL. **Currently client-side only**; the server derives TTL adaptively from execution time. See [#280](https://github.com/Wave-RF/WaveHouse/issues/280). ```go clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side — see #280 @@ -297,8 +234,7 @@ clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side ### `wavehouse.FetchTyped[Row](ctx, q)` -Execute the query and decode rows into `[]Row`. Package-level generic -function (Go has no generic methods) — takes the builder as its argument. +Executes the query and decodes rows into `[]Row`. ```go type PageCount struct { @@ -314,8 +250,7 @@ page, err := wavehouse.FetchTyped[PageCount](ctx, ### `.FetchUntyped(ctx)` -Execute the query and decode rows into `[]map[string]any`. The ordinary -(non-generic) method form of `FetchTyped`. +Executes the query and decodes rows into `[]map[string]any`. ```go page, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUntyped(ctx) @@ -333,9 +268,7 @@ if page.HasMore && page.Next != nil { ### `.Stream(opts)` -Open a live stream from the builder's table, applying `.Where()`/`.Select()` -filters and column projection client-side. See -[Streaming](/sdk/go/streaming). +Opens a live stream from the builder's table with client-side filtering and projection. See [Streaming](/sdk/go/streaming). ### Pagination @@ -349,30 +282,11 @@ type Page[T any] struct { } ``` -When `Limit` is set and the result contains at least that many rows, -`HasMore` is `true`. Cursor-based pagination's `Next` walks the **first** -`.OrderBy()` column — it adds a filter on that column using the last row's -value — so `Next` is only attached when the query has an explicit -`.OrderBy()`. With no order column the result still reports `HasMore` -honestly, but `Next` is `nil` (there is no deterministic cursor to build) — -add an `.OrderBy()` to paginate. If the order column was left out of an -explicit `.Select(...)` projection, `Next` quietly returns an empty page -instead of erroring (there is no cursor value to read). - -The cursor filter is strict (`gt`/`lt` against the last row's value) and uses -only that first `.OrderBy()` column, with no tie-breaker — so rows sharing the -boundary value with the last row of a page are skipped. Paginate on a column -that is unique per row (or made unique by a monotonic timestamp), or accept -that ties at a page edge can be dropped. The TypeScript SDK's `next()` has the -same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). - -One precision caveat on the untyped path (`FetchUntyped` / `TableRef.Fetch`): -rows decode into `map[string]any`, where JSON numbers become `float64`, so an -integer cursor column loses exactness past 2^53 and pagination can repeat or -skip a row at that scale — the same ceiling the TypeScript SDK has with JS -numbers. `FetchTyped` with an `int64` field keeps the cursor exact, and -codegen structs are unaffected (their 64-bit integer columns are `int64`/ -`uint64`, decoded exactly). +If `Limit` is set and results meet that limit, `HasMore` is `true`. `Next` walks the **first** `.OrderBy()` column using a filter on the last row's value; thus, `Next` requires an explicit `.OrderBy()`. Without one, `Next` is `nil`. If the order column is omitted from `.Select(...)`, `Next` returns an empty page. + +The cursor filter is strict (`gt`/`lt` on the first `.OrderBy()` column, no tie-breaker), so rows sharing a boundary value with the last row are skipped. Paginate on a per-row-unique column, or accept dropped ties; the TypeScript SDK's `next()` has the same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). + +On the untyped path (`FetchUntyped` / `TableRef.Fetch`), JSON numbers decode as `float64`, so integer cursors lose exactness past 2^53 and pagination can repeat or skip a row. `FetchTyped` with an `int64` field, or codegen structs, keep it exact. ```go page, err := clicks.Select(). @@ -393,21 +307,9 @@ for page.HasMore && page.Next != nil { } ``` ---- - ## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` -Execute a raw SQL query. `/v1/admin/query` is admin-only: for JWT callers, -the token must resolve to the policy admin role (`admin_role`, `"admin"` by -default) — a JWT request with no token, or an invalid/expired one, falls -back to the `default_role`, and is rejected unless the deployment sets -`default_role` to the admin role (permitted, but dev-only). Alternatively, a configured -operator key (`Authorization: Operator ` or `X-Operator-Key`) -authorizes `/v1/admin/*` without a JWT — but note `Config.Auth` always -sends its token as `Bearer `, so to use an operator key from this -SDK supply a `Config.HTTPClient` whose `Transport` sets the -`X-Operator-Key` header on each request. Package-level generic function — -use `map[string]any` for a dynamic/unknown schema. +Execute a raw SQL query via `/v1/admin/query`. This endpoint is admin-only: JWT tokens must resolve to the admin role (`admin_role`, default `"admin"`). Requests without valid tokens fall back to `default_role` and are rejected unless `default_role` is set to admin (dev-only). Alternatively, an operator key (`Authorization: Operator ` or `X-Operator-Key`) authorizes `/v1/admin/*`. Since `Config.Auth` uses `Bearer `, provide a `Config.HTTPClient` with a `Transport` that sets the `X-Operator-Key` header to use an operator key. Use `map[string]any` for dynamic schemas. ```go rows, err := wavehouse.SQL[map[string]any](ctx, wh, @@ -426,10 +328,5 @@ typed, err := wavehouse.SQL[PageTotal](ctx, wh, ``` :::note[No parameter binding through the SDK] -Positional `?` substitution is not supported, and the SDK has no way to -forward ClickHouse-style named params (the `WHERE id = {id:UInt32}` + -`param_id=42` query-string combo) — the proxy doesn't forward arbitrary -query-string params and `SQL[Row]` doesn't expose a hook to add them. Inline -literals into the SQL, or — for safe binding from user-supplied input — use -the structured query builder (`wh.From(table)...`). +Positional `?` substitution is unsupported. The SDK cannot forward ClickHouse named params (`WHERE id = {id:UInt32}` + `param_id=42`) because the proxy blocks arbitrary query-string params and `SQL[Row]` lacks a hook to add them. Use inline literals or the structured query builder (`wh.From(table)...`) for safe binding of user input. ::: diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 6a8590d5..8a076a41 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -11,9 +11,7 @@ ships with the module. Compare with the TypeScript SDK's ## Context Cancellation -Every non-streaming operation takes a `context.Context` as its first -argument — Go's equivalent of the TypeScript SDK's `AbortSignal` support. -Cancel it with a timeout or an explicit `cancel()`: +Non-streaming operations take a `context.Context` as their first argument (similar to TypeScript's `AbortSignal`). Cancel it using a timeout or explicit `cancel()`: ```go ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) @@ -26,35 +24,20 @@ if errors.As(err, &whErr) && whErr.Code == "ABORTED" { } ``` -Context cancellation returns immediately (no retry) with -`&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. +Cancellation returns immediately (no retry) with `&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. -Streams work differently: `.Stream(opts)` doesn't take a `context.Context` -at all — the returned `*StreamController` owns its own internal context and -background goroutine, torn down explicitly via `.Close()`. See -[Streaming](/sdk/go/streaming#streamoptions). - ---- +`.Stream(opts)` ignores `context.Context`; the returned `*StreamController` manages its own context and goroutine, closed via `.Close()`. See [Streaming](/sdk/go/streaming#streamoptions). ## Error Handling -The SDK never panics on API or network failures — every request-response -operation (queries, ingest, pipes, admin) returns `(T, error)` — or a bare -`error` for operations with no result body (`Pipes.Set`/`Delete`, -`Policy.Set`, `Schema.Refresh`, `Sys.Health`). Errors -originating from the HTTP exchange are `*wavehouse.Error` (unwrap with -`errors.As`); client-side failures before a request goes out (an `Auth` -provider error, a request-body marshal failure) are plain wrapped errors, -so handle the `errors.As == false` case too. Streaming lifecycle methods -(`Stream`, `Subscribe`, `Close`) don't return `(T, error)` — stream errors -are delivered via the subscriber's `Error` callback — and `Connected(ctx)` -returns plain errors. This is the direct Go equivalent of the TypeScript -SDK's "the SDK never throws" guarantee. +The SDK never panics on API or network failures. Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, while result-less operations (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`) return a bare `error`. + +HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`). Client-side failures (e.g., `Auth` provider, marshal failures) are plain wrapped errors; handle the `errors.As == false` case. Streaming methods (`Stream`, `Subscribe`, `Close`) do not return `(T, error)`; stream errors use the subscriber's `Error` callback. `Connected(ctx)` returns plain errors. This mirrors the TypeScript SDK's "never throws" guarantee. | Status | Code | Retryable | Description | |--------|------|-----------|--------------| | 400 | `HTTP_400` | No | Bad request (validation, missing fields) | -| 401 | `HTTP_401` | No | Present-but-invalid or expired JWT (a *missing* token is evaluated as `default_role`, so it succeeds or is denied with 403 — never 401) | +| 401 | `HTTP_401` | No | Invalid or expired JWT (missing tokens use `default_role`, resulting in success or 403) | | 403 | `HTTP_403` | No | Insufficient permissions | | 404 | `HTTP_404` | No | Table or pipe not found | | 429 | `HTTP_429` | Yes | Rate limited (auto-retries, honoring `Retry-After`, capped at 30s) | @@ -62,7 +45,7 @@ SDK's "the SDK never throws" guarantee. | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`, capped at 30s) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `context.Context` | -| 0 | `SSE_ERROR` | Yes | Stream connection failure, delivered to the subscriber's `Error` callback; the stream reconnects automatically | +| 0 | `SSE_ERROR` | Yes | Stream connection failure; delivered to subscriber's `Error` callback; auto-reconnects | ```go page, err := wh.From("clicks").Fetch(ctx) @@ -77,18 +60,9 @@ if err != nil { } ``` -`wavehouse.IsRetryable(err)` is a shortcut for the `errors.As` + `.Retryable` -check above. +`wavehouse.IsRetryable(err)` shortcuts the `errors.As` + `.Retryable` check. -Retries apply uniformly to every HTTP method the SDK issues (not just GET) — -matching the TypeScript SDK's `http.ts` behavior. For `/v1/ingest`, -at-least-once delivery on retry is a documented contract (see the API -docs' ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data) -note); dedup is the prescribed server-side safety net when duplicate -suppression matters. `/v1/admin/query` (raw SQL) is gated by `admin_role`, -so repeated execution on retry is an accepted risk for admin-only usage. - ---- +Retries apply to all HTTP methods, matching TypeScript's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see API docs ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup for duplicate suppression. `/v1/admin/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. ## Full API Tree @@ -143,8 +117,7 @@ NewClient(Config) → *Client ## Codegen CLI -Generate Go structs from a running WaveHouse instance. The module ships a -`wavehouse-codegen` command under `cmd/`: +Generate Go structs from a running WaveHouse instance using the `wavehouse-codegen` command in `cmd/`: ```bash export WAVEHOUSE_AUTH='' # avoids leaking the token via argv @@ -154,16 +127,13 @@ go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ --package myapp ``` -Or, working inside this repo (`clients/go/`): +Or, inside `clients/go/`: ```bash go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go ``` -Codegen reads `/v1/schema`, which is **admin-only**. Against a non-dev -server, provide an admin-role token or the request is denied with `403`. -Prefer the `WAVEHOUSE_AUTH` environment variable — a token passed with -`--auth ` ends up in shell history and process listings. +Codegen reads the admin-only `/v1/schema` endpoint; non-dev servers require an admin token or return `403`. Use `WAVEHOUSE_AUTH` instead of `--auth ` to keep tokens out of shell history and process listings. **Options:** @@ -171,13 +141,11 @@ Prefer the `WAVEHOUSE_AUTH` environment variable — a token passed with |------|-------------|---------| | `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | | `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | -| `--auth`, `-a` | Bearer token (if auth required); prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | +| `--auth`, `-a` | Bearer token; prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | | `--package`, `-p` | Go package name for the generated file | `main` | | `--help`, `-h` | Show usage and exit | — | -The output is run through `go/format` before being written — if a table or -column name would produce invalid Go source (rare, but possible with exotic -names), codegen fails loudly instead of writing broken code. +Output is processed via `go/format`. If a table or column name produces invalid Go source, codegen fails loudly. **Example output:** @@ -195,22 +163,9 @@ type ClicksRow struct { } ``` -(That's the exact output for the `clicks` table from the -[development quick-start](/development#quick-start) — `received_timestamp` -becomes `*string` + `,omitempty` because it has a `DEFAULT` clause.) - -Note the generator does **not** special-case initialisms: `event_id` becomes -`EventId`, not the Go-idiomatic `EventID` — each `_`-separated part simply -gets its first letter upper-cased. +(Example for the [development quick-start](/development#quick-start) `clicks` table; `received_timestamp` is `*string` + `,omitempty` due to its `DEFAULT` clause.) -Table and column names are converted to `PascalCase` for Go field/type names -(a leading digit gets an `X` prefix — e.g. a table named `2fa_events` -becomes `X2faEventsRow` — to stay a valid Go identifier). A column with -`has_default: true` in the schema becomes a **pointer field** with -`,omitempty` — the Go spelling of the TS codegen's `field?: T`: leave it -`nil` to omit the field (the server default applies), or point it at a -value to send it — including an explicit `0`/`false`/`""`, which a plain -value field with `omitempty` would silently drop. +The generator does not special-case initialisms: `event_id` becomes `EventId`, not the Go-idiomatic `EventID` — each `_`-separated part just gets its first letter upper-cased. Table and column names are converted to `PascalCase`; leading digits get an `X` prefix (e.g., `2fa_events` $\rightarrow$ `X2faEventsRow`). Columns with `has_default: true` become pointer fields with `,omitempty`: `nil` uses the server default, a pointed-at value is sent — including an explicit `0`/`false`/`""`. **ClickHouse → Go type mapping:** @@ -222,55 +177,30 @@ value field with `omitempty` would silently drop. | `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | | `Float32`, `BFloat16` | `float32` | | `Float64` | `float64` | -| `UInt128`/`UInt256`, `Int128`/`Int256` | `json.Number` (arbitrary-width unquoted numbers on the structured-query path) | -| `Decimal*` | `string` (marshaled as a quoted string on the structured-query path) | +| `UInt128`/`UInt256`, `Int128`/`Int256` | `json.Number` | +| `Decimal*` | `string` | | `Nullable(T)` | `*T` | | `LowCardinality(T)` | same as `T` | -| `Array(T)` | `[]T` — except `Array(UInt8)` → `json.RawMessage`: the wire is asymmetric (ingest takes a JSON array, but query responses currently base64-encode the column), and `RawMessage` is the one shape that decodes both; server-side normalization tracked in [#436](https://github.com/Wave-RF/WaveHouse/issues/436) | -| `Map(K, V)` | `map[K]V` (falls back to `map[string]any` if `K`/`V` can't be split) | +| `Array(T)` | `[]T` (except `Array(UInt8)` $\rightarrow$ `json.RawMessage` per [#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | +| `Map(K, V)` | `map[K]V` (fallback: `map[string]any`) | | `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | -This differs from the TypeScript SDK's mapping in one notable way: Go's -codegen preserves ClickHouse's integer **widths** (`UInt64` → `uint64`, not -a generic `number`), since Go — unlike TypeScript — has native fixed-width -integer types; 64-bit columns decode exactly where TS hits the JS-number -2^53 ceiling. Generated structs target the structured-query and pipe paths -(`/v1/query`, `/v1/pipes/*`), where the server re-marshals values as plain -JSON numbers. The raw-SQL path (`/v1/admin/query`) instead forwards -ClickHouse's own JSON, which **quotes** 64-bit-and-wider integers — use -`map[string]any` with `SQL[Row]` there rather than generated structs. +Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`. For the raw-SQL path (`/v1/admin/query`), which quotes 64-bit+ integers, use `map[string]any` with `SQL[Row]`. ## Testing -The Go SDK ships with unit tests colocated in `clients/go/` (its own Go -module — `clients/go/go.mod` — separate from the root `WaveHouse` module), -plus the Go half of the cross-language wire-format **conformance suite**: -`clients/go/conformance_test.go` replays the shared fixture -(`clients/go/testdata/wire_cases.json`) and asserts the Go SDK produces the -expected HTTP method, path, content type, and body for each case. The -TypeScript half — `tests/conformance/conformance_ts.mjs`, run with -`make test-conformance-ts` (it builds the TS SDK first) — replays the same -fixture, and CI runs both, keeping the two clients honest about the wire -format they both speak. +Unit tests are colocated in `clients/go/` (module `clients/go/go.mod`), separate from the root `WaveHouse` module. The cross-language wire-format **conformance suite** uses `clients/go/conformance_test.go` to replay the shared fixture (`clients/go/testdata/wire_cases.json`), asserting correct HTTP methods, paths, content types, and bodies. The TypeScript half—`tests/conformance/conformance_ts.mjs`, run via `make test-conformance-ts` (builds TS SDK first)—uses the same fixture; CI runs both to ensure wire format consistency. ```bash cd clients/go go test ./... ``` -E2E tests (build tag `e2e`) run against a live WaveHouse instance and have -their own Make target, separate from the repo's `make test-e2e`: +E2E tests (build tag `e2e`) run against a live WaveHouse instance via a dedicated Make target: ```bash WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH='' make test-go-sdk-e2e ``` -`WAVEHOUSE_URL` defaults to `http://localhost:8080`; `WAVEHOUSE_AUTH` is -optional (admin-only cases skip without it). When the server is unreachable -the suite skips instead of failing. - -Unlike the TypeScript SDK, the Go SDK isn't (yet) wired into the repo's -`make test-e2e` harness — see the TypeScript SDK's -[E2E Testing](/sdk/reference#e2e-testing) section for that suite's -architecture, which the Go client doesn't currently participate in. +`WAVEHOUSE_URL` defaults to `http://localhost:8080`; optional `WAVEHOUSE_AUTH` is for admin cases. The suite skips if the server is unreachable. Unlike the TypeScript SDK, Go isn't yet in the repo's `make test-e2e` harness (see [E2E Testing](/sdk/reference#e2e-testing)). diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index c105f28c..c16033b6 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -14,15 +14,11 @@ a `context.Context` or a browser's `EventSource`. ## Streaming -Streams use SSE (Server-Sent Events), parsed by hand over `net/http` (no -third-party SSE library — the SDK has zero runtime dependencies). +Streams use SSE (Server-Sent Events) parsed via `net/http` with zero runtime dependencies. ### `*StreamController` -Returned by `.Stream(opts)` on `*TableRef`, `*QueryBuilder`, `*PipeRef`, and -`*DLQNamespace` (the DLQ variant is not yet functional server-side — -[#197](https://github.com/Wave-RF/WaveHouse/issues/197)). Calling `.Stream` -returns immediately; the connection opens in a background goroutine. +Returned by `.Stream(opts)` on `*TableRef`, `*QueryBuilder`, `*PipeRef`, and `*DLQNamespace` (DLQ is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). Calling `.Stream` returns immediately; the connection opens in a background goroutine. ```go stream := wh.From("clicks").Stream(&wavehouse.StreamOptions{ @@ -33,8 +29,7 @@ defer stream.Close() ### `.Subscribe(sub) → func()` -Callback-based consumption. Returns an unsubscribe function. The -subscriber's `Status` callback fires immediately with the current status. +Callback-based consumption. Returns an unsubscribe function. The `Status` callback fires immediately with the current status. ```go unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ @@ -57,10 +52,11 @@ unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ defer unsub() ``` +Cleanup via `unsub()` removes the subscriber; the connection remains open for others and must be closed with `stream.Close()`. + ### Channel-based consumption — `.Events()` -The idiomatic Go alternative to the TypeScript SDK's async iterator: a -read-only channel, closed automatically when the stream shuts down. +A read-only channel, closed automatically when the stream shuts down. ```go stream := wh.From("clicks").Stream(nil) @@ -75,26 +71,14 @@ for event := range stream.Events() { ``` :::caution[`break` does not close the stream] -Unlike the TypeScript SDK's async iterator — where breaking out of a -`for await` loop auto-closes the underlying connection — breaking a Go -`for range stream.Events()` loop only stops consuming from the channel; the -background goroutine and its HTTP connection keep running. Always pair a -stream with `defer stream.Close()` (or an explicit `stream.Close()` on every -exit path) regardless of which consumption style you use. +Unlike the TypeScript SDK's async iterator, where breaking a `for await` loop closes the connection, breaking a Go `for range stream.Events()` loop only stops consumption — the background goroutine and HTTP connection persist. Always `defer stream.Close()`. ::: -The channel is buffered (256 events); a slow consumer that never drains it -causes the SDK to **drop** new events for that channel rather than block the -stream's read loop — the first drop logs one line via the standard `log` -package, further drops are silent (`.Subscribe` callbacks still fire per -event regardless of channel backpressure). +The channel is buffered (256 events). A slow consumer makes the SDK **drop** new events for that channel rather than block the read loop. The first drop logs via `log`; later drops are silent (`.Subscribe` callbacks fire regardless). ### `.Close()` -Explicitly close the stream and release its resources. Non-blocking — safe -to call from inside a subscriber callback (which runs on the stream's own -goroutine); it signals the goroutine to stop without waiting for it to -finish. +Explicitly closes the stream and releases resources. Non-blocking and safe to call from inside a subscriber callback. ```go stream.Close() @@ -102,8 +86,7 @@ stream.Close() ### `.Status()` -Returns the current `StreamStatus`. A method (not a field), since Go has no -JS-style reactive property access. +Returns the current `StreamStatus`. ```go status := stream.Status() @@ -111,11 +94,7 @@ status := stream.Status() ### `.Connected(ctx)` -**Go-only addition** — not present in the TypeScript SDK. Blocks until the -stream reaches `StatusLive` or `ctx` is canceled; returns an error if the -stream closes before connecting. Useful when you need to know a stream is -live before doing something else (e.g. before starting a producer in a -test). +Blocks until the stream reaches `StatusLive` or `ctx` is canceled; returns an error if the stream closes before connecting. Useful for ensuring a stream is live (e.g., in tests). ```go ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) @@ -131,8 +110,7 @@ if err := stream.Connected(ctx); err != nil { | ----- | ---- | ----------- | | `Since` | `string` | RFC3339 timestamp for gap-fill replay | -There's no `Signal`/context field here — a stream isn't canceled by passing -a `context.Context` into `.Stream()`; call `.Close()` instead (see above). +There's no `Signal`/context field: a stream isn't canceled by passing a `context.Context` into `.Stream()` — call `.Close()` instead. ### `StreamEvent` @@ -145,48 +123,26 @@ type StreamEvent struct { ``` :::note[`Events()` carries events only] -`Error` and `Status` are delivered exclusively through `.Subscribe(...)` — -the channel is typed `chan StreamEvent` and simply ends (closes) when the -stream closes, including on a terminal 401/403/404. Pair `Events()` with a -`Subscribe(&StreamSubscriber{Error: ..., Status: ...})` if you need to know -*why* a stream ended. +`Error` and `Status` are delivered exclusively via `.Subscribe(...)`. The channel closes on terminal errors (401/403/404). Pair `Events()` with a subscriber to determine why a stream ended. ::: :::note[The channel buffers from stream construction] -Events buffer into the channel (up to 256) from the moment `.Stream()` -constructs the stream, matching the TypeScript SDK — events arriving before -your first `Events()` call are **not** lost, so you don't have to call -`Events()` immediately. A consumer that never drains the channel still -drops everything past the 256th buffered event (with the one-time log line -described above). +Events buffer (up to 256) starting at `.Stream()`; events arriving before the first `Events()` call are not lost. ::: ### Transport Behavior | Transport | Reconnect | Protocol | | --------- | --------- | -------- | -| SSE | Automatic, with exponential backoff (capped at 30s) and gap-fill replay via the last-seen event ID | HTTP/2 recommended | - -Reconnect covers transport failures and retryable (5xx/429) responses. A -non-retryable response (401/403/404) is terminal: the error is delivered to -the subscriber's `Error` callback, status goes to `StatusClosed`, and the -stream does not reconnect — fix the cause (refresh the token, correct the -table) and open a new stream. An `Auth` provider error during a (re)connect -is treated as retryable (`SSE_ERROR`) and the stream keeps reconnecting — -`ClientOptions.MaxRetries` bounds request retries only, not stream -reconnects — so call `.Close()` if your token provider is failing -permanently. - -Auth is sent as an `Authorization: Bearer` header on every stream -(re)connection — see -[the note in the Getting Started guide](/sdk/go#creating-a-client). The -TypeScript SDK's "more than 5 concurrent connections" warning is a -browser-specific `EventSource` limit and doesn't apply here. +| SSE | Automatic, exponential backoff (max 30s), gap-fill replay via last event ID | HTTP/2 recommended | + +Reconnect covers transport failures and retryable (5xx/429) responses. Non-retryable ones (401/403/404) are terminal: the `Error` callback fires, status goes `StatusClosed`, no reconnect. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. + +Auth goes as an `Authorization: Bearer` header on every connection ([note in Getting Started](/sdk/go#creating-a-client)). Browser `EventSource` limits don't apply. ### Client-Side Stream Filtering -When a `*QueryBuilder` with `.Where()` filters or `.Select()` columns calls -`.Stream()`, the returned stream applies those filters client-side: +When a `*QueryBuilder` with `.Where()` or `.Select()` calls `.Stream()`, filters are applied client-side: ```go stream := wh.From("clicks"). @@ -197,22 +153,11 @@ stream := wh.From("clicks"). // Only events where page == "/home" are emitted, with only page + button fields ``` -Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, -`OpIn`, `OpLike`, `OpNotLike` — the same `FilterOp` set `.Where()` takes -everywhere (the SDK maps them to wire tokens such as `eq`/`neq` internally). -`OpLike` / `OpNotLike` match SQL LIKE semantics (`%` → any run of -characters, `_` → any single character), case-insensitively. `OpIn` accepts -any Go slice type on the right-hand side (`[]string`, `[]int`, `[]any`, -...), not just `[]any`. - ---- +Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the `FilterOp` set `.Where()` takes everywhere (mapped to wire tokens `eq`/`neq`). `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively. `OpIn` accepts any Go slice type (e.g., `[]string`, `[]int`). ## Live Queries -Live queries combine a historical backfill (`.FetchUntyped`) with a -real-time stream, providing a seamless initial-load + live-updates -experience. Only available on `*QueryBuilder` (there's no `TableRef.LiveQuery` -shortcut, matching the TypeScript SDK). +Live queries combine a historical backfill (`.FetchUntyped`) with a real-time stream for seamless initial loads and updates. They are available only on `*QueryBuilder` (no `TableRef.LiveQuery` shortcut), matching the TypeScript SDK. ```go lq := wh.From("clicks"). @@ -254,51 +199,31 @@ type StreamSubscriber struct { ``` :::note[`Initial` is always untyped] -Unlike the TypeScript SDK's `initial: (result: Result) => void`, the Go -SDK's `LiveQuery` doesn't accept a type parameter — `Initial` always -receives `[]map[string]any` plus a plain `error`, even if you'd otherwise -use `wavehouse.FetchTyped[Row]` for the same query outside a live query. -Decode into your own type inside the callback if you need one. +Unlike the TypeScript SDK's `initial: (result: Result) => void`, Go's `LiveQuery` takes no type parameter: `Initial` always receives `[]map[string]any` plus a plain `error`, even if you'd use `wavehouse.FetchTyped[Row]` for the same query outside a live query. Decode inside the callback if needed. ::: ### How it works -1. Subscribes to the stream **immediately** and buffers incoming events. -2. Runs the `.FetchUntyped(ctx)` query for historical data, calls - `sub.Initial(rows, err)` with the result. -3. Deduplicates buffered events against the **newest** `received_timestamp` - in the backfill — the maximum across all rows, not the last row's (an - `OrderBy(..., "desc")` puts the *oldest* row last). -4. Flushes remaining buffered events (re-checking for anything that arrived - mid-flush) and switches to live mode. +1. Subscribes to the stream immediately and buffers events. +2. Runs `.FetchUntyped(ctx)` for historical data, then calls `sub.Initial(rows, err)`. +3. Deduplicates buffered events against the maximum `received_timestamp` in the backfill (not necessarily the last row). +4. Flushes remaining buffered events and switches to live mode. -This "stream-first" approach ensures no events are lost between the fetch -and stream start. +This "stream-first" approach prevents event loss between fetch and stream start. :::caution[Dedup needs `received_timestamp` in the projection] -The dedup bound comes from the backfill rows' `received_timestamp` values. -`.SelectAll()` (or no projection) includes it; a `.Select(...)` projection -that omits it disables dedup, and events in the fetch/stream overlap window -are delivered twice — once in `Initial`, again via `Next`. +Dedup relies on `received_timestamp`. `.SelectAll()` (or no projection) includes it; a `.Select(...)` omitting it disables dedup, causing events in the overlap window to be delivered twice (via `Initial` and `Next`). ::: :::caution[`OpLike` matching differs between backfill and live] -Client-side `OpLike` matching is case-**insensitive**, but the backfill runs -server-side where the operator compiles to ClickHouse `LIKE`, which is -case-**sensitive**. A live query filtering on `OpLike` can therefore -disagree with itself: the backfill excludes rows the live stream includes. -Tracked in [#451](https://github.com/Wave-RF/WaveHouse/issues/451). - -`OpNotLike` never reaches the backfill at all — `/v1/query` rejects the -operator with a `400`, so a live query filtering on it fails its `Initial` -callback. See the operator table in -[Queries](/sdk/go/queries#wherecolumn-op-value). +Client-side `OpLike` is case-insensitive, but server-side backfills use ClickHouse `LIKE`, which is case-sensitive. Consequently, a live query filtering on `OpLike` may exclude rows in the backfill that it includes in the live stream ([#451](https://github.com/Wave-RF/WaveHouse/issues/451)). + +`OpNotLike` is rejected by `/v1/query` with a `400`, causing `Initial` callbacks to fail. See [Queries](/sdk/go/queries#wherecolumn-op-value). ::: ### `.Close()` -Shuts down the live query and its underlying stream. Safe to call more than -once (idempotent via `sync.Once`). +Shuts down the live query and its underlying stream. Safe to call more than once (idempotent via `sync.Once`). ```go lq.Close() From 59b27bbff7fd028fa062f67f439accb6cfc0e1cb Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:01:25 -0400 Subject: [PATCH 32/40] fix(sdk): bring the Go SDK current with main's API and SSE changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three catch-up changes, all client-side. The server is untouched. Routes: main merged every admin-gated endpoint under /v1/ops (#479) with no aliases, so thirteen call sites were 404ing against a current server — schema list/refresh, DLQ stats, raw SQL, policy get/put/validate, and pipes CRUD, plus the codegen CLI's schema fetch. Rewrote them along with the tests, the shared wire_cases.json fixture, and the Go SDK docs. The fixture is replayed by both conformance runners, so the stale paths broke the TypeScript half too; `make test-conformance-ts` is back to 45/45. ClientOptions.Headers: the TypeScript SDK gained options.headers in #456 and Go had no equivalent. Headers now apply to every request the client makes, REST and SSE alike — which is also how an operator sends the server's non-JWT X-Operator-Key. The SDK's own headers are set afterwards and win a collision; net/http canonicalizes names, so matching is case-insensitive; the map is copied at construction so later mutation can't reach into requests. SSE robustness, mirroring main's fetch-based rewrite (#470). The Go SDK already authenticated by header, so that part was never stale, but three gaps were: - A credentialed stream followed redirects. net/http drops Authorization on a cross-host hop while forwarding custom headers verbatim, so a redirect either downgraded the stream to default_role in silence or handed configured secrets to wherever it pointed. Now refused with a terminal SSE_REDIRECT. Uncredentialed streams still follow. - A 200 with any content type was treated as an event stream, so an auth gateway's login page left the stream sitting in StatusLive delivering nothing. Now a terminal SSE_BAD_CONTENT_TYPE. - Every failure collapsed into one retryable SSE_ERROR, and malformed frames came back as a bare fmt.Errorf, so errors.As and IsRetryable didn't work on them. Replaced with the taxonomy the TypeScript SDK uses — SSE_AUTH_ERROR, SSE_NETWORK_ERROR, SSE_CONNECT_ERROR, SSE_REDIRECT, SSE_BAD_CONTENT_TYPE, SSE_PARSE_ERROR, SSE_READ_ERROR — each with its own retryable flag, all delivered as *Error. Also documents what main changed underneath the Go SDK without changing its code: DateTime values arrive canonicalized to RFC 3339 UTC (#402), SSE applies policy row-filters per subscriber and fails closed (#381, #457), /v1/stream is ungated so WaveHouse never 401s a stream, and /v1/ops/dlq/stats is absent (404) when the DLQ is disabled rather than returning empty stats. Tests: terminal-failure table (bad content type, missing content type, credentialed redirect, non-HTTP scheme), redirect-followed-when- uncredentialed, typed retryable parse errors, and header precedence and copying on both transports. --- clients/go/client_test.go | 4 +- clients/go/cmd/wavehouse-codegen/main.go | 12 +- clients/go/conformance_test.go | 12 +- clients/go/dlq.go | 7 +- clients/go/http.go | 12 ++ clients/go/http_test.go | 101 +++++++++++- clients/go/namespaces_test.go | 6 +- clients/go/pipes.go | 8 +- clients/go/policy.go | 6 +- clients/go/schema.go | 4 +- clients/go/stream.go | 142 ++++++++++++++--- clients/go/stream_test.go | 186 ++++++++++++++++++++++ clients/go/table.go | 2 +- clients/go/testdata/wire_cases.json | 24 +-- clients/go/wavehouse.go | 22 ++- docs/src/content/docs/sdk/go/admin.md | 4 +- docs/src/content/docs/sdk/go/index.md | 24 ++- docs/src/content/docs/sdk/go/pipes.md | 2 +- docs/src/content/docs/sdk/go/queries.md | 2 +- docs/src/content/docs/sdk/go/reference.md | 15 +- docs/src/content/docs/sdk/go/streaming.md | 16 +- tests/conformance/conformance_ts.mjs | 12 +- 22 files changed, 545 insertions(+), 78 deletions(-) diff --git a/clients/go/client_test.go b/clients/go/client_test.go index aca92d0d..2ba55833 100644 --- a/clients/go/client_test.go +++ b/clients/go/client_test.go @@ -76,8 +76,8 @@ func TestClient_From(t *testing.T) { func TestClient_SQL(t *testing.T) { srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/v1/admin/query" { - t.Errorf("want /v1/admin/query, got %s", r.URL.Path) + if r.URL.Path != "/v1/ops/query" { + t.Errorf("want /v1/ops/query, got %s", r.URL.Path) } var body map[string]string _ = json.NewDecoder(r.Body).Decode(&body) diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go index 9876ee07..9fa68258 100644 --- a/clients/go/cmd/wavehouse-codegen/main.go +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -1,4 +1,4 @@ -// Command wavehouse-codegen reads a WaveHouse server's /v1/schema endpoint +// Command wavehouse-codegen reads a WaveHouse server's /v1/ops/schema endpoint // and generates Go struct definitions for use with the wavehouse SDK. // // Usage: @@ -56,7 +56,7 @@ func parseArgs() cliArgs { Options: --url, -u WaveHouse base URL (default: http://localhost:8080) --out, -o Output .go file path (default: ./wavehouse_types.go) - --auth, -a Bearer token for authenticated /v1/schema endpoint + --auth, -a Bearer token for authenticated /v1/ops/schema endpoint (prefer the WAVEHOUSE_AUTH env var — argv leaks into shell history and process listings) --package, -p Go package name (default: main) @@ -85,7 +85,7 @@ type tableSchema struct { } func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSchema, error) { - url := strings.TrimRight(baseURL, "/") + "/v1/schema" + url := strings.TrimRight(baseURL, "/") + "/v1/ops/schema" req, err := http.NewRequestWithContext(ctx, "GET", url, nil) if err != nil { return nil, fmt.Errorf("build schema request for %s: %w", url, err) @@ -124,7 +124,7 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc return m, nil } -// chTypeToGo maps a ClickHouse type string (as reported by /v1/schema) to a +// chTypeToGo maps a ClickHouse type string (as reported by /v1/ops/schema) to a // Go type name suitable for a JSON struct field. // // We deliberately don't import clickhouse-go's type catalog @@ -135,7 +135,7 @@ func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSc // types the *driver* scans query results into over the native protocol // (time.Time for Date/DateTime*, uuid.UUID for UUID, decimal.Decimal for // Decimal, net.IP for IPv4/IPv6, *big.Int for [U]Int128/256), not the types -// that round-trip cleanly through the JSON the /v1/schema and query +// that round-trip cleanly through the JSON the /v1/ops/schema and query // endpoints actually speak. ClickHouse's JSON output renders DateTime as // "2024-01-15 10:30:00" (no "T", no offset), which fails Go's default // time.Time JSON unmarshaling; big integers and decimals are similarly @@ -188,7 +188,7 @@ func chTypeToGo(chType string) string { // pipe paths (/v1/query, /v1/pipes/*), where the server scans ClickHouse // values into Go types and re-marshals them — so 64-bit integers arrive // as ordinary UNQUOTED JSON numbers and map to int64/uint64 exactly. - // (Only /v1/admin/query forwards ClickHouse's own JSON, which quotes + // (Only /v1/ops/query forwards ClickHouse's own JSON, which quotes // 64-bit ints; use map[string]any with SQL[Row] there.) if mapped, ok := map[string]string{ "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go index e90f8412..c8af2899 100644 --- a/clients/go/conformance_test.go +++ b/clients/go/conformance_test.go @@ -92,17 +92,17 @@ func TestConformance_WireFormat(t *testing.T) { // Return valid JSON so the SDK doesn't error on decode. w.Header().Set("Content-Type", "application/json") switch { - case strings.HasPrefix(r.URL.Path, "/v1/dlq"): + case strings.HasPrefix(r.URL.Path, "/v1/ops/dlq"): _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{}, Total: 0}) - case strings.HasPrefix(r.URL.Path, "/v1/schema") && r.Method == "GET": + case strings.HasPrefix(r.URL.Path, "/v1/ops/schema") && r.Method == "GET": _ = json.NewEncoder(w).Encode([]TableSchema{}) - case r.URL.Path == "/v1/admin/policy/validate" && r.Method == "POST": + case r.URL.Path == "/v1/ops/policy/validate" && r.Method == "POST": _ = json.NewEncoder(w).Encode(ValidationResult{Valid: true}) - case strings.HasPrefix(r.URL.Path, "/v1/admin/policy") && r.Method == "GET": + case strings.HasPrefix(r.URL.Path, "/v1/ops/policy") && r.Method == "GET": _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) - case strings.HasPrefix(r.URL.Path, "/v1/admin/pipes/") && r.Method == "GET": + case strings.HasPrefix(r.URL.Path, "/v1/ops/pipes/") && r.Method == "GET": _ = json.NewEncoder(w).Encode(Pipe{Name: "test", SQL: "SELECT 1"}) - case r.URL.Path == "/v1/admin/pipes" && r.Method == "GET": + case r.URL.Path == "/v1/ops/pipes" && r.Method == "GET": _ = json.NewEncoder(w).Encode([]Pipe{}) default: _ = json.NewEncoder(w).Encode([]map[string]any{}) diff --git a/clients/go/dlq.go b/clients/go/dlq.go index 0009e4e3..65712765 100644 --- a/clients/go/dlq.go +++ b/clients/go/dlq.go @@ -7,6 +7,11 @@ import ( ) // DLQNamespace provides admin-only dead-letter-queue statistics. +// +// The server registers /v1/ops/dlq/stats only when the DLQ is enabled, so on a +// deployment with dlq.enabled: false these calls return an [*Error] with +// Status 404 — "the DLQ is switched off", not "the DLQ is empty". Check +// Status before reading a zero DLQStats as a healthy result. type DLQNamespace struct { ctx httpContext createStream func(table string, opts *StreamOptions) *StreamController @@ -26,7 +31,7 @@ func (d *DLQNamespace) stats(ctx context.Context, params url.Values) (*DLQStats, var stats DLQStats if err := doRequest(ctx, d.ctx, requestOptions{ method: "GET", - path: "/v1/dlq/stats", + path: "/v1/ops/dlq/stats", params: params, }, &stats); err != nil { return nil, fmt.Errorf("get dlq stats: %w", err) diff --git a/clients/go/http.go b/clients/go/http.go index 9d149e90..eaab0e94 100644 --- a/clients/go/http.go +++ b/clients/go/http.go @@ -26,6 +26,17 @@ type httpContext struct { auth func(ctx context.Context) (string, error) maxRetries int httpClient *http.Client + headers map[string]string +} + +// applyConfiguredHeaders writes the client's configured headers onto a request. +// Call it *before* the SDK sets its own headers: Set replaces, so whatever the +// SDK writes afterwards wins a collision. http.Header canonicalizes names, so +// "x-tenant" and "X-Tenant" are the same entry. +func applyConfiguredHeaders(h http.Header, configured map[string]string) { + for k, v := range configured { + h.Set(k, v) + } } // requestOptions describes a single HTTP request. @@ -86,6 +97,7 @@ func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst a if err != nil { return fmt.Errorf("wavehouse: build request: %w", err) } + applyConfiguredHeaders(req.Header, hctx.headers) req.Header.Set("Content-Type", ct) req.Header.Set("Accept", "application/json") if authHeader != "" { diff --git a/clients/go/http_test.go b/clients/go/http_test.go index 57109f3b..52212c16 100644 --- a/clients/go/http_test.go +++ b/clients/go/http_test.go @@ -113,7 +113,7 @@ func TestDoRequest_AuthInjection(t *testing.T) { err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", - path: "/v1/schema", + path: "/v1/ops/schema", }, nil) if err != nil { t.Fatal(err) @@ -135,7 +135,7 @@ func TestDoRequest_4xxNotRetried(t *testing.T) { err := doRequest(context.Background(), hctx, requestOptions{ method: "GET", - path: "/v1/schema", + path: "/v1/ops/schema", }, nil) if !errIs(err, "HTTP_404") { @@ -199,7 +199,7 @@ func TestDoRequest_EmptyResponse(t *testing.T) { var result map[string]string err := doRequest(context.Background(), hctx, requestOptions{ method: "POST", - path: "/v1/schema/refresh", + path: "/v1/ops/schema/refresh", }, &result) if err != nil { t.Fatal(err) @@ -335,3 +335,98 @@ func TestBaseURLPathPrefixIsPreserved(t *testing.T) { } } } + +// TestConfiguredHeadersOnRESTRequests: ClientOptions.Headers apply to every +// REST call, are matched case-insensitively, and always lose to the SDK's own +// headers rather than appending alongside them. +func TestConfiguredHeadersOnRESTRequests(t *testing.T) { + tests := []struct { + name string + configured map[string]string + auth func(context.Context) (string, error) + header string + want string + }{ + { + name: "custom header is forwarded", + configured: map[string]string{"X-Operator-Key": "op-secret"}, + header: "X-Operator-Key", + want: "op-secret", + }, + { + name: "name matching is case-insensitive", + configured: map[string]string{"x-tenant-id": "acme"}, + header: "X-Tenant-Id", + want: "acme", + }, + { + name: "SDK Accept outranks a configured one", + configured: map[string]string{"Accept": "text/plain"}, + header: "Accept", + want: "application/json", + }, + { + name: "SDK Authorization outranks a configured one", + configured: map[string]string{"Authorization": "Bearer configured"}, + auth: StaticToken("real-token"), + header: "Authorization", + want: "Bearer real-token", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + var got http.Header + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Clone() + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `[]`) + })) + defer srv.Close() + + client := NewClient(Config{ + BaseURL: srv.URL, + Auth: tc.auth, + HTTPClient: srv.Client(), + Options: &ClientOptions{Headers: tc.configured}, + }) + if _, err := client.Schema.List(context.Background()); err != nil { + t.Fatalf("schema list: %v", err) + } + if v := got.Values(tc.header); len(v) != 1 { + t.Fatalf("want exactly one %s header, got %v", tc.header, v) + } + if v := got.Get(tc.header); v != tc.want { + t.Fatalf("want %s: %q, got %q", tc.header, tc.want, v) + } + }) + } +} + +// TestConfiguredHeadersAreCopied: mutating the caller's map after NewClient +// must not change what later requests send. +func TestConfiguredHeadersAreCopied(t *testing.T) { + var got http.Header + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Clone() + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `[]`) + })) + defer srv.Close() + + headers := map[string]string{"X-Tenant-Id": "acme"} + client := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{Headers: headers}, + }) + headers["X-Tenant-Id"] = "attacker" + delete(headers, "X-Tenant-Id") + + if _, err := client.Schema.List(context.Background()); err != nil { + t.Fatalf("schema list: %v", err) + } + if v := got.Get("X-Tenant-Id"); v != "acme" { + t.Fatalf("want the value captured at construction, got %q", v) + } +} diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go index e7340b6f..27453e6b 100644 --- a/clients/go/namespaces_test.go +++ b/clients/go/namespaces_test.go @@ -23,8 +23,8 @@ func TestSysNamespace_Health(t *testing.T) { func TestSchemaNamespace_List(t *testing.T) { c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { - if r.URL.Path != "/v1/schema" { - t.Errorf("want /v1/schema, got %s", r.URL.Path) + if r.URL.Path != "/v1/ops/schema" { + t.Errorf("want /v1/ops/schema, got %s", r.URL.Path) } _ = json.NewEncoder(w).Encode([]TableSchema{ {Name: "clicks", Columns: []Column{{Name: "page", Type: "String"}}}, @@ -132,7 +132,7 @@ func TestPipesNamespace_CRUD(t *testing.T) { c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": - if r.URL.Path == "/v1/admin/pipes" { + if r.URL.Path == "/v1/ops/pipes" { _ = json.NewEncoder(w).Encode([]Pipe{{Name: "p1", SQL: "SELECT 1"}}) } else { _ = json.NewEncoder(w).Encode(Pipe{Name: "p1", SQL: "SELECT 1"}) diff --git a/clients/go/pipes.go b/clients/go/pipes.go index 71b25720..5501fa86 100644 --- a/clients/go/pipes.go +++ b/clients/go/pipes.go @@ -16,7 +16,7 @@ func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { var pipes []Pipe if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", - path: "/v1/admin/pipes", + path: "/v1/ops/pipes", }, &pipes); err != nil { return nil, fmt.Errorf("list pipes: %w", err) } @@ -28,7 +28,7 @@ func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { var pipe Pipe if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", - path: "/v1/admin/pipes/" + url.PathEscape(name), + path: "/v1/ops/pipes/" + url.PathEscape(name), }, &pipe); err != nil { return nil, fmt.Errorf("get pipe %q: %w", name, err) } @@ -39,7 +39,7 @@ func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) error { if err := doRequest(ctx, p.ctx, requestOptions{ method: "PUT", - path: "/v1/admin/pipes/" + url.PathEscape(name), + path: "/v1/ops/pipes/" + url.PathEscape(name), body: def, }, nil); err != nil { return fmt.Errorf("set pipe %q: %w", name, err) @@ -51,7 +51,7 @@ func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) erro func (p *PipesNamespace) Delete(ctx context.Context, name string) error { if err := doRequest(ctx, p.ctx, requestOptions{ method: "DELETE", - path: "/v1/admin/pipes/" + url.PathEscape(name), + path: "/v1/ops/pipes/" + url.PathEscape(name), }, nil); err != nil { return fmt.Errorf("delete pipe %q: %w", name, err) } diff --git a/clients/go/policy.go b/clients/go/policy.go index 9fbc5bb6..7e356bee 100644 --- a/clients/go/policy.go +++ b/clients/go/policy.go @@ -15,7 +15,7 @@ func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { var pol Policy if err := doRequest(ctx, p.ctx, requestOptions{ method: "GET", - path: "/v1/admin/policy", + path: "/v1/ops/policy", }, &pol); err != nil { return nil, fmt.Errorf("get policy: %w", err) } @@ -26,7 +26,7 @@ func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { if err := doRequest(ctx, p.ctx, requestOptions{ method: "PUT", - path: "/v1/admin/policy", + path: "/v1/ops/policy", body: pol, }, nil); err != nil { return fmt.Errorf("set policy: %w", err) @@ -39,7 +39,7 @@ func (p *PolicyNamespace) Validate(ctx context.Context, pol *Policy) (*Validatio var result ValidationResult if err := doRequest(ctx, p.ctx, requestOptions{ method: "POST", - path: "/v1/admin/policy/validate", + path: "/v1/ops/policy/validate", body: pol, }, &result); err != nil { return nil, fmt.Errorf("validate policy: %w", err) diff --git a/clients/go/schema.go b/clients/go/schema.go index 3f5b4a88..c79dcb11 100644 --- a/clients/go/schema.go +++ b/clients/go/schema.go @@ -16,7 +16,7 @@ func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { var raw []TableSchema if err := doRequest(ctx, s.ctx, requestOptions{ method: "GET", - path: "/v1/schema", + path: "/v1/ops/schema", }, &raw); err != nil { return nil, fmt.Errorf("list schemas: %w", err) } @@ -31,7 +31,7 @@ func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { func (s *SchemaNamespace) Refresh(ctx context.Context) error { if err := doRequest(ctx, s.ctx, requestOptions{ method: "POST", - path: "/v1/schema/refresh", + path: "/v1/ops/schema/refresh", }, nil); err != nil { return fmt.Errorf("refresh schema: %w", err) } diff --git a/clients/go/stream.go b/clients/go/stream.go index 8307e2fb..48ce32e8 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -7,6 +7,7 @@ import ( "errors" "fmt" "log" + "mime" "net/http" "net/url" "reflect" @@ -246,21 +247,28 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str } if err != nil { - // A non-retryable API error (401/403/404, ...) is terminal: - // reconnecting can't fix a bad token or a missing table, and the - // TS SDK's EventSource likewise ends up closed on a non-200. - // Emit it and exit — the deferred cleanup sets StatusClosed. + // connect classifies its own failures (SSE_AUTH_ERROR, + // SSE_NETWORK_ERROR, SSE_REDIRECT, SSE_BAD_CONTENT_TYPE, + // SSE_READ_ERROR, HTTP_nnn), so pass the typed error straight + // through and let Retryable decide whether to reconnect. A + // non-retryable error is terminal: reconnecting can't fix a bad + // token, a missing table, or a proxy answering with HTML. var apiErr *Error - if errors.As(err, &apiErr) && !apiErr.Retryable { + if errors.As(err, &apiErr) { sc.emitError(apiErr) - return + if !apiErr.Retryable { + return + } + } else { + // Unclassified — retry, but keep the generic code so callers + // can still match on it. + sc.emitError(&Error{ + Status: 0, + Code: "SSE_ERROR", + Message: err.Error(), + Retryable: true, + }) } - sc.emitError(&Error{ - Status: 0, - Code: "SSE_ERROR", - Message: err.Error(), - Retryable: true, - }) } sc.setStatus(StatusReconnecting) @@ -281,7 +289,22 @@ func (sc *StreamController) run(ctx context.Context, hctx httpContext, table str func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table, since string) (string, bool, error) { u, err := url.Parse(hctx.baseURL + "/v1/stream") if err != nil { - return "", false, err + return "", false, &Error{ + Status: 0, + Code: "SSE_CONNECT_ERROR", + Message: fmt.Sprintf("invalid baseURL: %v", err), + Retryable: false, + } + } + // A non-HTTP scheme can never carry SSE. Terminal, not retryable: retrying + // a ws:// or file:// baseURL just spins. + if u.Scheme != "http" && u.Scheme != "https" { + return "", false, &Error{ + Status: 0, + Code: "SSE_CONNECT_ERROR", + Message: fmt.Sprintf("baseURL scheme %q is not http or https", u.Scheme), + Retryable: false, + } } q := u.Query() q.Set("table", table) @@ -294,7 +317,14 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if hctx.auth != nil { token, err := hctx.auth(ctx) if err != nil { - return "", false, fmt.Errorf("auth: %w", err) + // Retryable: a token endpoint having a bad minute shouldn't tear + // down a healthy long-lived stream. + return "", false, &Error{ + Status: 0, + Code: "SSE_AUTH_ERROR", + Message: err.Error(), + Retryable: true, + } } if token != "" { authHeader = "Bearer " + token @@ -307,22 +337,75 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table if err != nil { return "", false, err } + applyConfiguredHeaders(req.Header, hctx.headers) req.Header.Set("Accept", "text/event-stream") req.Header.Set("Cache-Control", "no-cache") if authHeader != "" { req.Header.Set("Authorization", authHeader) } - resp, err := hctx.httpClient.Do(req) + client := hctx.httpClient + credentialed := authHeader != "" || len(hctx.headers) > 0 + if credentialed { + // Refuse to follow a redirect while carrying a credential. net/http + // drops Authorization on a cross-host hop but forwards custom headers + // verbatim, so following one would either downgrade the stream to + // default_role without saying so, or hand configured secrets to + // wherever the redirect points. Copy the client so a caller-supplied + // one keeps its own CheckRedirect for every other request. + c := *client + c.CheckRedirect = func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + } + client = &c + } + + resp, err := client.Do(req) if err != nil { - return "", false, err + if ctx.Err() != nil { + return "", false, errAborted + } + return "", false, &Error{ + Status: 0, + Code: "SSE_NETWORK_ERROR", + Message: err.Error(), + Retryable: true, + } } defer func() { _ = resp.Body.Close() }() + if credentialed && resp.StatusCode >= 300 && resp.StatusCode < 400 { + return "", false, &Error{ + Status: resp.StatusCode, + Code: "SSE_REDIRECT", + Message: fmt.Sprintf( + "stream endpoint redirected to %q and the SDK did not follow it; redirects are refused while the request carries a credential", + resp.Header.Get("Location")), + Retryable: false, + } + } + if resp.StatusCode != http.StatusOK { return "", false, parseErrorResponse(resp) } + // A 200 that isn't an event stream means something between the caller and + // WaveHouse answered — a captive portal or an auth gateway's login page. + // Without this check the stream sits in StatusLive and silently delivers + // nothing. + if ct := resp.Header.Get("Content-Type"); !isEventStream(ct) { + shown := ct + if shown == "" { + shown = "(none)" + } + return "", false, &Error{ + Status: resp.StatusCode, + Code: "SSE_BAD_CONTENT_TYPE", + Message: fmt.Sprintf("expected Content-Type text/event-stream, got %s", shown), + Retryable: false, + } + } + sc.setStatus(StatusLive) // Parse SSE frames. @@ -372,7 +455,25 @@ func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table } } - return lastID, true, scanner.Err() + if scanErr := scanner.Err(); scanErr != nil { + return lastID, true, &Error{ + Status: 0, + Code: "SSE_READ_ERROR", + Message: scanErr.Error(), + Retryable: true, + } + } + return lastID, true, nil +} + +// isEventStream reports whether a Content-Type header names text/event-stream, +// ignoring any parameters (charset, boundary) and case. +func isEventStream(contentType string) bool { + mediaType, _, err := mime.ParseMediaType(contentType) + if err != nil { + return false + } + return mediaType == "text/event-stream" } // sseMessage matches the server's SSE event JSON shape. @@ -389,7 +490,12 @@ func (sc *StreamController) handleSSEData(data string) { // process-global logger, so consumers control visibility and a // malformed-frame flood can't spam host-application logs. Payload // deliberately omitted: event data can carry tenant/PII fields. - sc.emitError(fmt.Errorf("malformed SSE message (%d bytes): %w", len(data), err)) + sc.emitError(&Error{ + Status: 0, + Code: "SSE_PARSE_ERROR", + Message: fmt.Sprintf("malformed SSE message (%d bytes): %v", len(data), err), + Retryable: true, + }) return } diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index efe10fd2..c65bb863 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -466,3 +466,189 @@ func TestStreamBaseURLPathPrefixIsPreserved(t *testing.T) { t.Fatal("stream never reached the prefixed path") } } + +// TestStream_TerminalConnectFailures: every way a connection can fail in a way +// reconnecting cannot fix. Each case must surface a specific, non-retryable +// code and close the stream — the generic retryable SSE_ERROR would spin here. +func TestStream_TerminalConnectFailures(t *testing.T) { + tests := []struct { + name string + handler http.HandlerFunc + baseURL string // overrides the test server URL when non-empty + auth func(context.Context) (string, error) + wantCode string + }{ + { + name: "200 that is not an event stream", + handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.WriteHeader(http.StatusOK) + _, _ = io.WriteString(w, "Please sign in") + }, + wantCode: "SSE_BAD_CONTENT_TYPE", + }, + { + name: "200 with no Content-Type at all", + handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header()["Content-Type"] = nil + w.WriteHeader(http.StatusOK) + }, + wantCode: "SSE_BAD_CONTENT_TYPE", + }, + { + name: "credentialed request is redirected", + handler: func(w http.ResponseWriter, _ *http.Request) { + http.Redirect(w, &http.Request{}, "https://elsewhere.example/v1/stream", http.StatusFound) + }, + auth: StaticToken("secret-token"), + wantCode: "SSE_REDIRECT", + }, + { + name: "baseURL scheme cannot carry SSE", + handler: func(http.ResponseWriter, *http.Request) {}, + baseURL: "ws://example.invalid", + wantCode: "SSE_CONNECT_ERROR", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + srv := httptest.NewServer(tc.handler) + t.Cleanup(srv.Close) + + base := srv.URL + if tc.baseURL != "" { + base = tc.baseURL + } + client := NewClient(Config{BaseURL: base, Auth: tc.auth, HTTPClient: srv.Client()}) + + stream := client.From("clicks").Stream(nil) + defer stream.Close() + + errCh := make(chan error, 4) + stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) { + t.Fatalf("want *Error, got %T: %v", err, err) + } + if apiErr.Code != tc.wantCode { + t.Fatalf("want code %s, got %s (%v)", tc.wantCode, apiErr.Code, err) + } + if apiErr.Retryable { + t.Fatalf("%s must not be retryable", apiErr.Code) + } + case <-time.After(5 * time.Second): + t.Fatal("error never surfaced") + } + + select { + case <-stream.done: + case <-time.After(5 * time.Second): + t.Fatalf("stream never closed after terminal %s", tc.wantCode) + } + }) + } +} + +// TestStream_RedirectFollowedWhenUncredentialed: the refusal is scoped to +// requests carrying a credential. Without one there is nothing to leak or +// silently downgrade, so the redirect is followed as usual. +func TestStream_RedirectFollowedWhenUncredentialed(t *testing.T) { + target := sseServer(t, []string{sseFrame("2026-01-01T00:00:01Z", "/home")}) + + front := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, target.URL+"/v1/stream?table=clicks", http.StatusFound) + })) + t.Cleanup(front.Close) + + stream := streamClient(t, front).From("clicks").Stream(nil) + defer stream.Close() + + events := make(chan StreamEvent, 4) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { events <- e }}) + + select { + case e := <-events: + if e.Table != "clicks" { + t.Fatalf("want table clicks, got %s", e.Table) + } + case <-time.After(5 * time.Second): + t.Fatal("redirect was not followed for an uncredentialed stream") + } +} + +// TestStream_MalformedFrameIsTypedAndRetryable: a bad frame must arrive as an +// *Error so errors.As and IsRetryable work on it — a bare fmt.Errorf would +// leave callers string-matching. +func TestStream_MalformedFrameIsTypedAndRetryable(t *testing.T) { + srv := sseServer(t, []string{"id: 1\ndata: {not json\n\n"}) + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + errCh := make(chan error, 4) + stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) { + t.Fatalf("want *Error, got %T: %v", err, err) + } + if apiErr.Code != "SSE_PARSE_ERROR" { + t.Fatalf("want SSE_PARSE_ERROR, got %s", apiErr.Code) + } + if !IsRetryable(err) { + t.Fatal("a malformed frame must stay retryable") + } + if strings.Contains(apiErr.Message, "not json") { + t.Fatal("payload must not be echoed into the error message") + } + case <-time.After(5 * time.Second): + t.Fatal("error never surfaced") + } +} + +// TestStream_ConfiguredHeadersReachTheStream: ClientOptions.Headers apply to +// SSE, not just REST — and the SDK's own headers still win a collision. +func TestStream_ConfiguredHeadersReachTheStream(t *testing.T) { + seen := make(chan http.Header, 1) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case seen <- r.Header.Clone(): + default: + } + w.Header().Set("Content-Type", "text/event-stream") + w.WriteHeader(http.StatusOK) + if fl, ok := w.(http.Flusher); ok { + fl.Flush() + } + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + + client := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{Headers: map[string]string{ + "X-Operator-Key": "op-secret", + "accept": "application/json", // must lose to the SDK's own Accept + }}, + }) + stream := client.From("clicks").Stream(nil) + defer stream.Close() + + select { + case h := <-seen: + if got := h.Get("X-Operator-Key"); got != "op-secret" { + t.Fatalf("want configured header on the stream request, got %q", got) + } + if got := h.Get("Accept"); got != "text/event-stream" { + t.Fatalf("SDK Accept must win, got %q", got) + } + case <-time.After(5 * time.Second): + t.Fatal("stream request never arrived") + } +} diff --git a/clients/go/table.go b/clients/go/table.go index e66f7f05..73c2ea3f 100644 --- a/clients/go/table.go +++ b/clients/go/table.go @@ -81,7 +81,7 @@ func (t *TableRef) Schema(ctx context.Context) (*TableSchema, error) { var schema TableSchema if err := doRequest(ctx, t.ctx, requestOptions{ method: "GET", - path: "/v1/schema", + path: "/v1/ops/schema", params: url.Values{"table": {t.table}}, }, &schema); err != nil { return nil, fmt.Errorf("get schema for table %q: %w", t.table, err) diff --git a/clients/go/testdata/wire_cases.json b/clients/go/testdata/wire_cases.json index e721afb6..d715b3bc 100644 --- a/clients/go/testdata/wire_cases.json +++ b/clients/go/testdata/wire_cases.json @@ -454,7 +454,7 @@ "name": "raw SQL", "endpoint": "sql", "sql": "SELECT count() FROM clicks", - "expected_path": "/v1/admin/query", + "expected_path": "/v1/ops/query", "expected_method": "POST", "expected_body": { "sql": "SELECT count() FROM clicks" } }, @@ -467,19 +467,19 @@ { "name": "schema list", "endpoint": "schema_list", - "expected_path": "/v1/schema", + "expected_path": "/v1/ops/schema", "expected_method": "GET" }, { "name": "schema refresh", "endpoint": "schema_refresh", - "expected_path": "/v1/schema/refresh", + "expected_path": "/v1/ops/schema/refresh", "expected_method": "POST" }, { "name": "policy get", "endpoint": "policy_get", - "expected_path": "/v1/admin/policy", + "expected_path": "/v1/ops/policy", "expected_method": "GET" }, { @@ -500,14 +500,14 @@ { "name": "DLQ list", "endpoint": "dlq_list", - "expected_path": "/v1/dlq/stats", + "expected_path": "/v1/ops/dlq/stats", "expected_method": "GET" }, { "name": "DLQ table filter", "endpoint": "dlq_table", "table": "events", - "expected_path": "/v1/dlq/stats?table=events", + "expected_path": "/v1/ops/dlq/stats?table=events", "expected_method": "GET" }, { @@ -519,7 +519,7 @@ "events": {} } }, - "expected_path": "/v1/admin/policy", + "expected_path": "/v1/ops/policy", "expected_method": "PUT", "expected_content_type": "application/json", "expected_body": { @@ -538,7 +538,7 @@ "events": {} } }, - "expected_path": "/v1/admin/policy/validate", + "expected_path": "/v1/ops/policy/validate", "expected_method": "POST", "expected_content_type": "application/json", "expected_body": { @@ -551,14 +551,14 @@ { "name": "pipes list", "endpoint": "pipes_list", - "expected_path": "/v1/admin/pipes", + "expected_path": "/v1/ops/pipes", "expected_method": "GET" }, { "name": "pipes get", "endpoint": "pipes_get", "pipe_name": "my_pipe", - "expected_path": "/v1/admin/pipes/my_pipe", + "expected_path": "/v1/ops/pipes/my_pipe", "expected_method": "GET" }, { @@ -572,7 +572,7 @@ ], "description": "Top pages by view count" }, - "expected_path": "/v1/admin/pipes/my_pipe", + "expected_path": "/v1/ops/pipes/my_pipe", "expected_method": "PUT", "expected_content_type": "application/json", "expected_body": { @@ -587,7 +587,7 @@ "name": "pipes delete", "endpoint": "pipes_delete", "pipe_name": "my_pipe", - "expected_path": "/v1/admin/pipes/my_pipe", + "expected_path": "/v1/ops/pipes/my_pipe", "expected_method": "DELETE" }, { diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go index 5acfc46b..3698aedd 100644 --- a/clients/go/wavehouse.go +++ b/clients/go/wavehouse.go @@ -41,6 +41,16 @@ type ClientOptions struct { // MaxRetries is the maximum number of retry attempts for retryable errors. // Total attempts = MaxRetries + 1. Default: 2. MaxRetries int + + // Headers are sent on every request the client makes — REST calls and SSE + // streams alike. Use them for a gateway credential, a tenant selector, or + // tracing metadata that has no first-class option. + // + // The SDK's own headers win: Authorization, Accept, Content-Type, and the + // stream's Cache-Control are set after these and overwrite any entry that + // collides. Names are matched case-insensitively (canonicalized by + // net/http), and each entry replaces rather than appends. + Headers map[string]string } // StaticToken returns an Auth function that always returns the same token. @@ -72,6 +82,15 @@ func NewClient(cfg Config) *Client { maxRetries = cfg.Options.MaxRetries } + // Copy so a later mutation of the caller's map can't reach into requests. + var headers map[string]string + if cfg.Options != nil && len(cfg.Options.Headers) > 0 { + headers = make(map[string]string, len(cfg.Options.Headers)) + for k, v := range cfg.Options.Headers { + headers[k] = v + } + } + hc := cfg.HTTPClient if hc == nil { // Not http.DefaultClient: it's mutable global state another package @@ -85,6 +104,7 @@ func NewClient(cfg Config) *Client { auth: cfg.Auth, maxRetries: maxRetries, httpClient: hc, + headers: headers, }, } @@ -124,7 +144,7 @@ func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { var rows []Row err := doRequest(ctx, c.ctx, requestOptions{ method: "POST", - path: "/v1/admin/query", + path: "/v1/ops/query", body: map[string]string{"sql": query}, }, &rows) if err != nil { diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md index 65695dbf..7e06eeed 100644 --- a/docs/src/content/docs/sdk/go/admin.md +++ b/docs/src/content/docs/sdk/go/admin.md @@ -5,9 +5,11 @@ description: "Schema introspection, access-control policy, DLQ stats, and health Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. All except `client.Sys.Health` require the admin role (`policy.admin_role`)—see [Access Control](/access-control) and the TypeScript SDK's [Admin & System](/sdk/admin) page. +Every namespace on this page is admin-gated: the server mounts them under `/v1/ops/*` behind one gate, which a caller clears either with a JWT resolving to the policy admin role (`admin_role`, `"admin"` by default) or with the server's non-JWT [operator key](/api#authentication) sent as `X-Operator-Key` via [`ClientOptions.Headers`](/sdk/go#clientoptions). + ## Schema — `client.Schema` -Introspect ClickHouse table schemas. `Schema.List`, `Schema.Refresh`, and `From(t).Schema` hit the **admin-only** `/v1/schema*`; against any non-dev policy (anything but `default_role: admin`) build the client with an admin-role token or they return a `*wavehouse.Error` with `Status: 403`. +Introspect ClickHouse table schemas. `Schema.List`, `Schema.Refresh`, and `From(t).Schema` hit the **admin-gated** `/v1/ops/schema*`; against any non-dev policy (anything but `default_role: admin`) build the client with an admin-role token or they return a `*wavehouse.Error` with `Status: 403`. ```go // List all table schemas. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index f2a4fc7f..219c1105 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -97,6 +97,7 @@ The default client has no `Timeout`; use a `context.Context` deadline to prevent | Field | Type | Default | Description | |-------|------|---------|-------------| | `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures). | +| `Headers` | `map[string]string` | `nil` | Sent on every request the client makes — REST calls and SSE streams alike. | `*Client` is safe for concurrent use; state is immutable after `NewClient` and builder chains copy. Ensure your `Auth` function is concurrency-safe. @@ -104,6 +105,22 @@ The default client has no `Timeout`; use a `context.Context` deadline to prevent The 2-retry default only applies if `Config.Options` is `nil`. If `Options` is provided, an unset `MaxRetries` field defaults to Go's int zero value (`0`), which explicitly disables retries. Passing `&wavehouse.ClientOptions{}` removes the default retry behavior. ::: +`Headers` is the Go analog of the TypeScript SDK's [`options.headers`](/sdk#custom-headers) — a gateway credential, a tenant selector, or tracing metadata that has no first-class option. It is also how an operator sends the server's non-JWT [operator key](/api#authentication): + +```go +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Options: &wavehouse.ClientOptions{ + MaxRetries: 2, // Options opts out of the default — set it explicitly. + Headers: map[string]string{"X-Operator-Key": os.Getenv("WH_OPERATOR_KEY")}, + }, +}) +``` + +The SDK's own headers win: `Authorization`, `Accept`, `Content-Type`, and the stream's `Cache-Control` are set after yours and overwrite any entry that collides. Names are matched case-insensitively (`net/http` canonicalizes them), and each entry replaces rather than appends. The map is copied at `NewClient`, so mutating it afterwards changes nothing. + +For the two remaining TypeScript knobs there is no Go field, because `Config.HTTPClient` already covers them: `options.fetch` maps to supplying your own `*http.Client`, and `options.fetchOptions` maps to a custom `http.RoundTripper` on that client's `Transport`. + For static tokens, use `wavehouse.StaticToken(token)`: ```go @@ -114,7 +131,11 @@ wh := wavehouse.NewClient(wavehouse.Config{ ``` :::note[How the token is transmitted] -Unlike a browser's `EventSource`, Go's `net/http` client sets arbitrary headers on any request, so the Go SDK sends `Authorization: Bearer ` on every request, including SSE streams. No `?token=` query fallback (a TypeScript-in-the-browser concern; see its [equivalent note](/sdk#creating-a-client)). +The Go SDK sends `Authorization: Bearer ` on every request, including SSE streams, and never uses a `?token=` query fallback. Both SDKs work this way: the TypeScript SDK streams over `fetch` rather than `EventSource` for exactly this reason, so header auth is now the shared behavior rather than a Go-only property (see its [equivalent note](/sdk#creating-a-client)). The token is re-read from `Auth` on every reconnect attempt, so a rotating token keeps a long-lived stream alive. +::: + +:::caution[A credentialed stream will not follow a redirect] +When the stream request carries a credential — an `Auth` token or a `ClientOptions.Headers` entry — the SDK refuses any 3xx and fails the stream with a terminal `SSE_REDIRECT`. Following it would either strip `Authorization` on a cross-host hop and silently downgrade the stream to `default_role`, or forward your configured headers to wherever the redirect points. Uncredentialed streams follow redirects normally. ::: :::caution[Use HTTPS for authenticated non-local servers] @@ -170,6 +191,7 @@ Both SDKs share a wire format and feature set, verified by a shared `wire_cases. - **Streams closed explicitly.** `TableRef.Stream` and `QueryBuilder.Stream` omit `context.Context`. The returned `*StreamController` manages its own goroutine and connection, torn down by `.Close()` (deferred `stream.Close()` is usual). See [Streaming](/sdk/go/streaming). - **Generics on package functions.** Go lacks type parameters on methods; use `FetchTyped[Row]`, `Fetch[Row]`, or `SQL[Row]`. - **No implicit "await."** Call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly; `QueryBuilder` is not `PromiseLike`. +- **No third-party dependencies.** The Go SDK is stdlib-only, including its SSE frame parser. The TypeScript SDK carries exactly one runtime dependency (`eventsource-parser`, ~1.4 KB gzipped). - **Any slice batches.** Reflection allows `[]ClickRow{...}` to use the same NDJSON batch path as `[]map[string]any`. See [Queries → Insert](/sdk/go/queries#insertctx-data). ## Explore the Go SDK diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md index 69e426a1..7ec96f91 100644 --- a/docs/src/content/docs/sdk/go/pipes.md +++ b/docs/src/content/docs/sdk/go/pipes.md @@ -50,7 +50,7 @@ stream := wh.Pipe("top_pages", nil).Stream(nil) ## Pipes Admin — `client.Pipes` -Manage named query pipes. Requires the admin role (`policy.admin_role`). +Manage named query pipes. These sit behind the admin gate on `/v1/ops/*`, which a caller clears one of two ways: a JWT resolving to the policy admin role (`policy.admin_role`), or the server's non-JWT [operator key](/api#authentication) sent as `X-Operator-Key` via [`ClientOptions.Headers`](/sdk/go#clientoptions). ```go // List all pipes. diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 337cc042..8a7d223d 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -309,7 +309,7 @@ for page.HasMore && page.Next != nil { ## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` -Execute a raw SQL query via `/v1/admin/query`. This endpoint is admin-only: JWT tokens must resolve to the admin role (`admin_role`, default `"admin"`). Requests without valid tokens fall back to `default_role` and are rejected unless `default_role` is set to admin (dev-only). Alternatively, an operator key (`Authorization: Operator ` or `X-Operator-Key`) authorizes `/v1/admin/*`. Since `Config.Auth` uses `Bearer `, provide a `Config.HTTPClient` with a `Transport` that sets the `X-Operator-Key` header to use an operator key. Use `map[string]any` for dynamic schemas. +Execute a raw SQL query via `/v1/ops/query`. This endpoint is admin-only: JWT tokens must resolve to the admin role (`admin_role`, default `"admin"`). Requests without valid tokens fall back to `default_role` and are rejected unless `default_role` is set to admin (dev-only). Alternatively, an operator key (`Authorization: Operator ` or `X-Operator-Key`) authorizes `/v1/ops/*`. Since `Config.Auth` uses `Bearer `, provide a `Config.HTTPClient` with a `Transport` that sets the `X-Operator-Key` header to use an operator key. Use `map[string]any` for dynamic schemas. ```go rows, err := wavehouse.SQL[map[string]any](ctx, wh, diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index 8a076a41..e5e733ec 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -45,7 +45,14 @@ HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`). Client-sid | 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`, capped at 30s) | | 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | | 0 | `ABORTED` | No | Request canceled via `context.Context` | -| 0 | `SSE_ERROR` | Yes | Stream connection failure; delivered to subscriber's `Error` callback; auto-reconnects | +| 0 | `SSE_AUTH_ERROR` | Yes | The `Auth` provider returned an error for this attempt; the stream retries, so a token endpoint having a bad minute doesn't tear down a healthy stream | +| 0 | `SSE_NETWORK_ERROR` | Yes | Transport failure opening or holding the stream connection | +| 0 | `SSE_CONNECT_ERROR` | No | `BaseURL` is unparseable, or its scheme is not `http`/`https` — retrying cannot fix it | +| *3xx* | `SSE_REDIRECT` | No | The stream endpoint redirected while the request carried a credential, and the SDK refused to follow it | +| 200 | `SSE_BAD_CONTENT_TYPE` | No | A `200` that wasn't `text/event-stream` — something between you and WaveHouse answered (a captive portal, an auth gateway's login page) | +| 0 | `SSE_PARSE_ERROR` | Yes | A frame's JSON didn't decode; the frame is dropped and the stream continues | +| 0 | `SSE_READ_ERROR` | Yes | The connection failed mid-read; the stream reconnects from the last event ID | +| 0 | `SSE_ERROR` | Yes | Stream failure the SDK could not classify further | ```go page, err := wh.From("clicks").Fetch(ctx) @@ -62,7 +69,7 @@ if err != nil { `wavehouse.IsRetryable(err)` shortcuts the `errors.As` + `.Retryable` check. -Retries apply to all HTTP methods, matching TypeScript's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see API docs ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup for duplicate suppression. `/v1/admin/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. +Retries apply to all HTTP methods, matching TypeScript's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see API docs ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup for duplicate suppression. `/v1/ops/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. ## Full API Tree @@ -133,7 +140,7 @@ Or, inside `clients/go/`: go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go ``` -Codegen reads the admin-only `/v1/schema` endpoint; non-dev servers require an admin token or return `403`. Use `WAVEHOUSE_AUTH` instead of `--auth ` to keep tokens out of shell history and process listings. +Codegen reads the admin-only `/v1/ops/schema` endpoint; non-dev servers require an admin token or return `403`. Use `WAVEHOUSE_AUTH` instead of `--auth ` to keep tokens out of shell history and process listings. **Options:** @@ -186,7 +193,7 @@ The generator does not special-case initialisms: `event_id` becomes `EventId`, n | `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | | anything unrecognized | `any` | -Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`. For the raw-SQL path (`/v1/admin/query`), which quotes 64-bit+ integers, use `map[string]any` with `SQL[Row]`. +Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`. For the raw-SQL path (`/v1/ops/query`), which quotes 64-bit+ integers, use `map[string]any` with `SQL[Row]`. ## Testing diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index c16033b6..95950a0b 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -122,6 +122,8 @@ type StreamEvent struct { } ``` +Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3339 UTC, byte-identical to what `/v1/query` renders for the same stored value — the ingest handler rewrites them before publishing, so a live frame and a later query can't disagree on the spelling of an instant. Two consequences worth knowing: a value you sent as `2026-06-21T06:00:00.123+02:00` comes back as `2026-06-21T04:00:00.123Z` (same instant, different spelling), and the canonicalization is deliberately fail-open — a value the server can't parse, or one whose zone it can't resolve, is published verbatim. See [Timestamp canonicalization](/api#timestamp-canonicalization). + :::note[`Events()` carries events only] `Error` and `Status` are delivered exclusively via `.Subscribe(...)`. The channel closes on terminal errors (401/403/404). Pair `Events()` with a subscriber to determine why a stream ended. ::: @@ -136,9 +138,19 @@ Events buffer (up to 256) starting at `.Stream()`; events arriving before the fi | --------- | --------- | -------- | | SSE | Automatic, exponential backoff (max 30s), gap-fill replay via last event ID | HTTP/2 recommended | -Reconnect covers transport failures and retryable (5xx/429) responses. Non-retryable ones (401/403/404) are terminal: the `Error` callback fires, status goes `StatusClosed`, no reconnect. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. +Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR`, `SSE_PARSE_ERROR`, and `SSE_READ_ERROR`). Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them — see the [error-code table](/sdk/go/reference#error-handling). + +Note that `/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`. A `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. + +Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Getting Started](/sdk/go#creating-a-client)). The TypeScript SDK streams over `fetch` and authenticates the same way, so this is shared behavior rather than a Go-only property — what Go avoids is the browser's per-domain connection ceiling, not a different auth mechanism. + +Delivery across a reconnect is **at-least-once**: the server replays from the last event ID *inclusively*, so the first frame after a gap-fill is usually one you already saw. Replay reaches back only as far as the server's `mq.gap_window_minutes` (15 minutes by default); a longer outage resumes live with a hole. + +### Server-Side Policy Filtering + +Before anything reaches the client, the server applies the caller's policy to the stream: a table the role can't `select` never opens, denied columns are stripped from every frame, and a role carrying a row `filter` has non-matching rows withheld per subscriber — on live frames and on `Since` gap-fill replay alike. The claims are captured from the JWT at connect time. -Auth goes as an `Authorization: Bearer` header on every connection ([note in Getting Started](/sdk/go#creating-a-client)). Browser `EventSource` limits don't apply. +Two things follow. **Event-id gaps are normal on a filtered stream** — a gap means a row was withheld, not that a frame was dropped. And **the row filter fails closed**: a comparison the server can't prove — an unresolvable claim, a type it can't compare — withholds the row rather than passing it. See [Access control](/access-control#row-level-security). ### Client-Side Stream Filtering diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 0a0f3ef8..4f937cda 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -47,17 +47,17 @@ const server = createServer((req, res) => { body: Buffer.concat(chunks).toString("utf-8"), }; res.setHeader("Content-Type", "application/json"); - if (req.url?.startsWith("/v1/dlq")) { + if (req.url?.startsWith("/v1/ops/dlq")) { res.end(JSON.stringify({ tables: {}, total: 0 })); - } else if (req.url?.startsWith("/v1/schema") && req.method === "GET") { + } else if (req.url?.startsWith("/v1/ops/schema") && req.method === "GET") { res.end(JSON.stringify([])); - } else if (req.url === "/v1/admin/policy/validate" && req.method === "POST") { + } else if (req.url === "/v1/ops/policy/validate" && req.method === "POST") { res.end(JSON.stringify({ valid: true })); - } else if (req.url?.startsWith("/v1/admin/policy") && req.method === "GET") { + } else if (req.url?.startsWith("/v1/ops/policy") && req.method === "GET") { res.end(JSON.stringify({ tables: {} })); - } else if (req.url?.startsWith("/v1/admin/pipes/") && req.method === "GET") { + } else if (req.url?.startsWith("/v1/ops/pipes/") && req.method === "GET") { res.end(JSON.stringify({ name: "test", sql: "SELECT 1" })); - } else if (req.url === "/v1/admin/pipes" && req.method === "GET") { + } else if (req.url === "/v1/ops/pipes" && req.method === "GET") { res.end(JSON.stringify([])); } else if (req.url?.startsWith("/v1/ingest")) { // Same shapes the real server returns (internal/api/ingest.go). From 5d26c34eaea6b0d17c3e6324cc2685d6ef49d3c9 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:01:38 -0400 Subject: [PATCH 33/40] docs(changelog): note the Go SDK's Headers option in the unreleased entry --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f6bd31a7..e45eac98 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,7 +10,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added -- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{queries,streaming,pipes,admin,reference}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{queries,streaming,pipes,admin,reference}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. - **"Was this page helpful?" feedback widget on every docs page** (`docs/src/components/PageFeedback.astro` (new), `docs/src/components/Footer.astro`): a thumbs-up / thumbs-down vote below the page content, captured to PostHog as `docs_feedback` with `{ helpful, page }`. It renders from `Footer.astro`'s sidebar branch — the same indirection the Cloud CTA uses — rather than a per-page import or frontmatter flag, so every content page gets it automatically, including ones not written yet; it sits *below* the Cloud CTA on the pages that carry one, and splash pages (the homepage and 404) take the other footer branch and never render it. One vote per page per visitor: the choice is remembered in `localStorage` keyed by pathname, and a revisit renders the thanks message instead of re-prompting (storage is a nicety, not the record — a browser with storage disabled still votes). - **Settings-directory validation — `wavehouse validate [dir]`** (`internal/settings/` (new: `settings.go`, `validate.go`, `decode.go`, `finding.go`, + tests), `cmd/wavehouse/validate.go` (new, + tests), `cmd/wavehouse/main.go`): first piece of the file-based control plane (settings live in a directory of JSON documents — `roles.json`, `policies.json`, `pipes.json`, `config.json` — that a running instance will hot-reload; this change is validation-only — boot loading and reload wiring land separately). `settings.Validate(dir)` is the single gate every consumer of the directory runs: deliberately pure (no network, no ClickHouse — table/column existence stays with schema discovery, per Bring-Your-Own-Schema), and it collects **all** findings in one pass instead of failing on the first. Checks, layered: the directory holds exactly the four files (a missing file is an error — an empty document is `{}`, so absence always means deletion or a wrong path; any unexpected entry — file or directory — is an error so a typoed `polices.json` or a stray backup can't be silently ignored; dot-prefixed entries are the one carve-out, since erroring on vim swap files or the `..data` machinery Kubernetes ConfigMap mounts publish through would break hand editing and the cloud fan-out's mount pattern alike); strict JSON syntax (unknown fields rejected — the JSON form of the retired-config-key trap; empty/truncated files rejected, never read as an empty document; a leading UTF-8 byte order mark named as such instead of surfacing as a cryptic invalid-character error; a directory, unreadable file, or non-regular file (a FIFO would hang the read forever waiting for a writer; a stat gate rejects it — following symlinks, so Kubernetes ConfigMap mounts' symlink layout still passes) squatting on a settings filename named as the one real problem, not double-reported as "missing"; a top-level `null` rejected — the one well-formed document that decodes into a zero value without error, so it would silently read as "no settings"; trailing content rejected; duplicated object keys detected by a token-level pass, since `encoding/json` silently keeps the last copy); per-file shape rules (role names non-empty/unique, pipe names/SQL/param types, `config.json` bounds mirroring boot-config validation — its sections are the *tenant-owned* behavioral tunables (dedupe id_field/require_id plus per-table overrides under `dedupe.tables` — each entry overrides only the fields it names, resolving table → global → compiled default per field, so the effective id_field can never be empty — an explicit empty, whitespace-only, or whitespace-padded id_field is rejected at both levels, since an exact-match JSON key lookup would silently miss every row ([#222](https://github.com/Wave-RF/WaveHouse/issues/222)'s shape, unblocked by the file design since table names are runtime-resolved like policy grants); query default_max_rows, schema refresh_interval, CORS origins); platform-owned knobs like the SSE keepalives deliberately stay boot config); and cross-file referential integrity (every role a policy grant, `default_role`/`admin_role`, or pipe allowlist references must be declared in `roles.json`; an empty role string in a grant or allowlist is named as such — it matches no request and authorizes nobody). Warnings don't invalidate: a grant scoping the admin role (an unconditional bypass — dead config), `default_role` = admin, and a `default` on a required pipe parameter are flagged but legal. An empty `policies.json` means no policy — fail closed, matching deleted-policy semantics — and draws a warning naming the total lockout, so it announces itself at validation time instead of one 403 at a time. The CLI (`cmd/wavehouse/validate.go`, following the `health` subcommand pattern) takes the directory as an argument or from `WH_SETTINGS_DIR`, prints findings, and exits 0/1/2 (valid/invalid/usage) so CI and operators can gate config changes before they reach a running instance. The dispatch in `main.go` also grows `help` and `version` subcommands, and an unknown command is now a usage error instead of silently falling through and starting the server (`wavehouse validat` booting a listener is not a typo anyone wants); each subcommand parses its arguments with a stdlib `flag.FlagSet`, so `wavehouse -h` prints command-specific help and a stray flag or argument is a usage error rather than being silently swallowed. `WH_SETTINGS_DIR` has a single authority: `config.EnvSettingsDir`, with a reflection test pinning the `settings.dir` struct tag to it. The directory's location joins boot config as `settings.dir` (`WH_SETTINGS_DIR`; `internal/config/config.go`, `config.yaml`, `docs/src/content/docs/configuration.mdx`) — boot-tier by necessity, since it's the pointer the reload machinery follows; no default, same silent-misconfiguration reasoning as `policy.file_path`. From 90d59ef0a84461ce3f14a1027a4dd6c7b98e80f8 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:20:55 -0400 Subject: [PATCH 34/40] style(docs): unwrap hard-wrapped prose in the Go SDK pages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The six Go SDK pages were authored before #489 landed WH001 (no-hard-wrapped-prose), so they wrapped prose at ~76 columns while the rest of the docs tree had been reflowed. `make fix` output, whitespace only — verified no content changed in any of the four files. --- docs/src/content/docs/sdk/go/index.md | 9 ++------- docs/src/content/docs/sdk/go/queries.md | 9 +-------- docs/src/content/docs/sdk/go/reference.md | 6 +----- docs/src/content/docs/sdk/go/streaming.md | 9 +-------- 4 files changed, 5 insertions(+), 28 deletions(-) diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md index 219c1105..65afc19f 100644 --- a/docs/src/content/docs/sdk/go/index.md +++ b/docs/src/content/docs/sdk/go/index.md @@ -3,15 +3,10 @@ title: "Go SDK" description: "Zero-dependency Go client SDK — query builder, real-time streaming, codegen." --- -`github.com/Wave-RF/WaveHouse/clients/go` — zero third-party runtime -dependency Go client for WaveHouse (stdlib only). +`github.com/Wave-RF/WaveHouse/clients/go` — zero third-party runtime dependency Go client for WaveHouse (stdlib only). :::tip[Looking for the TypeScript SDK?] -This page and the rest of `/sdk/go/*` cover the Go client. The -JavaScript/TypeScript client (`@wavehouse/sdk`) has its own docs starting at -[SDK Overview](/sdk) — the two SDKs speak the same wire format, so anything -you learn about WaveHouse's query builder, streaming, or admin endpoints on -either page mostly carries over. +This page and the rest of `/sdk/go/*` cover the Go client. The JavaScript/TypeScript client (`@wavehouse/sdk`) has its own docs starting at [SDK Overview](/sdk) — the two SDKs speak the same wire format, so anything you learn about WaveHouse's query builder, streaming, or admin endpoints on either page mostly carries over. ::: ## Installation diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md index 8a7d223d..c1fc8e2a 100644 --- a/docs/src/content/docs/sdk/go/queries.md +++ b/docs/src/content/docs/sdk/go/queries.md @@ -3,14 +3,7 @@ title: "Go SDK Queries" description: "Tables, the chainable query builder, pagination, and raw SQL in the WaveHouse Go SDK." --- -Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: -table references, the chainable query builder, cursor pagination, and the -admin-only raw-SQL escape hatch. Every request-response operation takes a -`context.Context` as its first argument and returns `(T, error)`; the -chainable builder methods and `.Stream(opts)` are the exceptions — see -[Error Handling](/sdk/go#error-handling). Compare with the TypeScript SDK's -[Queries](/sdk/queries) page, which covers the same surface with a -`Result`-returning, `PromiseLike` builder. +Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: table references, the chainable query builder, cursor pagination, and the admin-only raw-SQL escape hatch. Every request-response operation takes a `context.Context` as its first argument and returns `(T, error)`; the chainable builder methods and `.Stream(opts)` are the exceptions — see [Error Handling](/sdk/go#error-handling). Compare with the TypeScript SDK's [Queries](/sdk/queries) page, which covers the same surface with a `Result`-returning, `PromiseLike` builder. ## Tables — `client.From(table)` diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md index e5e733ec..b7123fce 100644 --- a/docs/src/content/docs/sdk/go/reference.md +++ b/docs/src/content/docs/sdk/go/reference.md @@ -3,11 +3,7 @@ title: "Go SDK Reference & CLI" description: "Error codes, context cancellation, the full API tree, and the codegen CLI for the WaveHouse Go SDK." --- -Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: -cancellation, the error model behind every request-response call's `(T, error)` return, -the complete API tree at a glance, and the `wavehouse-codegen` tool that -ships with the module. Compare with the TypeScript SDK's -[Reference & CLI](/sdk/reference) page. +Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: cancellation, the error model behind every request-response call's `(T, error)` return, the complete API tree at a glance, and the `wavehouse-codegen` tool that ships with the module. Compare with the TypeScript SDK's [Reference & CLI](/sdk/reference) page. ## Context Cancellation diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 95950a0b..69487e16 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -3,14 +3,7 @@ title: "Go SDK Streaming & Live Queries" description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in the WaveHouse Go SDK." --- -Real-time consumption with `github.com/Wave-RF/WaveHouse/clients/go`: SSE -event streams from tables, builders, and pipes, plus live queries that -backfill history before going live. Builders and table refs come from -[Queries](/sdk/go/queries). Compare with the TypeScript SDK's -[Streaming & Live Queries](/sdk/streaming) page — the two implement the same -protocol and mostly the same client-side filtering, but connection lifecycle -differs: Go streams are goroutine-backed and closed explicitly, not tied to -a `context.Context` or a browser's `EventSource`. +Real-time consumption with `github.com/Wave-RF/WaveHouse/clients/go`: SSE event streams from tables, builders, and pipes, plus live queries that backfill history before going live. Builders and table refs come from [Queries](/sdk/go/queries). Compare with the TypeScript SDK's [Streaming & Live Queries](/sdk/streaming) page — the two implement the same protocol and mostly the same client-side filtering, but connection lifecycle differs: Go streams are goroutine-backed and closed explicitly, not tied to a `context.Context` or a browser's `EventSource`. ## Streaming From c460c02c23fddc0a7f94ec86d176c968a088b1d9 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:22:15 -0400 Subject: [PATCH 35/40] build(lint): put the shared conformance runner under Biome MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit biome.json's files.includes covered clients/ts, tests/e2e/sdk, docs, and scripts, but not tests/conformance — so the 311-line conformance_ts.mjs this branch adds was invisible to make lint-ts / fmt-ts / fix-ts. Including it surfaced one formatting fix, applied here. --- biome.json | 1 + tests/conformance/conformance_ts.mjs | 11 ++++++++--- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/biome.json b/biome.json index e29be7f3..700f3729 100644 --- a/biome.json +++ b/biome.json @@ -10,6 +10,7 @@ "clients/ts/src/**", "clients/ts/*.{ts,js,mjs,cjs,json}", "tests/e2e/sdk/**/*.{ts,js,mjs,cjs,json}", + "tests/conformance/**/*.{ts,js,mjs,cjs,json}", "docs/**/*.{ts,js,mjs,cjs,json}", "scripts/**/*.{ts,js,mjs,cjs}" ] diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs index 4f937cda..b145e994 100644 --- a/tests/conformance/conformance_ts.mjs +++ b/tests/conformance/conformance_ts.mjs @@ -22,12 +22,16 @@ let createClient; try { ({ createClient } = await import(join(__dirname, "../../clients/ts/dist/index.js"))); } catch (err) { - console.error("Cannot load the TypeScript SDK build. Run the SDK build first (e.g. `pnpm --dir clients/ts build`)."); + console.error( + "Cannot load the TypeScript SDK build. Run the SDK build first (e.g. `pnpm --dir clients/ts build`).", + ); console.error(err.message); process.exit(1); } -const cases = JSON.parse(readFileSync(join(__dirname, "../../clients/go/testdata/wire_cases.json"), "utf-8")); +const cases = JSON.parse( + readFileSync(join(__dirname, "../../clients/go/testdata/wire_cases.json"), "utf-8"), +); let lastCapture = { method: "", path: "", contentType: "", body: "" }; @@ -303,7 +307,8 @@ for (const f of failures) { if (failed > 0 || skipped > 0 || passed === 0) { if (passed === 0) console.log(" ✗ nothing ran — every case skipped or the fixture is empty\n"); - if (skipped > 0) console.log(" ✗ skipped cases break cross-SDK parity — wire up the endpoint above\n"); + if (skipped > 0) + console.log(" ✗ skipped cases break cross-SDK parity — wire up the endpoint above\n"); process.exit(1); } else { console.log(" ✓ All cases passed\n"); From 941b09758bcaf178d7bdc7cd3b225fc925ac7345 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:22:15 -0400 Subject: [PATCH 36/40] style(docs): unwrap the Go SDK aside in sdk/index.mdx MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit WH001's autofix deliberately never runs over .mdx, so this aside — added on this branch before the rule landed — had to be joined by hand. Text unchanged. --- docs/src/content/docs/sdk/index.mdx | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index 8d62bdfb..652f9ba5 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -8,14 +8,7 @@ import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components `@wavehouse/sdk` — TypeScript client for WaveHouse. One runtime dependency: `eventsource-parser` (~1.4 KB gzipped, itself dependency-free), which frames the SSE stream. :::tip[Writing Go instead?] -WaveHouse also ships an official Go SDK -(`github.com/Wave-RF/WaveHouse/clients/go`) — zero third-party dependencies, -`context.Context`-first, generics for typed rows. See the -[Go SDK docs](/sdk/go). The two clients speak the same wire format, so -everything below about tables, the query builder, streaming, and admin -endpoints carries over conceptually, but API and lifecycle details differ — -Go uses context-first calls and package-level generics, and streams must be -closed explicitly. +WaveHouse also ships an official Go SDK (`github.com/Wave-RF/WaveHouse/clients/go`) — zero third-party dependencies, `context.Context`-first, generics for typed rows. See the [Go SDK docs](/sdk/go). The two clients speak the same wire format, so everything below about tables, the query builder, streaming, and admin endpoints carries over conceptually, but API and lifecycle details differ — Go uses context-first calls and package-level generics, and streams must be closed explicitly. ::: ## Installation From c538ee37e98fea919c6f04913728c7b810a66874 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:22:28 -0400 Subject: [PATCH 37/40] build(deps): track the nested clients/go module in Dependabot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The gomod entry was `directory: /`, which covers the root module only — Dependabot does not descend into nested modules, so clients/go/go.mod was untracked. Converted to `directories:` (the form the github-actions entry already uses for the same reason) so both share one schedule, group, and commit prefix rather than forking the policy into a second block. clients/go is stdlib-only today, so this catches the first dependency it takes on rather than closing a gap that already exists — the same argument the github-actions comment block makes about composite actions. --- .github/dependabot.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index e1c5d695..38d5c98c 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,8 +1,17 @@ version: 2 updates: # Go modules + # + # TWO directories, not one — the same trap as github-actions below: + # `directory: /` covers the root module only, and Dependabot does not + # descend into nested modules. clients/go is its own module, so its + # go.mod needs its own entry here. That module is stdlib-only today + # (no `require` block, no go.sum), so this catches the first dependency + # it takes on rather than closing a gap that already exists. - package-ecosystem: gomod - directory: / + directories: + - / + - /clients/go schedule: interval: weekly day: monday From 167e16db68f44c9db2dd5b98343c43fb25716bd3 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:25:06 -0400 Subject: [PATCH 38/40] build(labels): map the Go SDK paths to area/sdk MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit area/sdk globbed clients/ts/** and tests/e2e/sdk/** only, so every file in the new Go SDK and the shared conformance runner went unlabeled. Adds clients/go/** and tests/conformance/**. Also broadens the dependencies label from root-anchored go.mod/go.sum to **/go.mod and **/go.sum, so the nested clients/go module's manifests get labeled the way clients/ts/package.json already does via **/package.json. Left area/infra root-anchored — the TypeScript SDK's manifest doesn't carry that label either. Verified by replaying the config's globs over the branch's diff: clients/go sources land on area/sdk + go, the codegen CLI on area/sdk, and clients/go/go.mod on area/sdk + dependencies. Note this takes effect from the next PR, not this one: housekeeping.yml resolves labeler.yml from the default branch, not the PR head. --- .github/labeler.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/.github/labeler.yml b/.github/labeler.yml index 4df42c01..d164e157 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -43,7 +43,9 @@ - changed-files: - any-glob-to-any-file: - "clients/ts/**" + - "clients/go/**" - "tests/e2e/sdk/**" + - "tests/conformance/**" "area/docs": - changed-files: @@ -83,8 +85,8 @@ dependencies: - changed-files: - any-glob-to-any-file: - - "go.mod" - - "go.sum" + - "**/go.mod" + - "**/go.sum" - "**/package.json" - "**/pnpm-lock.yaml" - "**/package-lock.json" From 73edcbe3e74b5d9c444b26e6e379857f589deb52 Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:33:09 -0400 Subject: [PATCH 39/40] test(cov): give the Go SDK its own coverage floor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit test-go-sdk ran a bare `go test -race ./...` — no profile, no threshold — so the Go SDK would have shipped as the only component in the repo with no coverage gate while its TypeScript peer has three. It now collects covdata the same way the root-module Go suites do and renders through scripts/cov, with a `go-sdk` suite gated at 75% against a measured 82.7% (clients/go 88.6%, cmd/wavehouse-codegen 55.8%). That ratio sits inside the band the existing floors already use — unit is 80 against ~91% actual — leaving headroom without being slack enough to rot. Gated standalone rather than merged into the Go total: the total feeds threshold.total and the README badge, and folding a shipped client library into the server's project-wide number would move that badge for unrelated reasons. The same separation the TypeScript SDK gets via ts-*. Being a nested module, clients/go is invisible to the root -coverpkg anyway, so it cannot leak into the total and needs no exclude.paths entry — verified: zero clients/go rows in the merged profile, Go total unmoved at 81.2%. One wrinkle worth recording: `go tool cover -html` resolves a profile's package paths through the module in the working directory, so rendering from the repo root fails to find the nested packages. renderHTML now runs with cmd.Dir set to clients/go and absolute paths. CI needed no functional change — the unit job already uploads the whole tmp/coverage tree, so the fragment arrives and the coverage job gates it. --- .claude/commands/cover.md | 5 +- .github/workflows/ci.yml | 6 ++ .testcoverage.yml | 16 +++++- AGENTS.md | 2 +- Makefile | 27 ++++++++- docs/src/content/docs/development.md | 6 +- scripts/cov/main.go | 83 +++++++++++++++++++++++++++- 7 files changed, 132 insertions(+), 13 deletions(-) diff --git a/.claude/commands/cover.md b/.claude/commands/cover.md index 021def65..93b5a33e 100644 --- a/.claude/commands/cover.md +++ b/.claude/commands/cover.md @@ -1,6 +1,6 @@ --- description: Render coverage HTML for a suite and surface drops below threshold -argument-hint: [unit|integration|e2e|sdk|merge|all] (default: merge whatever exists) +argument-hint: [unit|integration|e2e|go-sdk|sdk|merge|all] (default: merge whatever exists) --- Generate the coverage report and surface anything below threshold from `.testcoverage.yml`. @@ -13,10 +13,11 @@ Behavior: - **unit**: `make test-unit` (gates per-suite + writes `tmp/coverage/unit/`) - **integration**: `make test-integration` (requires Docker) - **e2e**: `make test-e2e` (requires Docker; orchestrator + cover binary) +- **go-sdk**: `make test-go-sdk` (nested module `clients/go`; gates against `suites.go-sdk`, rendered separately and never merged into the Go total) - **ts-unit**: `make test-ts` (SDK unit tests + coverage + gate against `suites.ts-unit`) - **ts-e2e**: emitted as a side effect of `make test-e2e` (the orchestrator always passes `--coverage` to the e2e vitest run; informational only, no standalone gate) - **ts-total**: `make cov` (runs `cov report` — one consolidated Go + TS summary with per-suite HTML links + all gates; fails if *no* suite has data) -- **all**: `make test-all` (all four suites sequentially + `make cov`) +- **all**: `make test-all` (every suite sequentially + `make cov`) After the run completes: diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 12fc070c..deebc508 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -206,6 +206,12 @@ jobs: go-cache-suffix: "-unit" - name: Run Go unit tests + SDK vitest + Go SDK tests run: make test-unit test-ts test-go-sdk test-conformance-ts COV_DEFER=1 + # This one fragment carries THREE suites' data: unit covdata, ts-unit + # (vitest/istanbul) and go-sdk covdata from the nested clients/go + # module — every target above ran under COV_DEFER, so the `coverage` + # job renders and gates all three. No per-suite paths here on purpose: + # uploading the whole tmp/coverage tree means a new suite collected by + # this job needs no workflow change. - name: Upload coverage fragment uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: diff --git a/.testcoverage.yml b/.testcoverage.yml index 669d65be..ee02af8d 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -12,7 +12,9 @@ # together, so `threshold.total` below applies to the *project-wide* # coverage number — unit alone covers only `./internal/...` packages # exercised by `*_test.go` files (below total threshold); merged -# coverage adds integration- and e2e-only paths and clears it. +# coverage adds integration- and e2e-only paths and clears it. The +# nested-module suites (go-sdk) are gated separately and never merged +# in — see suites.go-sdk. profile: tmp/coverage/total/coverage.txt local-prefix: github.com/Wave-RF/WaveHouse @@ -30,6 +32,18 @@ suites: unit: 80 integration: 20 e2e: 60 + # Go SDK (clients/go) — a NESTED module, so its coverage is rendered and + # gated on its own and is deliberately NOT part of the merged Go total + # above. Nothing from clients/go can leak into that total: the root + # module can't see a nested one (`go list ./...` at the repo root never + # yields clients/go), so the unit/integration/e2e `-coverpkg=./...` never + # reaches these files — which is also why there is no `^clients/go/` + # entry under exclude.paths; there is nothing to exclude. + # Measured 82.7% when this floor was set (SDK package 88.6%, + # cmd/wavehouse-codegen 55.8%). 75 leaves the same order of headroom + # unit's 80 leaves under its real ~91%. Raise it as the codegen + # command's tests fill in. + go-sdk: 75 # TypeScript SDK suites — see scripts/cov for the merge / render logic. # vitest gates via --coverage.thresholds.statements; the merged ts-total # is gated by `cov ts-merge` against the value below. Tune ts-total diff --git a/AGENTS.md b/AGENTS.md index 10a2dcb6..c06b32c5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -125,7 +125,7 @@ Tooling notes (the non-obvious bits `make help` won't tell you): - **Policy helpers**: Use `policy.NewMemoryStore(p)` for in-memory policy testing without NATS. - **Pipes helpers**: Use `pipes.NewMemoryStore(queries...)` for in-memory pipes testing without NATS. - **Response assertions**: Use `testutil.AssertJSONResponse(t, rec, status, expected)` and `testutil.AssertJSONContains(t, rec, status, substring)`. -- **Coverage target**: 80% project-wide (CI enforces `threshold.total` in `.testcoverage.yml` against the merged unit + integration + e2e profile). Per-suite minima also enforced: unit 80%, integration 20%, e2e 60%, sdk 50%. Aim for 80%+ on new code. Coverage is published as a README badge (Go merged-total) and as PR comments via GitHub Code Quality — see `.github/workflows/README.md` "Coverage publishing"; the gate is unchanged. +- **Coverage target**: 80% project-wide (CI enforces `threshold.total` in `.testcoverage.yml` against the merged unit + integration + e2e profile). Per-suite minima also enforced: unit 80%, integration 20%, e2e 60%, go-sdk 75%, ts SDK 50%. The Go SDK (`clients/go`) is a nested module — invisible to the root module's `-coverpkg=./...`, so it is gated on its own `suites.go-sdk` floor and never merged into the project-wide total. Aim for 80%+ on new code. Coverage is published as a README badge (Go merged-total) and as PR comments via GitHub Code Quality — see `.github/workflows/README.md` "Coverage publishing"; the gate is unchanged. - **Every new function should have corresponding test cases.** Run `make lint` and `make test` before considering work complete. - **E2E tests via SDK**: The TypeScript SDK is the primary E2E test harness. Tests in `tests/e2e/sdk/` exercise the full pipeline (ingest → ClickHouse → query) and simultaneously validate backend behavior and SDK correctness. Use `make test-e2e` to run. Add new E2E scenarios as `tests/e2e/sdk/*.test.ts` files using helpers from `tests/e2e/sdk/helpers.ts`. - **Per-suite table isolation**: Each e2e test file owns its own ClickHouse tables — `clicks_` / `events_` / `users_`, generated from `tests/e2e/sdk/tables.ts` and created by `setup.ts`. A new test file must (1) add its suite name to `SUITES` in `tables.ts` and (2) get its names via `const T = suiteTables("")`, then reference `T.clicks` etc. — never a bare `clicks`. This makes cross-file *data* contamination structurally impossible. Files still run **sequentially** (`vitest.config.ts` `maxWorkers: 1`): running them in parallel is blocked by shared *global policy* state (several files read-modify-write the single policy document; `streaming.test.ts` flips the global `default_role`), so policy-mutating tests snapshot the full policy and restore it. Dropping `maxWorkers: 1` is a deferred follow-up tracked in #214 (per-table policy storage; see `docs/src/content/docs/ingest-pipeline.md` § Deferred). diff --git a/Makefile b/Makefile index 95ff1879..a4696334 100644 --- a/Makefile +++ b/Makefile @@ -199,6 +199,9 @@ ACTIONLINT := $(LOCAL_BIN)/actionlint-$(ACTIONLINT_VERSION) COV_UNIT := tmp/coverage/unit COV_INT := tmp/coverage/integration COV_E2E := tmp/coverage/e2e +# go-sdk is the nested module at clients/go — same layout, own gate, but +# deliberately outside COV_TOTAL (see test-go-sdk below). +COV_GOSDK := tmp/coverage/go-sdk COV_TOTAL := tmp/coverage/total # --- Coverage Thresholds ------------------------------------------------------ @@ -805,10 +808,28 @@ test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against sui # go.mod), so it's outside test-unit's ./internal/... ./cmd/... scope and # needs its own target. -race because the SDK's streaming subsystem is the # most concurrent code in the repo. +# +# Coverage is collected exactly like the root-module Go suites (covdata into +# tmp/coverage//data via -test.gocoverdir), so `cov render go-sdk` +# renders + gates it with no new machinery and CI's coverage fragment — +# `path: tmp/coverage` on the unit job, which already runs this target — +# carries it to the `coverage` job unchanged. -coverpkg=./... resolves +# inside clients/go, so the denominator is the SDK package + the codegen +# command, nothing from the server. +# +# The go-sdk suite is NOT part of the merged Go total: a nested module is +# invisible to the root module (`go list ./...` at the repo root never +# yields clients/go), so the other suites' -coverpkg=./... cannot reach +# these files — they can't leak into tmp/coverage/total, and no +# exclude.paths entry is needed to keep them out. Same separation the TS +# SDK gets via ts-*. Gate: suites.go-sdk in .testcoverage.yml. .PHONY: test-go-sdk -test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests +test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests + render coverage + gate threshold @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" - @cd clients/go && go test -race ./... + @rm -rf $(COV_GOSDK)/data && mkdir -p $(COV_GOSDK)/data + @cd clients/go && GOCOVERDIR="$(CURDIR)/$(COV_GOSDK)/data" go test -cover -coverpkg=./... -race ./... \ + -args -test.gocoverdir="$(CURDIR)/$(COV_GOSDK)/data" + @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render go-sdk; fi # test-conformance-ts: the TS half of the cross-SDK wire-format conformance # suite (the Go half is clients/go/conformance_test.go, run by test-go-sdk). @@ -830,7 +851,7 @@ test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVE .PHONY: test-all test-all: ## Run all suites sequentially + one consolidated Go + TS coverage report + gates @$(MAKE) test-unit COV_DEFER=1 - @$(MAKE) test-go-sdk + @$(MAKE) test-go-sdk COV_DEFER=1 @$(MAKE) test-conformance-ts @$(MAKE) test-ts COV_DEFER=1 @$(MAKE) test-integration COV_DEFER=1 diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index a1373581..cc07e18f 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -324,7 +324,7 @@ make ci make cov ``` -Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. +Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-go-sdk`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. `test-go-sdk` is gated but **not** merged: `clients/go` is a nested Go module, invisible to the root module's `-coverpkg=./...`, so its statements can never reach `tmp/coverage/total` — it carries its own `suites.go-sdk` floor instead, the same way the TS SDK carries `ts-*`. The remaining SDK/conformance targets (`test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. **Verbose output**: Use `V=1` to switch from the compact `pkgname-and-test-fails` format to `standard-verbose` on `test-unit` / `test-integration`, and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-ts` and the Go SDK / conformance targets ignore it. @@ -338,7 +338,7 @@ Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test- | -------- | -------- | ------- | ------- | | Unit tests | `internal/*/_test.go` | No | `make test` | | SDK unit tests (TS) | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | -| SDK unit tests (Go) | `clients/go/*_test.go` (nested Go module) | No | `make test-go-sdk` (runs with `-race`) | +| SDK unit tests (Go) | `clients/go/*_test.go` (nested Go module) | No | `make test-go-sdk` (runs with `-race`, always includes coverage + gate) | | Wire-format conformance | `clients/go/conformance_test.go` + `tests/conformance/conformance_ts.mjs`, both replaying `clients/go/testdata/wire_cases.json` | No | Go half via `make test-go-sdk`; TS half via `make test-conformance-ts` | | SDK E2E (Go, live server) | `clients/go/e2e_test.go` (`//go:build e2e`) | No | `make test-go-sdk-e2e` (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | | Integration tests (Go) | `tests/integration/*_test.go` | Yes | `make test-integration` | @@ -534,7 +534,7 @@ Run `make help` to see all targets. Key ones: | **Test** | | | `make test` | Alias for `test-unit` + `test-go-sdk` | | `make test-unit` | Go unit tests + render coverage + gate suite threshold | -| `make test-go-sdk` | Go SDK (`clients/go`, nested module) unit tests with `-race` | +| `make test-go-sdk` | Go SDK (`clients/go`, nested module) unit tests with `-race` + render coverage + gate `suites.go-sdk` (own gate; never merged into the Go total) | | `make test-go-sdk-e2e` | Go SDK E2E against a live server (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | | `make test-conformance-ts` | TS SDK wire-format conformance against the shared `wire_cases.json` fixture (builds the TS SDK first) | | `make test-integration` | Go integration tests (requires Docker) + coverage gate | diff --git a/scripts/cov/main.go b/scripts/cov/main.go index f4250bfa..2a3948bb 100644 --- a/scripts/cov/main.go +++ b/scripts/cov/main.go @@ -59,6 +59,30 @@ const ( // see the ts-total path below. var goSuites = []string{"unit", "integration", "e2e"} +// Go suites that produce covdata in the same layout as goSuites and are +// rendered + gated identically, but are deliberately NOT merged into the +// Go total: they come from a NESTED module (clients/go has its own +// go.mod). The root module cannot see a nested one — `go list ./...` at +// the repo root never yields clients/go — so the goSuites' -coverpkg=./... +// can't reach these files in the first place; they can only ever be in +// tmp/coverage/total if we put them there, which we don't. Folding a +// shipped client library into the server's project-wide number (and into +// the README badge `cov badge` derives from it) would move that number for +// reasons that have nothing to do with the server, so the SDK gets its own +// floor instead — the same separation the TS SDK gets via ts-*. +var standaloneGoSuites = []string{"go-sdk"} + +// suiteModuleDir maps a suite to the module directory its covdata was +// produced in, for suites that aren't the root module. `go tool cover +// -html` reads the source of every package named in the profile and +// resolves it through the module in the process's working directory, so a +// nested module's profile has to be rendered from inside that module — +// from the repo root the tool fails with "no required module provides +// package github.com/Wave-RF/WaveHouse/clients/go/...". `go tool covdata +// textfmt` has no such constraint (it only reads the covdata files), so +// only the HTML step needs the chdir. +var suiteModuleDir = map[string]string{"go-sdk": "clients/go"} + // TypeScript SDK suites (vitest). ts-unit comes from clients/ts; ts-e2e // from tests/e2e/sdk run with --coverage. Both produce Istanbul-format // coverage-final.json that `cov ts-merge` combines into ts-total. @@ -212,7 +236,7 @@ func goSuiteCoverage(c *config, suite string) (rows []pkgRow, total, covered int if err = sh("go", "tool", "covdata", "textfmt", "-i="+dataDir, "-o", profile); err != nil { return nil, 0, 0, "", err } - if err = sh("go", "tool", "cover", "-html="+profile, "-o", htmlOut); err != nil { + if err = renderHTML(suite, profile, htmlOut); err != nil { return nil, 0, 0, "", err } rows, total, covered, err = parseCoverage(profile, c, c.excludesFor(suite)) @@ -222,6 +246,27 @@ func goSuiteCoverage(c *config, suite string) (rows []pkgRow, total, covered int return rows, total, covered, htmlOut, nil } +// renderHTML turns a textfmt profile into the clickable HTML report. For a +// suite whose covdata came from a nested module (see suiteModuleDir) the +// tool runs with that module as its working directory — otherwise it can't +// resolve the profile's package paths to source and bails — so the profile +// and output paths are made absolute first. +func renderHTML(suite, profile, htmlOut string) error { + dir, nested := suiteModuleDir[suite] + if !nested { + return sh("go", "tool", "cover", "-html="+profile, "-o", htmlOut) + } + absProfile, err := filepath.Abs(profile) + if err != nil { + return err + } + absHTML, err := filepath.Abs(htmlOut) + if err != nil { + return err + } + return shIn(dir, "go", "tool", "cover", "-html="+absProfile, "-o", absHTML) +} + func renderSuite(c *config, suite string) error { rows, total, covered, htmlOut, err := goSuiteCoverage(c, suite) if err != nil { @@ -247,7 +292,7 @@ func renderSuite(c *config, suite string) error { // "one side legitimately absent" (skip, fine) from "nothing ran at all" // (fail, because the caller expected a gate). func hasAnyCoverage() bool { - for _, s := range goSuites { + for _, s := range slices.Concat(goSuites, standaloneGoSuites) { if hasCovdata(filepath.Join(root, s, "data")) { return true } @@ -298,6 +343,13 @@ func merge(c *config) error { for _, s := range goSuites { fmt.Printf(" %s%-13s%s %s\n", cyan, s+":", reset, suitePct(c, s)) } + // Nested-module Go suites: gated on their own, never merged above. + for _, s := range standaloneGoSuites { + if pct := suitePct(c, s); pct != "n/a" { + fmt.Printf(" %s%-13s%s %s %s(separate gate; not in merge above)%s\n", + cyan, s+":", reset, pct, yellow, reset) + } + } // Surface TS SDK coverage alongside the Go total — informational only, // not part of the Go merged number above. `make cov` is the gate. for _, s := range append(tsSuites, "ts-total") { @@ -551,6 +603,25 @@ func report(c *config) error { }) } + // --- Nested-module Go suites (own gate, below the Go total) --- + // Rendered and gated exactly like the suites above, but listed after + // the total they are deliberately not part of — see standaloneGoSuites. + for i, s := range standaloneGoSuites { + if !hasCovdata(filepath.Join(root, s, "data")) { + rows = append(rows, reportRow{name: s, pct: "n/a", rule: i == 0}) + continue + } + _, total, covered, html, err := goSuiteCoverage(c, s) + if err != nil { + return err + } + th := thresholdFor(c, s) + rows = append(rows, reportRow{ + name: s, pct: formatPctBare(covered, total), gated: true, thresh: th, + pass: meetsThreshold(covered, total, th), html: html, rule: i == 0, + }) + } + // --- TS suites + merged ts-total --- merged, err := mergeTSArtifacts("html", "json-summary") if err != nil { @@ -917,9 +988,15 @@ func meetsThreshold(covered, total, threshold int) bool { // sh runs an external command with stdio wired through. Every call site // passes "go" as the program and a fixed series of "tool", "", // flag, … args; the only variable bits are paths we computed ourselves. +func sh(name string, args ...string) error { return shIn("", name, args...) } + +// shIn is sh with an explicit working directory ("" = inherit ours) — for +// the one tool that cares which module it runs in, `go tool cover -html` +// on a nested module's profile. See renderHTML. // #nosec G204,G702 — name and args are not user input. -func sh(name string, args ...string) error { +func shIn(dir, name string, args ...string) error { cmd := exec.CommandContext(context.Background(), name, args...) + cmd.Dir = dir cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr return cmd.Run() From 3ed16983e418ffdd1aed37d3f98ed10b0d53d48b Mon Sep 17 00:00:00 2001 From: Jack Woods Date: Fri, 21 Aug 2026 15:48:03 -0400 Subject: [PATCH 40/40] fix(sdk): compare stream filter timestamps as instants, not as text MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The client-side filter behind .Stream() compared every string operand lexically. That was defensible before #402; it isn't now. The server canonicalizes every top-level DateTime/DateTime64 value to RFC 3339 UTC before publishing, so a payload reads 2026-06-21T04:00:00Z while a caller's filter constant may name the same instant as 2026-06-21T06:00:00+02:00 — and lexically the payload sorts BELOW the constant, so OpGte withheld a row that was chronologically equal, and OpEq called two spellings of one instant different. Both sides now parse as instants, mirroring what internal/policy's row filter does for a DateTime column. Deliberately narrow: only RFC 3339 with an explicit offset or Z counts. A zone-less spelling names an instant only relative to the column's declared timezone, which the server reads from the schema and a stream subscriber does not have — reading it as UTC would move the instant, so those fall through rather than being silently reinterpreted. A ',' fraction is ISO 8601 but not ClickHouse, and is refused for the same reason the ingest grammar refuses it. The operand length is pre-gated at 64 bytes like the server's, so a megabyte 'timestamp' isn't scanned once per filter per event. Ordering an instant against a non-instant now fails closed instead of falling back to text, which could admit rows the query path excludes — the same direction the server errs in. The usual trigger is a zone-less constant, which now yields no rows rather than wrong ones. Also: a column missing from the payload no longer matches the literal string "" through the fmt.Sprint equality fallback. Not fixed here, and worth its own change: the TypeScript SDK's matchesFilters compares the same way, and its compareOrdered comment still asserts that lexicographic order 'is correct for ISO-8601 timestamps' — true only while both sides share an offset spelling, which #402 stopped guaranteeing. Its unknown-operator branch also returns true where Go and the server fail closed. --- CHANGELOG.md | 2 + clients/go/stream.go | 73 +++++++++++++- clients/go/stream_test.go | 116 ++++++++++++++++++++++ docs/src/content/docs/sdk/go/streaming.md | 14 +++ 4 files changed, 204 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e45eac98..849b018f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,8 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Fixed +- **Go SDK client-side stream filters compare timestamps as instants, not as text** (`clients/go/stream.go`, `clients/go/stream_test.go`, `docs/src/content/docs/sdk/go/streaming.md`): the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing (#402), so a payload reads `2026-06-21T04:00:00Z` while a caller's filter constant may name the same instant as `2026-06-21T06:00:00+02:00`. Compared as text those disagree in both directions — lexically the payload sorts *below* the constant, so `OpGte` withheld a row that was chronologically equal. Both sides now parse as instants, mirroring the server's row-filter rule for DateTime columns. Only unambiguous spellings count (RFC 3339 with an explicit offset or `Z`): a zone-less constant names an instant only relative to the column's declared timezone, which a stream subscriber doesn't have, so reading it as UTC would move the instant. Ordering an instant against a non-instant now fails closed rather than falling back to text comparison. Also fixes a missing column matching the literal string `""` through the equality fallback. The TypeScript SDK's `matchesFilters` has the same text-comparison behavior and needs the same change for parity. + - **TypeScript SDK documentation corrections found while writing the Go SDK's parity docs** (`docs/src/content/docs/sdk/{reference,queries,streaming}.md`): the error-code table described `401` as "missing or invalid JWT" when a *missing* token is actually evaluated as `default_role` — succeeding or denied with `403`, never `401` (`internal/auth/auth.go`, `internal/api/errors.go`) — and only a present-but-invalid or expired token yields `401`. `.aggregate()` was documented as accepting a "custom fn", but the server enforces an allowlist (`internal/query/builder.go`); the allowed set is now listed. Live queries gained a caution for a real footgun: the backfill dedup boundary comes from the fetched rows' `received_timestamp`, so a `.select(...)` projection omitting that column silently disables dedup and delivers overlap-window events twice. No SDK code changed — these were pre-existing gaps between the TS docs and the server's behavior. ## [0.1.0] - 2026-08-19 diff --git a/clients/go/stream.go b/clients/go/stream.go index 48ce32e8..a9388896 100644 --- a/clients/go/stream.go +++ b/clients/go/stream.go @@ -640,17 +640,68 @@ func evaluateFilter(actual any, op string, expected any, re *regexp.Regexp) bool } // equalValues compares two values for equality, normalizing numeric types -// (JSON decodes numbers as float64, but callers may pass int). +// (JSON decodes numbers as float64, but callers may pass int) and comparing +// timestamps as instants rather than as text — see asInstant. func equalValues(a, b any) bool { + // nil only equals nil: without this the fmt.Sprint fallback would match a + // missing column against the literal string "". + if a == nil || b == nil { + return a == nil && b == nil + } if af, aOK := toFloat64(a); aOK { if bf, bOK := toFloat64(b); bOK { return af == bf } } + if at, aOK := asInstant(a); aOK { + if bt, bOK := asInstant(b); bOK { + return at.Equal(bt) + } + } // fmt.Sprint is safe for all types (no panic on maps/slices). return fmt.Sprint(a) == fmt.Sprint(b) } +// maxTimeOperandChars mirrors the server's row-filter pre-gate: the longest +// spelling the ingest grammar accepts (RFC 3339 with nanoseconds and a numeric +// offset) is 35 bytes, so 64 is generous slack while keeping a megabyte +// "timestamp" from being scanned once per filter per event. +const maxTimeOperandChars = 64 + +// asInstant reports the instant a value denotes, but only for spellings that +// name one unambiguously — RFC 3339 with an explicit offset or `Z`. +// +// This exists because the server canonicalizes every top-level DateTime value +// to RFC 3339 UTC before publishing (#402), so a payload reads `...T04:00:00Z` +// while a caller's filter constant may name the same instant as +// `...T06:00:00+02:00`. Comparing those as text is wrong in both directions: +// lexically the payload sorts *below* the constant, so `gte` misses a row that +// is chronologically equal. The server compares DateTime columns as instants +// for exactly this reason; this is the client-side twin of that rule. +// +// Deliberately narrow. A zone-less spelling ("2026-06-21 04:00:00") names an +// instant only relative to the column's declared timezone, which the server +// reads from the schema and a stream subscriber does not have. Guessing UTC +// would move the instant, so those fail to parse here and fall through to text +// comparison rather than being silently reinterpreted. +func asInstant(v any) (time.Time, bool) { + s, ok := v.(string) + if !ok || len(s) > maxTimeOperandChars { + return time.Time{}, false + } + // ClickHouse has no ',' decimal separator, but Go's RFC3339Nano accepts one + // per ISO 8601. Reject it so the client can't admit a spelling the server + // would refuse. + if strings.ContainsRune(s, ',') { + return time.Time{}, false + } + t, err := time.Parse(time.RFC3339Nano, s) + if err != nil { + return time.Time{}, false + } + return t, true +} + // evaluateIn checks whether actual is contained in the expected slice. // Reflection handles []any and typed slices (e.g., []string, []int) alike. func evaluateIn(actual, expected any) bool { @@ -681,6 +732,26 @@ func compareOrdered(actual, expected any) (int, bool) { } } } + // Timestamps compare chronologically, not lexically. If either side names + // an instant the other must too: ordering a canonicalized payload against a + // spelling that isn't a provable instant is meaningless, and text + // comparison there would admit rows the query path excludes. Fail closed, + // as the server's row filter does for a DateTime column. + aTime, aIsTime := asInstant(actual) + bTime, bIsTime := asInstant(expected) + if aIsTime || bIsTime { + if !aIsTime || !bIsTime { + return 0, false + } + switch { + case aTime.Before(bTime): + return -1, true + case aTime.After(bTime): + return 1, true + default: + return 0, true + } + } if aStr, ok := actual.(string); ok { if bStr, ok := expected.(string); ok { switch { diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go index c65bb863..533c14f2 100644 --- a/clients/go/stream_test.go +++ b/clients/go/stream_test.go @@ -652,3 +652,119 @@ func TestStream_ConfiguredHeadersReachTheStream(t *testing.T) { t.Fatal("stream request never arrived") } } + +// TestEvaluateFilter_TimestampsCompareAsInstants: the server canonicalizes +// every top-level DateTime value to RFC 3339 UTC before publishing (#402), so +// a payload and a caller's filter constant routinely spell the same instant +// differently. Comparing those as text disagrees with the server's row filter, +// which compares DateTime columns chronologically. +func TestEvaluateFilter_TimestampsCompareAsInstants(t *testing.T) { + // The canonicalized payload value, and the same instant in +02:00 — which + // sorts ABOVE it lexically ("06" > "04") while being chronologically equal. + const canonical = "2026-06-21T04:00:00Z" + const sameInstantOffset = "2026-06-21T06:00:00+02:00" + const oneSecondLater = "2026-06-21T06:00:01+02:00" + + tests := []struct { + name string + actual any + op string + expected any + want bool + }{ + {"equal across offsets", canonical, "eq", sameInstantOffset, true}, + {"neq is false across offsets", canonical, "neq", sameInstantOffset, false}, + {"gte holds at the same instant", canonical, "gte", sameInstantOffset, true}, + {"lte holds at the same instant", canonical, "lte", sameInstantOffset, true}, + {"gt is false at the same instant", canonical, "gt", sameInstantOffset, false}, + {"lt sees a later offset instant", canonical, "lt", oneSecondLater, true}, + {"gt is false against a later instant", canonical, "gt", oneSecondLater, false}, + {"in matches across offsets", canonical, "in", []any{"2020-01-01T00:00:00Z", sameInstantOffset}, true}, + + // Same-offset spellings must keep working exactly as before. + {"gt within UTC", "2026-06-21T04:00:01Z", "gt", canonical, true}, + {"lt within UTC", "2026-06-21T03:59:59Z", "lt", canonical, true}, + {"eq identical text", canonical, "eq", canonical, true}, + + // Sub-second precision survives the round trip. + {"fractional seconds order correctly", "2026-06-21T04:00:00.500Z", "gt", canonical, true}, + + // A zone-less constant names an instant only relative to the column's + // declared timezone, which a stream subscriber does not have. It must + // not be silently read as UTC — ordering fails closed. + {"zone-less constant fails closed on gt", canonical, "gt", "2026-06-21 03:00:00", false}, + {"zone-less constant fails closed on lt", canonical, "lt", "2026-06-21 05:00:00", false}, + + // A ',' fraction is ISO 8601 but not ClickHouse, so it is not an instant. + {"comma fraction is not an instant", canonical, "eq", "2026-06-21T04:00:00,000Z", false}, + + // Non-timestamp strings keep lexicographic ordering. + {"plain strings still order lexically", "banana", "gt", "apple", true}, + {"plain strings still compare equal", "apple", "eq", "apple", true}, + + // Numbers are untouched by any of this. + {"numbers still order numerically", 100.0, "gt", 9.0, true}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := evaluateFilter(tc.actual, tc.op, tc.expected, nil); got != tc.want { + t.Fatalf("evaluateFilter(%v, %q, %v) = %v, want %v", + tc.actual, tc.op, tc.expected, got, tc.want) + } + }) + } +} + +// TestEqualValues_NilOnlyEqualsNil: a column missing from the payload must not +// match the literal string "" through the fmt.Sprint fallback. +func TestEqualValues_NilOnlyEqualsNil(t *testing.T) { + tests := []struct { + name string + a, b any + want bool + }{ + {"nil equals nil", nil, nil, true}, + {"nil does not equal the string ", nil, "", false}, + {"the string does not equal nil", "", nil, false}, + {"nil does not equal empty string", nil, "", false}, + {"nil does not equal zero", nil, 0, false}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := equalValues(tc.a, tc.b); got != tc.want { + t.Fatalf("equalValues(%v, %v) = %v, want %v", tc.a, tc.b, got, tc.want) + } + }) + } +} + +// TestStream_FilterMatchesCanonicalizedPayload: the end-to-end shape of the +// same bug — a caller filters on a non-UTC spelling and the server delivers the +// canonicalized one. +func TestStream_FilterMatchesCanonicalizedPayload(t *testing.T) { + frame := `event: message +id: 2026-06-21T04:00:00Z +data: {"table_name":"clicks","received_timestamp":"2026-06-21T04:00:00Z","data":{"page":"/home","event_ts":"2026-06-21T04:00:00Z"}} + +` + srv := sseServer(t, []string{frame}) + + stream := streamClient(t, srv).From("clicks"). + SelectAll(). + Where("event_ts", OpGte, "2026-06-21T06:00:00+02:00"). + Stream(nil) + defer stream.Close() + + events := make(chan StreamEvent, 4) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { events <- e }}) + + select { + case e := <-events: + if e.Data["page"] != "/home" { + t.Fatalf("unexpected row: %v", e.Data) + } + case <-time.After(5 * time.Second): + t.Fatal("a row chronologically equal to the filter constant was withheld") + } +} diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md index 69487e16..e136c814 100644 --- a/docs/src/content/docs/sdk/go/streaming.md +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -160,6 +160,20 @@ stream := wh.From("clicks"). Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the `FilterOp` set `.Where()` takes everywhere (mapped to wire tokens `eq`/`neq`). `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively. `OpIn` accepts any Go slice type (e.g., `[]string`, `[]int`). +#### How values are compared + +The client-side evaluator mirrors the server's row-filter comparison rules rather than comparing everything as text: + +- **Timestamps compare chronologically.** Since the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing, a payload reads `2026-06-21T04:00:00Z` while your filter constant may name the same instant as `2026-06-21T06:00:00+02:00`. Comparing those as text is wrong in both directions — lexically the payload sorts *below* the constant, so `OpGte` would miss a row that is chronologically equal. Both sides are parsed as instants instead. +- **Only unambiguous spellings count as instants.** RFC 3339 with an explicit offset or `Z`. A zone-less spelling like `2026-06-21 04:00:00` names an instant only relative to the column's declared timezone, which the server reads from the schema and a stream subscriber does not have — guessing UTC would move the instant. Such a constant is not treated as a timestamp. +- **Ordering an instant against a non-instant fails closed.** If one side parses as a timestamp and the other does not, `OpGt`/`OpGte`/`OpLt`/`OpLte` withhold the row rather than falling back to text comparison, which could otherwise admit rows the query path excludes. The usual cause is a zone-less filter constant — give it an offset. +- **A missing column equals only `nil`.** A column absent from the payload does not match the string `""`. +- **Numbers compare numerically**, so `9 < 100` as you would expect rather than as text. + +:::caution[Integer precision above 2^53] +Event data decodes through `encoding/json` into `map[string]any`, so JSON numbers arrive as `float64`. An integer column beyond `Number.MAX_SAFE_INTEGER` (2^53) has already lost exactness before any filter runs — the server compares such columns in their exact storage domain, so a client-side filter on a very large `UInt64` can disagree with the server's verdict. Filter on a string or timestamp column instead when exactness at that magnitude matters. +::: + ## Live Queries Live queries combine a historical backfill (`.FetchUntyped`) with a real-time stream for seamless initial loads and updates. They are available only on `*QueryBuilder` (no `TableRef.LiveQuery` shortcut), matching the TypeScript SDK.