diff --git a/.claude/commands/cover.md b/.claude/commands/cover.md index 021def65..93b5a33e 100644 --- a/.claude/commands/cover.md +++ b/.claude/commands/cover.md @@ -1,6 +1,6 @@ --- description: Render coverage HTML for a suite and surface drops below threshold -argument-hint: [unit|integration|e2e|sdk|merge|all] (default: merge whatever exists) +argument-hint: [unit|integration|e2e|go-sdk|sdk|merge|all] (default: merge whatever exists) --- Generate the coverage report and surface anything below threshold from `.testcoverage.yml`. @@ -13,10 +13,11 @@ Behavior: - **unit**: `make test-unit` (gates per-suite + writes `tmp/coverage/unit/`) - **integration**: `make test-integration` (requires Docker) - **e2e**: `make test-e2e` (requires Docker; orchestrator + cover binary) +- **go-sdk**: `make test-go-sdk` (nested module `clients/go`; gates against `suites.go-sdk`, rendered separately and never merged into the Go total) - **ts-unit**: `make test-ts` (SDK unit tests + coverage + gate against `suites.ts-unit`) - **ts-e2e**: emitted as a side effect of `make test-e2e` (the orchestrator always passes `--coverage` to the e2e vitest run; informational only, no standalone gate) - **ts-total**: `make cov` (runs `cov report` — one consolidated Go + TS summary with per-suite HTML links + all gates; fails if *no* suite has data) -- **all**: `make test-all` (all four suites sequentially + `make cov`) +- **all**: `make test-all` (every suite sequentially + `make cov`) After the run completes: diff --git a/.github/dependabot.yml b/.github/dependabot.yml index e1c5d695..38d5c98c 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -1,8 +1,17 @@ version: 2 updates: # Go modules + # + # TWO directories, not one — the same trap as github-actions below: + # `directory: /` covers the root module only, and Dependabot does not + # descend into nested modules. clients/go is its own module, so its + # go.mod needs its own entry here. That module is stdlib-only today + # (no `require` block, no go.sum), so this catches the first dependency + # it takes on rather than closing a gap that already exists. - package-ecosystem: gomod - directory: / + directories: + - / + - /clients/go schedule: interval: weekly day: monday diff --git a/.github/labeler.yml b/.github/labeler.yml index 4df42c01..d164e157 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -43,7 +43,9 @@ - changed-files: - any-glob-to-any-file: - "clients/ts/**" + - "clients/go/**" - "tests/e2e/sdk/**" + - "tests/conformance/**" "area/docs": - changed-files: @@ -83,8 +85,8 @@ dependencies: - changed-files: - any-glob-to-any-file: - - "go.mod" - - "go.sum" + - "**/go.mod" + - "**/go.sum" - "**/package.json" - "**/pnpm-lock.yaml" - "**/package-lock.json" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3c0b05f9..deebc508 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -204,8 +204,14 @@ jobs: uses: ./.github/actions/setup-env with: go-cache-suffix: "-unit" - - name: Run Go unit tests + SDK vitest tests - run: make test-unit test-ts COV_DEFER=1 + - name: Run Go unit tests + SDK vitest + Go SDK tests + run: make test-unit test-ts test-go-sdk test-conformance-ts COV_DEFER=1 + # This one fragment carries THREE suites' data: unit covdata, ts-unit + # (vitest/istanbul) and go-sdk covdata from the nested clients/go + # module — every target above ran under COV_DEFER, so the `coverage` + # job renders and gates all three. No per-suite paths here on purpose: + # uploading the whole tmp/coverage tree means a new suite collected by + # this job needs no workflow change. - name: Upload coverage fragment uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: diff --git a/.testcoverage.yml b/.testcoverage.yml index 669d65be..ee02af8d 100644 --- a/.testcoverage.yml +++ b/.testcoverage.yml @@ -12,7 +12,9 @@ # together, so `threshold.total` below applies to the *project-wide* # coverage number — unit alone covers only `./internal/...` packages # exercised by `*_test.go` files (below total threshold); merged -# coverage adds integration- and e2e-only paths and clears it. +# coverage adds integration- and e2e-only paths and clears it. The +# nested-module suites (go-sdk) are gated separately and never merged +# in — see suites.go-sdk. profile: tmp/coverage/total/coverage.txt local-prefix: github.com/Wave-RF/WaveHouse @@ -30,6 +32,18 @@ suites: unit: 80 integration: 20 e2e: 60 + # Go SDK (clients/go) — a NESTED module, so its coverage is rendered and + # gated on its own and is deliberately NOT part of the merged Go total + # above. Nothing from clients/go can leak into that total: the root + # module can't see a nested one (`go list ./...` at the repo root never + # yields clients/go), so the unit/integration/e2e `-coverpkg=./...` never + # reaches these files — which is also why there is no `^clients/go/` + # entry under exclude.paths; there is nothing to exclude. + # Measured 82.7% when this floor was set (SDK package 88.6%, + # cmd/wavehouse-codegen 55.8%). 75 leaves the same order of headroom + # unit's 80 leaves under its real ~91%. Raise it as the codegen + # command's tests fill in. + go-sdk: 75 # TypeScript SDK suites — see scripts/cov for the merge / render logic. # vitest gates via --coverage.thresholds.statements; the merged ts-total # is gated by `cov ts-merge` against the value below. Tune ts-total diff --git a/AGENTS.md b/AGENTS.md index 9967534a..c06b32c5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -60,7 +60,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 11. **Hasura-style access control: fail-closed (security)** — `policy.IsAdmin` (role == `admin_role`, **exact case-sensitive**, default `"admin"`) is the single admin check, shared by `Evaluate`/`ResolveRole`/`Validate`/the `/v1/ops` gate/`RoleAllowed`. Empty/absent role matches nothing (no `"*"` wildcard); `Validate` rejects empty role keys; a `nil` policy (deleted) denies **everyone incl. admin** via a role — a total lockout for token-based callers, so bootstrap from the policy file, never an implicit admin grant (**exception:** the operator key's `auth.IsOperator` bit passes the `/v1/ops` gate even under a `nil` policy — a deliberate break-glass restore over HTTP, see #7). `default_role` is the one sanctioned roleless exception (`ResolveRole` maps empty → it pre-eval); `default_role == admin_role` is permitted but dev-only and loudly warned (`policy.DefaultRoleGrantsAdmin`). Preserve when touching `internal/policy` (policy twin of #13; see #159). Detail: architecture.md § `policy/`. 12. **Structured queries: column authz fail-closed (security)** — `POST /v1/query?table={table}`: typed AST validated against schema, permission-enforced, timestamp-bucketed for cache, `DefaultMaxRows` (10,000) cap. Every column reference — projection, aggregation args, `filters`, `group_by`, `order_by`, `time_range` — is authorized inside `query.Build` (the single chokepoint that enumerates them all), so no clause can skip the role's `allow_columns`/`deny_columns` check (#223). A `select_all` read by a *column-restricted* role expands to its allowed columns via `policy.AllowedProjection`, never a bare `SELECT *`; *unrestricted*/admin roles keep `SELECT *` (`policy.RestrictsColumns` decides). Omitting `columns` selects nothing (`ErrEmptyProjection` → `200 []`); `["*"]` is the literal column `*` (schema-gated, not a wildcard); a table-granted role with no readable columns fails closed (`ErrNoReadableColumns` → `403`). Structured and live-stream (`stream.filterColumns`) reads share the one per-column decision `policy.IsColumnAllowed`, so column visibility can't drift. Row visibility has the same one-source guarantee (#319): `Evaluate` resolves a role's row-`filter` once (`resolvePredicates`), and both surfaces consume that single resolution — the query path renders it to SQL (`predicatesToSQL`), the stream evaluates it in memory per subscriber (`ResolvedPermissions.RowVisible`, whose type-aware comparison fails closed on anything it can't prove about the ingested payload — `policy.ColumnSpec`, with `DateTime`/`DateTime64` operands compared as instants through the ingest grammar (`discovery.Column.TimeParser`) and claim constants rendered canonically and digit-exact by the one shared rule `policy.CanonicalScalar` (#457 — which also refuses a float64 at/past 2^53 rather than match a neighboring ID, and whose ok=false — an absent claim, a structured value, no canonical form — makes the predicate match no rows on BOTH surfaces: `1 = 0` in SQL, every row withheld in memory); numeric comparison runs in the column's STORAGE domain (`policy.NumericSpec`, classified by `discovery.NumericStorageOf` — Float width rounding, Decimal scale truncation, integer exactness, both operands narrowed as ClickHouse narrows stored value and bound constant, out-of-range operands refused rather than modeled; the `tests/integration` differential oracle holds in-range verdicts equal to a live ClickHouse's and the never-admit-where-SQL-hides direction for the refused out-of-range ones); an event whose insert later fails into the DLQ is the one residual payload-vs-stored asymmetry, documented in the access-control enforcement caution) — so row visibility can't drift either. Preserve when touching `internal/query` or the structured-query handler. Detail: architecture.md § `query/`. 13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). Detail: architecture.md § `pipes/`. -14. **TypeScript SDK** — `@wavehouse/sdk`: typed query builder, real-time SSE over `fetch`, live queries (incrementable/decomposable/poll aggregation), codegen CLI. Exactly one runtime dependency — `eventsource-parser` (SSE framing, itself dependency-free); adding a second needs the same scrutiny the first got. The canonical client (see §SDK Sync). +14. **Client SDKs** — TypeScript (`@wavehouse/sdk` in `clients/ts/`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`, package `wavehouse`, in `clients/go/`) are both canonical, officially supported clients with full API-tree parity. Each ships a typed query builder, real-time SSE over header-authenticated HTTP, live queries (incrementable/decomposable/poll aggregation), and a codegen CLI. TypeScript has exactly one runtime dependency — `eventsource-parser` (SSE framing, itself dependency-free); adding a second needs the same scrutiny the first got. Go has zero third-party runtime dependencies (stdlib-only, hand-rolled SSE framing). See §SDK Sync. 15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. The OTLP endpoint/TLS/custom-CA/mTLS/headers are delegated to the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` env vars — `InitProvider` passes **no** endpoint/header options. Known gap, intentionally not patched in WaveHouse app code: the pinned gRPC logs exporter (`otlploggrpc` v0.19/v0.20) ignores the env TLS-cert vars, so a custom/private CA and mutual TLS apply to traces/metrics but **not** the logs signal (public-CA/system-roots TLS and plaintext still work for logs) — upstream bug open-telemetry/opentelemetry-go#6661. A malformed `OTEL_EXPORTER_OTLP_HEADERS` is logged and skipped by the SDK (fail-soft), not fatal. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`. 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. 17. **Non-fatal boot** — schema-discovery failure on boot is non-fatal: `cmd/wavehouse` records an `api.BootState`, binds `:8080`, serves 503 on `/livez`/`/readyz` with the diagnostic, and retries via `SchemaRegistry.RetryRefresh` (backoff 2s → 60s). Bounds supervisor restart loops. @@ -125,7 +125,7 @@ Tooling notes (the non-obvious bits `make help` won't tell you): - **Policy helpers**: Use `policy.NewMemoryStore(p)` for in-memory policy testing without NATS. - **Pipes helpers**: Use `pipes.NewMemoryStore(queries...)` for in-memory pipes testing without NATS. - **Response assertions**: Use `testutil.AssertJSONResponse(t, rec, status, expected)` and `testutil.AssertJSONContains(t, rec, status, substring)`. -- **Coverage target**: 80% project-wide (CI enforces `threshold.total` in `.testcoverage.yml` against the merged unit + integration + e2e profile). Per-suite minima also enforced: unit 80%, integration 20%, e2e 60%, sdk 50%. Aim for 80%+ on new code. Coverage is published as a README badge (Go merged-total) and as PR comments via GitHub Code Quality — see `.github/workflows/README.md` "Coverage publishing"; the gate is unchanged. +- **Coverage target**: 80% project-wide (CI enforces `threshold.total` in `.testcoverage.yml` against the merged unit + integration + e2e profile). Per-suite minima also enforced: unit 80%, integration 20%, e2e 60%, go-sdk 75%, ts SDK 50%. The Go SDK (`clients/go`) is a nested module — invisible to the root module's `-coverpkg=./...`, so it is gated on its own `suites.go-sdk` floor and never merged into the project-wide total. Aim for 80%+ on new code. Coverage is published as a README badge (Go merged-total) and as PR comments via GitHub Code Quality — see `.github/workflows/README.md` "Coverage publishing"; the gate is unchanged. - **Every new function should have corresponding test cases.** Run `make lint` and `make test` before considering work complete. - **E2E tests via SDK**: The TypeScript SDK is the primary E2E test harness. Tests in `tests/e2e/sdk/` exercise the full pipeline (ingest → ClickHouse → query) and simultaneously validate backend behavior and SDK correctness. Use `make test-e2e` to run. Add new E2E scenarios as `tests/e2e/sdk/*.test.ts` files using helpers from `tests/e2e/sdk/helpers.ts`. - **Per-suite table isolation**: Each e2e test file owns its own ClickHouse tables — `clicks_` / `events_` / `users_`, generated from `tests/e2e/sdk/tables.ts` and created by `setup.ts`. A new test file must (1) add its suite name to `SUITES` in `tables.ts` and (2) get its names via `const T = suiteTables("")`, then reference `T.clicks` etc. — never a bare `clicks`. This makes cross-file *data* contamination structurally impossible. Files still run **sequentially** (`vitest.config.ts` `maxWorkers: 1`): running them in parallel is blocked by shared *global policy* state (several files read-modify-write the single policy document; `streaming.test.ts` flips the global `default_role`), so policy-mutating tests snapshot the full policy and restore it. Dropping `maxWorkers: 1` is a deferred follow-up tracked in #214 (per-table policy storage; see `docs/src/content/docs/ingest-pipeline.md` § Deferred). @@ -294,7 +294,7 @@ Then run the reviewers relevant to the PR's diff (the same set from `scripts/pre Documentation *prose* — accuracy against the code, runnable examples, clarity, completeness — **and code↔docs sync** (code that changed but whose docs didn't) are reviewed by the **`docs-reviewer`** subagent, not the code-focused `pre-push-reviewer`. The canonical rubric is `.github/prompts/docs-review.md`. It complements the deterministic prose tools — misspell, markdownlint, starlight-links-validator — reviewing only what they can't, and it never edits docs or posts PR comments. -**Scope** is the canonical docs-prose set from `scripts/docs-prose.sh` — a *denylist*: every tracked `.md`/`.mdx` EXCEPT `.claude/**`, `.github/**`, `CHANGELOG.md`, `AGENTS.md`, `CLAUDE.md`, `*.draft.md`/`*.old.md`, `PERF-CLAIMS-REVIEW.md`, `docs/posthog-setup-report.md`. So it covers the Starlight site under `docs/src/content/` **and** the governance docs (`README.md`, the SDK readme `clients/ts/README.md`, `CONTRIBUTING.md`, `SECURITY.md`, `CODE_OF_CONDUCT.md`, `SUPPORT.md`) — new docs are picked up automatically. `CODE_OF_CONDUCT.md`/`SUPPORT.md` are deep-reviewed only on change or material suspicion. +**Scope** is the canonical docs-prose set from `scripts/docs-prose.sh` — a *denylist*: every tracked `.md`/`.mdx` EXCEPT `.claude/**`, `.github/**`, `CHANGELOG.md`, `AGENTS.md`, `CLAUDE.md`, `*.draft.md`/`*.old.md`, `PERF-CLAIMS-REVIEW.md`, `docs/posthog-setup-report.md`. So it covers the Starlight site under `docs/src/content/` **and** the governance docs (`README.md`, the SDK readmes `clients/ts/README.md` / `clients/go/README.md`, `CONTRIBUTING.md`, `SECURITY.md`, `CODE_OF_CONDUCT.md`, `SUPPORT.md`) — new docs are picked up automatically. `CODE_OF_CONDUCT.md`/`SUPPORT.md` are deep-reviewed only on change or material suspicion. **It is a hard pre-push gate**, run in parallel with the other pre-push reviewers (see §Pre-push self-review). Invoked with the **default (branch) scope** it emits a `VERDICT:` line; on `ship_it` the `review-marker.sh` SubagentStop hook writes `tmp/docs-reviewer-passed-`, which the push gate requires — unconditionally, on every PR-branch push (even code-only ones). Run it via **`/docs-review`**; with **no arg** that's the gating review (branch scope), while an explicit **path/glob** or **`all`** is **advisory** (no `VERDICT:`, no marker) for ad-hoc audits. The whole dev team runs Claude Code and this command is tracked in-repo, so everyone runs it themselves; there is intentionally **no PR/cloud path** for docs review. @@ -360,22 +360,22 @@ Diagrams render inside the Starlight content column (~46–58rem wide) as build- ## SDK Sync -The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) is the canonical client and ships from this repo. When backend changes alter the public API surface, the SDK needs corresponding updates. The `pre-commit` git hook flags likely misses informationally; consult this table when deciding what to update. +The TypeScript SDK (`@wavehouse/sdk` in `clients/ts/`) and Go SDK (`github.com/Wave-RF/WaveHouse/clients/go`, in `clients/go/`) are both canonical, officially supported clients. Both ship from this repo with full API-tree parity. When backend changes alter the public API surface, both SDKs need corresponding updates. The `pre-commit` git hook flags likely misses informationally; consult this table when deciding what to update. | Backend change | SDK considerations | | -------------- | ------------------ | -| New user-facing API endpoint | Add a typed client method (in `clients/ts/src/client.ts` or the relevant subsystem file: `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`, etc.); update the matching SDK doc page under `docs/src/content/docs/sdk/` (`queries`, `streaming`, `pipes`, `admin`, or `reference` by topic — plus the API tree in `reference.md`) | -| Change to JWT auth / role extraction | Update auth handling in `clients/ts/src/http.ts` and types in `clients/ts/src/client.ts` | -| Change to `EventMessage` / ingest event format | Update payload types in `clients/ts/src/` (some are codegen-regenerated — re-run the SDK codegen CLI) | -| New / changed structured query AST | Update `clients/ts/src/query-builder.ts` types + builder methods | -| Change to live-query aggregation classification | Update live-query helpers in `clients/ts/src/stream/` | -| Named pipes API change | Update `clients/ts/src/pipes.ts` | -| Policy / access-control change | Update `clients/ts/src/policy.ts` | -| ClickHouse schema-driven type changes | Re-run the SDK codegen CLI; commit regenerated types | +| New user-facing API endpoint | Add a typed client method in **both** SDKs (TS: `clients/ts/src/` — `client.ts`, `query-builder.ts`, `pipes.ts`, `policy.ts`, `stream/`; Go: `clients/go/` — corresponding file). Update doc pages under `docs/src/content/docs/sdk/` for both `ts/` and `go/`. Add a wire case to `clients/go/testdata/wire_cases.json` with dispatch in both conformance runners. | +| Change to JWT auth / role extraction | TS: `clients/ts/src/http.ts` + `client.ts`. Go: `clients/go/http.go` + `wavehouse.go`. | +| Change to `EventMessage` / ingest event format | Update payload types in both SDKs (some are codegen-regenerated — re-run both codegen CLIs). | +| New / changed structured query AST | TS: `clients/ts/src/query-builder.ts`. Go: `clients/go/query_builder.go` + `types.go`. | +| Change to live-query aggregation classification | TS: `clients/ts/src/stream/`. Go: `clients/go/live_query.go`. | +| Named pipes API change | TS: `clients/ts/src/pipes.ts`. Go: `clients/go/pipes.go`. | +| Policy / access-control change | TS: `clients/ts/src/policy.ts`. Go: `clients/go/policy.go`. | +| ClickHouse schema-driven type changes | Re-run both SDK codegen CLIs; commit regenerated types. | Internal-only backend changes (middleware refactors, observability internals, dedup implementation, sweeper logic, NATS plumbing) generally don't need SDK updates. Use judgement — table above is the source of truth; nothing automated nudges you. -**The decision test**: would a `@wavehouse/sdk` user's *code* need to change to take advantage of (or be compatible with) this change? If yes, SDK update needed. If no (purely internal optimization), no. +**The decision test**: would a user's *code* need to change to take advantage of (or be compatible with) this change? If yes, both SDKs need updates. If no (purely internal optimization), no. ## Common Tasks @@ -416,6 +416,14 @@ Internal-only backend changes (middleware refactors, observability internals, de ```text cmd/ → Binary entry points (thin — just wiring) +clients/ts/ → TypeScript SDK (@wavehouse/sdk) +clients/go/ → Go SDK (github.com/Wave-RF/WaveHouse/clients/go) + wavehouse.go, http.go, errors.go, types.go → Client core (constructor, transport, errors, shared types) + query_builder.go, table.go → Structured query builder + per-table typed client + stream.go, live_query.go → SSE streaming + live queries + pipes.go, policy.go, schema.go, dlq.go, sys.go → Subsystem clients (pipes, policy, schema, DLQ, health) + cmd/wavehouse-codegen/main.go → Codegen CLI + testdata/wire_cases.json → Wire-format conformance fixtures internal/api/ → HTTP layer (handlers, router, middleware, schema/DLQ/policy/pipes endpoints) internal/auth/ → JWT/JWKS authentication middleware (HMAC or JWKS, role extraction from claims) internal/cache/ → Caching (interface + L1/L2/tiered implementations) diff --git a/CHANGELOG.md b/CHANGELOG.md index 713b72c7..849b018f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,11 +10,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ### Added +- **Go SDK — official Go client with full API-tree parity against the TypeScript SDK** (`clients/go/` (new nested Go module), `tests/conformance/conformance_ts.mjs` (new), `docs/src/content/docs/sdk/go/{index,queries,streaming,pipes,admin,reference}.md` (new), `docs/src/content/docs/sdk/index.mdx`, `docs/src/config/sidebar.ts`, `Makefile`, `.github/workflows/ci.yml`, `AGENTS.md`, `CONTRIBUTING.md`, `README.md`, `SUPPORT.md`, `docs/src/content/docs/{development,architecture,getting-started,why-wavehouse,404,api,claude-code,deployment}.md`, `docs/src/content/docs/{access-control,pipes,reverse-proxy}.mdx`, `docs/src/content/docs/sdk/{queries,streaming,pipes,admin,reference}.md`, `docs/src/content/docs/index.mdx`): install with `go get github.com/Wave-RF/WaveHouse/clients/go` (package `wavehouse`; a *nested* Go module with its own `go.mod`, invisible to root `go list` — hence the dedicated make targets). Zero third-party runtime dependencies, `context.Context`-first, generics for typed rows (`FetchTyped[Row]`, `Fetch[Row]`, `SQL[Row]`), an immutable chainable query builder with keyset pagination, real-time SSE streaming with reconnect/backfill (`Stream`, `LiveQuery`), admin namespaces (Schema/Policy/Pipes/DLQ/Sys), per-client `Headers` applied to REST and SSE alike (the Go analog of the TypeScript SDK's `options.headers`, and how an operator sends `X-Operator-Key`), and a `wavehouse-codegen` CLI that generates row structs from `/v1/ops/schema`. Wire-format parity is enforced by a shared fixture (`clients/go/testdata/wire_cases.json`) replayed by two conformance runners — Go (`conformance_test.go`, in `make test-go-sdk`) and TS (`make test-conformance-ts`) — both run by CI's unit job and local `make ci`. New make targets: `test-go-sdk` (with `-race`), `test-go-sdk-e2e` (live server, `WAVEHOUSE_URL`/`WAVEHOUSE_AUTH`), `test-conformance-ts`, `lint-go-sdk`, `verify-go-sdk`; the `test`/`lint`/`fix` aggregates cover the nested module. Docs ship as a six-page tree under `/sdk/go/`. Releases ride the tag-driven scheme already in place: `make release-sdk-go` cuts a `clients/go/vX.Y.Z` tag (`scripts/release.sh`), which the Go module proxy serves directly — no publish workflow needed. - **"Was this page helpful?" feedback widget on every docs page** (`docs/src/components/PageFeedback.astro` (new), `docs/src/components/Footer.astro`): a thumbs-up / thumbs-down vote below the page content, captured to PostHog as `docs_feedback` with `{ helpful, page }`. It renders from `Footer.astro`'s sidebar branch — the same indirection the Cloud CTA uses — rather than a per-page import or frontmatter flag, so every content page gets it automatically, including ones not written yet; it sits *below* the Cloud CTA on the pages that carry one, and splash pages (the homepage and 404) take the other footer branch and never render it. One vote per page per visitor: the choice is remembered in `localStorage` keyed by pathname, and a revisit renders the thanks message instead of re-prompting (storage is a nicety, not the record — a browser with storage disabled still votes). - **Settings-directory validation — `wavehouse validate [dir]`** (`internal/settings/` (new: `settings.go`, `validate.go`, `decode.go`, `finding.go`, + tests), `cmd/wavehouse/validate.go` (new, + tests), `cmd/wavehouse/main.go`): first piece of the file-based control plane (settings live in a directory of JSON documents — `roles.json`, `policies.json`, `pipes.json`, `config.json` — that a running instance will hot-reload; this change is validation-only — boot loading and reload wiring land separately). `settings.Validate(dir)` is the single gate every consumer of the directory runs: deliberately pure (no network, no ClickHouse — table/column existence stays with schema discovery, per Bring-Your-Own-Schema), and it collects **all** findings in one pass instead of failing on the first. Checks, layered: the directory holds exactly the four files (a missing file is an error — an empty document is `{}`, so absence always means deletion or a wrong path; any unexpected entry — file or directory — is an error so a typoed `polices.json` or a stray backup can't be silently ignored; dot-prefixed entries are the one carve-out, since erroring on vim swap files or the `..data` machinery Kubernetes ConfigMap mounts publish through would break hand editing and the cloud fan-out's mount pattern alike); strict JSON syntax (unknown fields rejected — the JSON form of the retired-config-key trap; empty/truncated files rejected, never read as an empty document; a leading UTF-8 byte order mark named as such instead of surfacing as a cryptic invalid-character error; a directory, unreadable file, or non-regular file (a FIFO would hang the read forever waiting for a writer; a stat gate rejects it — following symlinks, so Kubernetes ConfigMap mounts' symlink layout still passes) squatting on a settings filename named as the one real problem, not double-reported as "missing"; a top-level `null` rejected — the one well-formed document that decodes into a zero value without error, so it would silently read as "no settings"; trailing content rejected; duplicated object keys detected by a token-level pass, since `encoding/json` silently keeps the last copy); per-file shape rules (role names non-empty/unique, pipe names/SQL/param types, `config.json` bounds mirroring boot-config validation — its sections are the *tenant-owned* behavioral tunables (dedupe id_field/require_id plus per-table overrides under `dedupe.tables` — each entry overrides only the fields it names, resolving table → global → compiled default per field, so the effective id_field can never be empty — an explicit empty, whitespace-only, or whitespace-padded id_field is rejected at both levels, since an exact-match JSON key lookup would silently miss every row ([#222](https://github.com/Wave-RF/WaveHouse/issues/222)'s shape, unblocked by the file design since table names are runtime-resolved like policy grants); query default_max_rows, schema refresh_interval, CORS origins); platform-owned knobs like the SSE keepalives deliberately stay boot config); and cross-file referential integrity (every role a policy grant, `default_role`/`admin_role`, or pipe allowlist references must be declared in `roles.json`; an empty role string in a grant or allowlist is named as such — it matches no request and authorizes nobody). Warnings don't invalidate: a grant scoping the admin role (an unconditional bypass — dead config), `default_role` = admin, and a `default` on a required pipe parameter are flagged but legal. An empty `policies.json` means no policy — fail closed, matching deleted-policy semantics — and draws a warning naming the total lockout, so it announces itself at validation time instead of one 403 at a time. The CLI (`cmd/wavehouse/validate.go`, following the `health` subcommand pattern) takes the directory as an argument or from `WH_SETTINGS_DIR`, prints findings, and exits 0/1/2 (valid/invalid/usage) so CI and operators can gate config changes before they reach a running instance. The dispatch in `main.go` also grows `help` and `version` subcommands, and an unknown command is now a usage error instead of silently falling through and starting the server (`wavehouse validat` booting a listener is not a typo anyone wants); each subcommand parses its arguments with a stdlib `flag.FlagSet`, so `wavehouse -h` prints command-specific help and a stray flag or argument is a usage error rather than being silently swallowed. `WH_SETTINGS_DIR` has a single authority: `config.EnvSettingsDir`, with a reflection test pinning the `settings.dir` struct tag to it. The directory's location joins boot config as `settings.dir` (`WH_SETTINGS_DIR`; `internal/config/config.go`, `config.yaml`, `docs/src/content/docs/configuration.mdx`) — boot-tier by necessity, since it's the pointer the reload machinery follows; no default, same silent-misconfiguration reasoning as `policy.file_path`. - **Docs-site analytics for search, code copies, 404s, docs section, and live-demo connectivity** (`docs/src/components/DocsTracking.astro` (new), `docs/src/components/{PostHog,Footer,LiveDemo}.astro`): the site tracked its own CTAs but nothing a reader did on the way to one, so the questions that decide what to write next — what people search for and *don't* find, which snippets get copied, which dead links keep getting followed — had no data behind them. `docs_search` fires a second after the query settles rather than once per keystroke, carrying `query` and `result_count` read off Pagefind's own results message (the rendered list is capped at its page size, so counting the DOM would under-report); `result_count: 0` is the event worth having. `code_copied` (`page`, `language`) watches Expressive Code's copy buttons from the document rather than re-binding every code block on every navigation — the hero's install chip is not an EC block and keeps its own `hero_install_copied`. `docs_404` (`path`, `referrer`) turns broken inbound links into a list instead of a hunch. A `doc_section` property (the first path segment, `home` for `/`) puts every event in a docs area without each tracker carrying its own copy; it's stamped at capture time by a `before_send` hook in `posthog.init()` rather than `register()`, because a queued `register()` replays only after init has already captured the first hard-load `$pageview` — which would then carry the previous visit's persisted value — and `history_change` navigations update the URL before capture fires, so reading `location` in the hook is always current. `live_demo_connected` fires once per mount when the hero's SSE feed comes up rather than on its first row — named for what it measures (the demo backend answered), since a quiet minute on the repo is not a disengaged reader. The three site-wide trackers share one new `DocsTracking.astro` rendered from the footer (like `MermaidZoom` / `ScrollHints`) and delegate from `document`, since Pagefind, Expressive Code, and the 404 route all own their own markup — some of it created after page load. +### Fixed + +- **Go SDK client-side stream filters compare timestamps as instants, not as text** (`clients/go/stream.go`, `clients/go/stream_test.go`, `docs/src/content/docs/sdk/go/streaming.md`): the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing (#402), so a payload reads `2026-06-21T04:00:00Z` while a caller's filter constant may name the same instant as `2026-06-21T06:00:00+02:00`. Compared as text those disagree in both directions — lexically the payload sorts *below* the constant, so `OpGte` withheld a row that was chronologically equal. Both sides now parse as instants, mirroring the server's row-filter rule for DateTime columns. Only unambiguous spellings count (RFC 3339 with an explicit offset or `Z`): a zone-less constant names an instant only relative to the column's declared timezone, which a stream subscriber doesn't have, so reading it as UTC would move the instant. Ordering an instant against a non-instant now fails closed rather than falling back to text comparison. Also fixes a missing column matching the literal string `""` through the equality fallback. The TypeScript SDK's `matchesFilters` has the same text-comparison behavior and needs the same change for parity. + +- **TypeScript SDK documentation corrections found while writing the Go SDK's parity docs** (`docs/src/content/docs/sdk/{reference,queries,streaming}.md`): the error-code table described `401` as "missing or invalid JWT" when a *missing* token is actually evaluated as `default_role` — succeeding or denied with `403`, never `401` (`internal/auth/auth.go`, `internal/api/errors.go`) — and only a present-but-invalid or expired token yields `401`. `.aggregate()` was documented as accepting a "custom fn", but the server enforces an allowlist (`internal/query/builder.go`); the allowed set is now listed. Live queries gained a caution for a real footgun: the backfill dedup boundary comes from the fetched rows' `received_timestamp`, so a `.select(...)` projection omitting that column silently disables dedup and delivers overlap-window events twice. No SDK code changed — these were pre-existing gaps between the TS docs and the server's behavior. + ## [0.1.0] - 2026-08-19 The first public release. Everything below shipped in it — the sections are grouped the way Keep a Changelog asks for, but since there is no previous release to compare against, a reader upgrading from nothing can treat the whole file as "Added". The date is the intended cut date; correct it if tagging slips, and move anything merged in between up from `## Unreleased`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a1d656b8..015b72b4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -39,13 +39,14 @@ Open a [feature request issue](https://github.com/Wave-RF/WaveHouse/issues/new?t The pre-push hook (installed by `make tools`) blocks a push until the tree has been validated locally: a code change needs `make ci`, a docs/prose-only change needs only `make verify` (the same split CI makes). `make lint` / `make test` / `make build` are fast inner-loop subsets. -2. Write tests for new functionality. Unit tests go alongside the code in `internal/`. Integration tests go in `tests/` with the `//go:build integration` tag. +2. Write tests for new functionality. Unit tests go alongside the code in `internal/`; SDK tests live in `clients/ts/src/` and `clients/go/`. Integration tests go in `tests/` with the `//go:build integration` tag. 3. Update documentation if your change affects: - API endpoints → update `docs/src/content/docs/api.md` - Configuration options → update `docs/src/content/docs/configuration.mdx` - Deployment → update `docs/src/content/docs/deployment.md` - Architecture → update `docs/src/content/docs/architecture.md` + - Client SDK surface → update **both** SDKs (`clients/ts/src/`, `clients/go/`), their doc trees (`docs/src/content/docs/sdk/` and `.../sdk/go/`), and the shared wire fixture `clients/go/testdata/wire_cases.json`; see AGENTS.md §SDK Sync 4. Follow the commit message format (see below). @@ -89,7 +90,7 @@ test(cache): add tiered cache stampede test ## Code Style -- **Formatting**: Code must be formatted with `gofumpt` (a strict superset of `gofmt`). `make fmt` checks it (CI runs the same target); `make fix` applies it. +- **Formatting**: Code must be formatted with `gofumpt` (a strict superset of `gofmt`). `make fmt` checks the root module; the nested `clients/go` module is checked by `make verify` (its `verify-go-sdk` leaf, which the pre-commit hook and CI run). `make fix` applies gofumpt to both. - **Linting**: All lint checks in `.golangci.yml` must pass (see `make lint`). - **Naming**: Follow [Go naming conventions](https://go.dev/doc/effective_go#names). - **Interfaces**: Define interfaces where they are consumed, not where they are implemented. diff --git a/Makefile b/Makefile index 3920763c..a4696334 100644 --- a/Makefile +++ b/Makefile @@ -199,6 +199,9 @@ ACTIONLINT := $(LOCAL_BIN)/actionlint-$(ACTIONLINT_VERSION) COV_UNIT := tmp/coverage/unit COV_INT := tmp/coverage/integration COV_E2E := tmp/coverage/e2e +# go-sdk is the nested module at clients/go — same layout, own gate, but +# deliberately outside COV_TOTAL (see test-go-sdk below). +COV_GOSDK := tmp/coverage/go-sdk COV_TOTAL := tmp/coverage/total # --- Coverage Thresholds ------------------------------------------------------ @@ -364,12 +367,16 @@ fmt-ts: pnpm-install $(call run,Biome (format),$(PNPM) -s -w run format,run make fix to apply formatting) .PHONY: lint -lint: lint-go lint-ts lint-md lint-prose ## Lint across Go (golangci-lint) + TS/JSON (Biome) + Markdown (markdownlint) + docs prose (misspell). Run `make fix` to apply --fix. +lint: lint-go lint-go-sdk lint-ts lint-md lint-prose ## Lint across Go (root + clients/go golangci-lint) + TS/JSON (Biome) + Markdown (markdownlint) + docs prose (misspell). Run `make fix` to apply --fix. .PHONY: lint-go lint-go: $(GOLANGCI_LINT) go-mod-download $(call run,golangci-lint,$(GOLANGCI_LINT) run ./... --allow-parallel-runners,run make fix to auto-fix what is fixable) +.PHONY: lint-go-sdk +lint-go-sdk: $(GOLANGCI_LINT) + $(call run,golangci-lint (Go SDK),cd clients/go && $(GOLANGCI_LINT) run ./... --allow-parallel-runners,) + .PHONY: lint-ts lint-ts: pnpm-install $(call run,Biome (lint + format + imports),$(PNPM) -s -w run check,run make fix to auto-fix what is fixable) @@ -451,6 +458,12 @@ endif tidy: ## Verify go.mod/go.sum are tidy (run `make fix` to apply) $(call run,go.mod tidy,go mod tidy -diff,run make fix to tidy go.mod and go.sum) +# verify-go-sdk: nested module at clients/go/ — invisible to root go list. +.PHONY: verify-go-sdk +verify-go-sdk: ## Static checks for the Go SDK (clients/go, a nested module) — go vet + gofumpt + $(call run,go vet (Go SDK),cd clients/go && go vet ./...,) + $(call run,gofumpt (Go SDK),$(GOFUMPT) -l clients/go | (! grep .),run make fix to apply formatting) + # fix: apply auto-fixes everywhere, fanned out into three tracks that touch # disjoint files — Go (.go + go.mod/sum), TS/JS/JSON (Biome), Markdown — so they # run in parallel safely. Two of the three are themselves serial chains, because @@ -480,6 +493,8 @@ fix-go: $(GOLANGCI_LINT) @$(GOFUMPT) -w $(GO_DIRS) @$(GOIMPORTS) -w $(GO_DIRS) @$(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners + @echo "$(CYAN)==> Applying Go auto-fixes (Go SDK — nested module, outside GO_DIRS)...$(RESET)" + @$(GOFUMPT) -w clients/go && $(GOIMPORTS) -w clients/go && cd clients/go && go mod tidy && $(GOLANGCI_LINT) run --fix ./... --allow-parallel-runners .PHONY: fix-ts fix-ts: pnpm-install @@ -525,7 +540,9 @@ fix-prose: $(MISSPELL) # slowest tool, not the slowest *group* (e.g. golangci no longer drags Biome + # markdownlint along behind it). # -# Leaves (14): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck on the Go +# Leaves (16): tidy, fmt-go (gofumpt), lint-go (golangci), vulncheck, +# lint-go-sdk (golangci) + verify-go-sdk (go vet + gofumpt), both on the +# nested clients/go module, on the Go # side; lint-ts (biome check) + lint-md (markdownlint) + lint-prose (misspell, # docs spelling) + test-md-rules (node --test over the WH001/WH002 fixtures) # for JS/TS + Markdown + prose; lint-sh (shellcheck), lint-gha (actionlint), @@ -543,7 +560,7 @@ verify: ## Run all static checks across the repo (Go + TS + docs, parallelized) @printf "$(GREEN)$(BOLD)✔ All static checks passed$(RESET)\n" .PHONY: verify-parallel -verify-parallel: tidy fmt-go lint-go lint-ts lint-md lint-prose lint-sh lint-gha test-classify-paths test-md-rules test-release-channel vulncheck check-docs typecheck-ts +verify-parallel: tidy fmt-go lint-go lint-go-sdk lint-ts lint-md lint-prose lint-sh lint-gha test-classify-paths test-md-rules test-release-channel vulncheck check-docs typecheck-ts verify-go-sdk # typecheck-ts: tsc --noEmit on the SDK. Its own target (was inline in verify's # recipe) so it can run as a parallel leaf of verify-parallel. @@ -744,7 +761,7 @@ test-unit: go-mod-download ## Run Go unit tests + render coverage + gate thresho # Hidden alias: `make test` matches `go test ./...` muscle memory; test-unit # is the explicit form. .PHONY: test -test: test-unit +test: test-unit test-go-sdk .PHONY: test-integration test-integration: go-mod-download ## Run Go integration tests + render coverage + gate threshold (requires Docker) @@ -787,12 +804,55 @@ test-ts: pnpm-install ## Run SDK vitest unit tests + coverage + gate against sui $(if $(COV_DEFER),,--coverage.thresholds.statements=$$(go run ./scripts/cov threshold ts-unit)) $(ARGS) @if [ -z "$(COV_DEFER)" ]; then printf "$(GREEN)==> ts-unit gate passed$(RESET) HTML: tmp/coverage/ts-unit/index.html\n"; fi +# test-go-sdk: unit tests for clients/go/ — a nested Go module (its own +# go.mod), so it's outside test-unit's ./internal/... ./cmd/... scope and +# needs its own target. -race because the SDK's streaming subsystem is the +# most concurrent code in the repo. +# +# Coverage is collected exactly like the root-module Go suites (covdata into +# tmp/coverage//data via -test.gocoverdir), so `cov render go-sdk` +# renders + gates it with no new machinery and CI's coverage fragment — +# `path: tmp/coverage` on the unit job, which already runs this target — +# carries it to the `coverage` job unchanged. -coverpkg=./... resolves +# inside clients/go, so the denominator is the SDK package + the codegen +# command, nothing from the server. +# +# The go-sdk suite is NOT part of the merged Go total: a nested module is +# invisible to the root module (`go list ./...` at the repo root never +# yields clients/go), so the other suites' -coverpkg=./... cannot reach +# these files — they can't leak into tmp/coverage/total, and no +# exclude.paths entry is needed to keep them out. Same separation the TS +# SDK gets via ts-*. Gate: suites.go-sdk in .testcoverage.yml. +.PHONY: test-go-sdk +test-go-sdk: ## Run Go SDK (clients/go, a nested module) unit tests + render coverage + gate threshold + @printf "$(CYAN)==> Running Go SDK tests...$(RESET)\n" + @rm -rf $(COV_GOSDK)/data && mkdir -p $(COV_GOSDK)/data + @cd clients/go && GOCOVERDIR="$(CURDIR)/$(COV_GOSDK)/data" go test -cover -coverpkg=./... -race ./... \ + -args -test.gocoverdir="$(CURDIR)/$(COV_GOSDK)/data" + @if [ -z "$(COV_DEFER)" ]; then go run ./scripts/cov render go-sdk; fi + +# test-conformance-ts: the TS half of the cross-SDK wire-format conformance +# suite (the Go half is clients/go/conformance_test.go, run by test-go-sdk). +# Both replay clients/go/testdata/wire_cases.json. +.PHONY: test-conformance-ts +test-conformance-ts: build-ts ## Run TS SDK wire-format conformance against the shared fixture + @printf "$(CYAN)==> Running TS wire-format conformance...$(RESET)\n" + @node tests/conformance/conformance_ts.mjs + +# test-go-sdk-e2e: E2E against live server. WAVEHOUSE_URL + WAVEHOUSE_AUTH env vars. +.PHONY: test-go-sdk-e2e +test-go-sdk-e2e: ## Run Go SDK E2E tests against a live WaveHouse instance (WAVEHOUSE_URL, WAVEHOUSE_AUTH) + @printf "$(CYAN)==> Running Go SDK E2E tests...$(RESET)\n" + @cd clients/go && go test -tags e2e -v -count=1 -timeout 60s ./... + # Aggregator: recipe-based with $(MAKE) calls so suites run sequentially even # under `make -j N`. The suites bind ports / spin testcontainers / start the # release binary, so concurrent execution is unsafe. .PHONY: test-all test-all: ## Run all suites sequentially + one consolidated Go + TS coverage report + gates @$(MAKE) test-unit COV_DEFER=1 + @$(MAKE) test-go-sdk COV_DEFER=1 + @$(MAKE) test-conformance-ts @$(MAKE) test-ts COV_DEFER=1 @$(MAKE) test-integration COV_DEFER=1 @$(MAKE) test-e2e COV_DEFER=1 @@ -832,7 +892,7 @@ cov: go-mod-download ## Consolidated coverage report (Go + TS) + gate against th # marker that standalone `make verify` writes is instead written by ci's own # `ci-marker.sh write` below — it touches both the ci and verify markers. .PHONY: ci-parallel -ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts +ci-parallel: verify-parallel build build-cover build-ts build-docs test test-ts test-conformance-ts .PHONY: ci ci: ## Full pipeline — parallel checks, then sequential heavy suites + coverage diff --git a/README.md b/README.md index 4a02098a..34d16188 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ If you're building user-facing analytics, WaveHouse is like **Supabase for Click - **Query** — in-process Ristretto cache + `singleflight` coalescing; type-safe structured query AST; Tinybird-style named pipes (parameterized SQL endpoints). - **Real-time** — native SSE push, broadcast *before* the ClickHouse flush, with JetStream gap-fill for late/reconnecting clients. - **Security** — Hasura-style per-table, per-role column + row policies with JWT claim templating, stored in NATS KV. -- **Client** — `@wavehouse/sdk`: TypeScript client with query builder, live queries, streaming, and schema codegen; one runtime dependency (an SSE frame parser, ~1.4 KB gzipped). +- **Client SDKs** — TypeScript (`@wavehouse/sdk`) and Go (`github.com/Wave-RF/WaveHouse/clients/go`): query builder, live queries, streaming, and schema codegen in both. TypeScript has one runtime dependency (an SSE frame parser, ~1.4 KB gzipped); Go has none. ## How it compares diff --git a/biome.json b/biome.json index e29be7f3..700f3729 100644 --- a/biome.json +++ b/biome.json @@ -10,6 +10,7 @@ "clients/ts/src/**", "clients/ts/*.{ts,js,mjs,cjs,json}", "tests/e2e/sdk/**/*.{ts,js,mjs,cjs,json}", + "tests/conformance/**/*.{ts,js,mjs,cjs,json}", "docs/**/*.{ts,js,mjs,cjs,json}", "scripts/**/*.{ts,js,mjs,cjs}" ] diff --git a/clients/go/README.md b/clients/go/README.md new file mode 100644 index 00000000..64e19cac --- /dev/null +++ b/clients/go/README.md @@ -0,0 +1,225 @@ +# WaveHouse Go SDK + +Official Go client for [WaveHouse](https://github.com/Wave-RF/WaveHouse) — a schema-aware real-time API gateway for ClickHouse. + +**Zero third-party runtime dependencies** (stdlib only). + +**[Full SDK documentation on wavehouse.dev](https://wavehouse.dev/sdk/go)** + +## Install + +```bash +go get github.com/Wave-RF/WaveHouse/clients/go +``` + +## Quick Start + +```go +package main + +import ( + "context" + "fmt" + "log" + + wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +) + +func main() { + // Create an unauthenticated client (uses the server's default_role). + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + }) + + // Health check. + if err := client.Sys.Health(context.Background()); err != nil { + log.Fatal(err) + } + + ctx := context.Background() + + // Insert a row. + _, err := client.From("clicks").Insert(ctx, map[string]any{ + "page": "/home", "button": "cta", + }) + if err != nil { + log.Fatal(err) + } + + // Query with the fluent builder. + page, err := client.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + OrderBy("page", "asc"). + Limit(10). + FetchUntyped(ctx) + if err != nil { + log.Fatal(err) + } + for _, row := range page.Data { + fmt.Println(row["page"], row["button"]) + } +} +``` + +## Authentication + +```go +// Static token. +client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), +}) + +// Dynamic token (e.g. rotated). +client = wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: func(ctx context.Context) (string, error) { + return fetchFreshToken(ctx) + }, +}) +``` + +`BaseURL` may include a path prefix (`https://app.example.com/api/warehouse`). A trailing `/` is trimmed, and request paths are appended, for both REST and SSE alike ([Config](https://wavehouse.dev/sdk/go#config)). + +## Typed Queries (Generics) + +```go +type ClickRow struct { + Page string `json:"page"` + Button string `json:"button"` + DurationMS int `json:"duration_ms"` +} + +page, err := wavehouse.FetchTyped[ClickRow](ctx, + client.From("clicks").Select("page", "button", "duration_ms").Limit(100), +) +// page.Data is []ClickRow +``` + +## Batch Insert (NDJSON) + +```go +// Array of maps — serialized to NDJSON automatically. +result, _ := client.From("clicks").Insert(ctx, []map[string]any{ + {"page": "/a", "button": "cta"}, + {"page": "/b", "button": "nav"}, +}) +// result.OK, result.Total, result.Succeeded, result.Failed + +// Pre-formatted NDJSON string. +result, _ = client.From("clicks").InsertNDJSON(ctx, + `{"page":"/a"}`+"\n"+`{"page":"/b"}`, +) +``` + +## Streaming (SSE) + +```go +stream := client.From("clicks").Stream(&wavehouse.StreamOptions{ + Since: "2026-01-01T00:00:00Z", +}) +defer stream.Close() + +// Channel-based consumption. +for event := range stream.Events() { + fmt.Println(event.Table, event.Data) +} + +// Or callback-based. +unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ + Next: func(e wavehouse.StreamEvent) { fmt.Println(e.Data) }, + Status: func(s wavehouse.StreamStatus) { fmt.Println("status:", s) }, +}) +defer unsub() +``` + +## Live Queries + +```go +lq := client.From("clicks"). + SelectAll(). + OrderBy("received_timestamp", "desc"). + Limit(100). + LiveQuery(&wavehouse.StreamSubscriber{ + Initial: func(rows []map[string]any, err error) { + // Historical backfill. + fmt.Println("initial rows:", len(rows)) + }, + Next: func(e wavehouse.StreamEvent) { + // Live events after backfill. + fmt.Println("live:", e.Data) + }, + }, nil) +defer lq.Close() +``` + +## Named Pipes + +```go +// Execute a pipe. +rows, _ := wavehouse.Fetch[map[string]any](ctx, + client.Pipe("top_pages", map[string]any{"limit": 10}), +) + +// Admin: manage pipes. +client.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ + SQL: "SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}", + AllowedRoles: []string{"viewer", "admin"}, +}) +pipes, _ := client.Pipes.List(ctx) +client.Pipes.Delete(ctx, "old_pipe") +``` + +## Admin + +```go +// Schema introspection (admin-only). +schemas, _ := client.Schema.List(ctx) +client.Schema.Refresh(ctx) + +// Policy management (admin-only). +policy, _ := client.Policy.Get(ctx) +client.Policy.Set(ctx, policy) +result, _ := client.Policy.Validate(ctx, policy) + +// DLQ stats (admin-only). +stats, _ := client.DLQ.List(ctx) + +// Raw SQL (admin-only). +rows, _ := wavehouse.SQL[map[string]any](ctx, client, "SELECT count() FROM clicks") +``` + +## Codegen + +Generate Go structs from a running WaveHouse instance: + +```bash +export WAVEHOUSE_AUTH='' # avoids leaking the token via argv +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ + --url http://localhost:8080 \ + --out ./db_types.go \ + --package myapp +``` + +See the [full type mapping in the docs](https://wavehouse.dev/sdk/go/reference#codegen-cli). + +## Error Handling + +Request-response ops return `(T, error)`, or bare `error` for body-less calls (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP errors are `*wavehouse.Error` (unwrap with `errors.As`); failures before the request goes out (`Auth` provider, body marshal) are plain wrapped errors, so handle `errors.As == false` too. Streaming lifecycle (`Stream`, `Subscribe`, `Close`, `Connected`) reports via callbacks or plain errors: + +```go +page, err := client.From("clicks").Fetch(ctx) +if err != nil { + var whErr *wavehouse.Error + if errors.As(err, &whErr) { + fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } +} +``` + +The HTTP layer retries 5xx, 429, and network errors with exponential backoff (2 retries by default). `Retry-After` on 503/429 is honored, capped at 30s. Context cancellation returns `ABORTED` immediately. + +## License + +Apache-2.0 diff --git a/clients/go/client_test.go b/clients/go/client_test.go new file mode 100644 index 00000000..2ba55833 --- /dev/null +++ b/clients/go/client_test.go @@ -0,0 +1,122 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "testing" +) + +func TestNewClient_Defaults(t *testing.T) { + c := NewClient(Config{BaseURL: "http://localhost:8080"}) + if c.ctx.maxRetries != 2 { + t.Fatalf("want default maxRetries=2, got %d", c.ctx.maxRetries) + } + if c.ctx.baseURL != "http://localhost:8080" { + t.Fatalf("want baseURL, got %s", c.ctx.baseURL) + } +} + +func TestNewClient_StripsTrailingSlashes(t *testing.T) { + c := NewClient(Config{BaseURL: "http://localhost:8080///"}) + if c.ctx.baseURL != "http://localhost:8080" { + t.Fatalf("want stripped URL, got %s", c.ctx.baseURL) + } +} + +func TestNewClient_CustomMaxRetries(t *testing.T) { + c := NewClient(Config{ + BaseURL: "http://localhost:8080", + Options: &ClientOptions{MaxRetries: 5}, + }) + if c.ctx.maxRetries != 5 { + t.Fatalf("want 5, got %d", c.ctx.maxRetries) + } +} + +func TestNewClient_HasNamespaces(t *testing.T) { + c := NewClient(Config{BaseURL: "http://localhost:8080"}) + // Compared as concrete typed pointers, not boxed into map[string]any: a nil + // typed pointer in an interface is never == nil, so the map form passed + // even if NewClient stopped assigning a namespace entirely. + if c.Sys == nil { + t.Error("Sys namespace is nil") + } + if c.Schema == nil { + t.Error("Schema namespace is nil") + } + if c.Policy == nil { + t.Error("Policy namespace is nil") + } + if c.Pipes == nil { + t.Error("Pipes namespace is nil") + } + if c.DLQ == nil { + t.Error("DLQ namespace is nil") + } +} + +func TestClient_From(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + // Verify the table name appears in the URL. + if r.URL.Query().Get("table") != "events" { + t.Errorf("want table=events, got %s", r.URL.Query().Get("table")) + } + _ = json.NewEncoder(w).Encode([]map[string]any{}) + })) + t.Cleanup(srv.Close) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + // Checked, not discarded: if Fetch returns before issuing the request, the + // handler never runs and the table= assertion above proves nothing. + if _, err := c.From("events").Fetch(context.Background()); err != nil { + t.Fatal(err) + } +} + +func TestClient_SQL(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/ops/query" { + t.Errorf("want /v1/ops/query, got %s", r.URL.Path) + } + var body map[string]string + _ = json.NewDecoder(r.Body).Decode(&body) + if body["sql"] != "SELECT 1" { + t.Errorf("want sql=SELECT 1, got %s", body["sql"]) + } + _ = json.NewEncoder(w).Encode([]map[string]any{{"x": 1}}) + })) + t.Cleanup(srv.Close) + c := NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) + rows, err := SQL[map[string]any](context.Background(), c, "SELECT 1") + if err != nil { + t.Fatal(err) + } + if len(rows) != 1 { + t.Fatalf("want 1 row, got %d", len(rows)) + } +} + +func TestStaticToken(t *testing.T) { + fn := StaticToken("abc") + token, err := fn(context.Background()) + if err != nil { + t.Fatal(err) + } + if token != "abc" { + t.Fatalf("want abc, got %s", token) + } +} + +func TestPolicyFilter_MarshalOperators(t *testing.T) { + empty := "" + raw, err := json.Marshal(PolicyFilter{Eq: &empty}) + if err != nil { + t.Fatal(err) + } + // Intentional empty-string comparison survives; unset operators are + // omitted entirely, never sent as null. + if string(raw) != `{"_eq":""}` { + t.Fatalf(`want {"_eq":""}, got %s`, raw) + } +} diff --git a/clients/go/cmd/wavehouse-codegen/main.go b/clients/go/cmd/wavehouse-codegen/main.go new file mode 100644 index 00000000..9fa68258 --- /dev/null +++ b/clients/go/cmd/wavehouse-codegen/main.go @@ -0,0 +1,390 @@ +// Command wavehouse-codegen reads a WaveHouse server's /v1/ops/schema endpoint +// and generates Go struct definitions for use with the wavehouse SDK. +// +// Usage: +// +// WAVEHOUSE_AUTH= wavehouse-codegen --url http://localhost:8080 --out ./db.go +package main + +import ( + "context" + "encoding/json" + "fmt" + "go/format" + "net/http" + "os" + "slices" + "strings" + "time" + "unicode" +) + +type cliArgs struct { + url string + out string + auth string + pkg string +} + +// flagValue consumes and returns the value following os.Args[*i], erroring +// out instead of silently falling back to the default when it's missing. +func flagValue(i *int) string { + flag := os.Args[*i] + *i++ + if *i >= len(os.Args) { + fmt.Fprintf(os.Stderr, "Error: missing value for %s (use --help)\n", flag) + os.Exit(2) + } + return os.Args[*i] +} + +func parseArgs() cliArgs { + args := cliArgs{url: "http://localhost:8080", out: "./wavehouse_types.go", pkg: "main"} + for i := 1; i < len(os.Args); i++ { + switch os.Args[i] { + case "--url", "-u": + args.url = flagValue(&i) + case "--out", "-o": + args.out = flagValue(&i) + case "--auth", "-a": + args.auth = flagValue(&i) + case "--package", "-p": + args.pkg = flagValue(&i) + case "--help", "-h": + fmt.Println(`wavehouse-codegen — Generate Go types from WaveHouse schema + +Options: + --url, -u WaveHouse base URL (default: http://localhost:8080) + --out, -o Output .go file path (default: ./wavehouse_types.go) + --auth, -a Bearer token for authenticated /v1/ops/schema endpoint + (prefer the WAVEHOUSE_AUTH env var — argv leaks into + shell history and process listings) + --package, -p Go package name (default: main) + --help, -h Show this help`) + os.Exit(0) + default: + fmt.Fprintf(os.Stderr, "Error: unknown argument %q (use --help)\n", os.Args[i]) + os.Exit(2) + } + } + if args.auth == "" { + args.auth = os.Getenv("WAVEHOUSE_AUTH") + } + return args +} + +type column struct { + Name string `json:"name"` + Type string `json:"type"` + HasDefault bool `json:"has_default"` +} + +type tableSchema struct { + Name string `json:"name"` + Columns []column `json:"columns"` +} + +func fetchSchemas(ctx context.Context, baseURL, auth string) (map[string]tableSchema, error) { + url := strings.TrimRight(baseURL, "/") + "/v1/ops/schema" + req, err := http.NewRequestWithContext(ctx, "GET", url, nil) + if err != nil { + return nil, fmt.Errorf("build schema request for %s: %w", url, err) + } + if auth != "" { + req.Header.Set("Authorization", "Bearer "+auth) + } + client := &http.Client{Timeout: 30 * time.Second} + resp, err := client.Do(req) + if err != nil { + return nil, fmt.Errorf("fetch schema from %s: %w", url, err) + } + defer func() { _ = resp.Body.Close() }() + if resp.StatusCode != 200 { + return nil, fmt.Errorf("schema fetch failed: HTTP %d", resp.StatusCode) + } + + // Server returns either []tableSchema or map[string]tableSchema. + var raw json.RawMessage + if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil { + return nil, fmt.Errorf("read schema response: %w", err) + } + // Try array first. + var arr []tableSchema + if err := json.Unmarshal(raw, &arr); err == nil { + m := make(map[string]tableSchema, len(arr)) + for _, t := range arr { + m[t.Name] = t + } + return m, nil + } + var m map[string]tableSchema + if err := json.Unmarshal(raw, &m); err != nil { + return nil, fmt.Errorf("decode schema JSON: %w", err) + } + return m, nil +} + +// chTypeToGo maps a ClickHouse type string (as reported by /v1/ops/schema) to a +// Go type name suitable for a JSON struct field. +// +// We deliberately don't import clickhouse-go's type catalog +// (github.com/ClickHouse/clickhouse-go/v2/lib/column) for this. It's public +// and does expose a real ClickHouse-type-string parser — +// column.Type(chType).Column(name, sc).ScanType() — but it answers a +// different question than the one we're asking. That catalog maps to the Go +// types the *driver* scans query results into over the native protocol +// (time.Time for Date/DateTime*, uuid.UUID for UUID, decimal.Decimal for +// Decimal, net.IP for IPv4/IPv6, *big.Int for [U]Int128/256), not the types +// that round-trip cleanly through the JSON the /v1/ops/schema and query +// endpoints actually speak. ClickHouse's JSON output renders DateTime as +// "2024-01-15 10:30:00" (no "T", no offset), which fails Go's default +// time.Time JSON unmarshaling; big integers and decimals are similarly +// rendered as JSON strings, not driver-native types. Adopting the driver's +// ScanType() as-is would produce generated structs that don't unmarshal the +// server's actual JSON, and would drag uuid/decimal/orb/net imports into +// generated output that today has zero non-stdlib dependencies. So we keep +// the hand-rolled JSON-oriented mapping below, informed by (but not bound +// to) the type set clickhouse-go's lib/column recognizes. +func chTypeToGo(chType string) string { + // Unwrap Nullable → pointer. + if strings.HasPrefix(chType, "Nullable(") && strings.HasSuffix(chType, ")") { + inner := chType[9 : len(chType)-1] + return "*" + chTypeToGo(inner) + } + // Unwrap LowCardinality. + if strings.HasPrefix(chType, "LowCardinality(") && strings.HasSuffix(chType, ")") { + return chTypeToGo(chType[15 : len(chType)-1]) + } + // Unwrap SimpleAggregateFunction(func, InnerType) — readable columns in + // AggregatingMergeTree/SummingMergeTree rollup tables. The value on the + // wire is just InnerType; the aggregate function name only describes how + // merges combine rows. + if strings.HasPrefix(chType, "SimpleAggregateFunction(") && strings.HasSuffix(chType, ")") { + inner := chType[len("SimpleAggregateFunction(") : len(chType)-1] + if comma := findTopLevelComma(inner); comma != -1 { + return chTypeToGo(strings.TrimSpace(inner[comma+1:])) + } + return "any" + } + // String-like. + switch { + case chType == "String", + strings.HasPrefix(chType, "FixedString("), + chType == "UUID", + strings.HasPrefix(chType, "DateTime"), + strings.HasPrefix(chType, "Date"), + // Time/Time64 are ClickHouse's newer time-of-day types (distinct + // from DateTime); same JSON-string-not-RFC3339 story applies. + strings.HasPrefix(chType, "Time"), + strings.HasPrefix(chType, "Enum8("), + strings.HasPrefix(chType, "Enum16("), + chType == "IPv4", + chType == "IPv6": + return "string" + case chType == "Bool", chType == "Boolean": + return "bool" + } + // Numeric — map lookup. Generated structs target the structured-query and + // pipe paths (/v1/query, /v1/pipes/*), where the server scans ClickHouse + // values into Go types and re-marshals them — so 64-bit integers arrive + // as ordinary UNQUOTED JSON numbers and map to int64/uint64 exactly. + // (Only /v1/ops/query forwards ClickHouse's own JSON, which quotes + // 64-bit ints; use map[string]any with SQL[Row] there.) + if mapped, ok := map[string]string{ + "UInt8": "uint8", "UInt16": "uint16", "UInt32": "uint32", "UInt64": "uint64", + "Int8": "int8", "Int16": "int16", "Int32": "int32", "Int64": "int64", + "Float32": "float32", "Float64": "float64", "BFloat16": "float32", + }[chType]; ok { + return mapped + } + switch { + case strings.HasPrefix(chType, "Decimal"): + // Decimals are marshaled as quoted strings on the structured path + // (shopspring decimal.MarshalJSON quotes by default). + return "string" + case strings.HasPrefix(chType, "UInt128"), + strings.HasPrefix(chType, "UInt256"), + strings.HasPrefix(chType, "Int128"), + strings.HasPrefix(chType, "Int256"): + // 128/256-bit ints scan into *big.Int server-side and marshal as + // unquoted JSON numbers of arbitrary width — json.Number preserves + // them exactly where int64/uint64 would overflow. + return "json.Number" + } + // Array. + if strings.HasPrefix(chType, "Array(") && strings.HasSuffix(chType, ")") { + inner := chTypeToGo(chType[6 : len(chType)-1]) + // Array(UInt8) is asymmetric on the wire: ingest requires a real JSON + // array, but /v1/query responses currently base64-encode it (the + // server scans into []byte and encoding/json base64s that — #436). + // json.RawMessage is the + // only shape that round-trips both directions without a decode error. + if inner == "uint8" { + return "json.RawMessage" + } + return "[]" + inner + } + // Map. + if strings.HasPrefix(chType, "Map(") && strings.HasSuffix(chType, ")") { + inner := chType[4 : len(chType)-1] + comma := findTopLevelComma(inner) + if comma != -1 { + k := chTypeToGo(strings.TrimSpace(inner[:comma])) + v := chTypeToGo(strings.TrimSpace(inner[comma+1:])) + return "map[" + k + "]" + v + } + return "map[string]any" + } + return "any" +} + +func findTopLevelComma(s string) int { + depth := 0 + for i := range len(s) { + switch s[i] { + case '(': + depth++ + case ')': + depth-- + case ',': + if depth == 0 { + return i + } + } + } + return -1 +} + +func pascalCase(s string) string { + parts := strings.FieldsFunc(s, func(r rune) bool { + return r == '_' || r == '-' || r == ' ' || r == '.' + }) + var sb strings.Builder + for _, p := range parts { + if len(p) == 0 { + continue + } + runes := []rune(p) + runes[0] = unicode.ToUpper(runes[0]) + sb.WriteString(string(runes)) + } + result := sb.String() + if result == "" { + return result + } + // Go identifiers can't start with a digit (e.g. a table named + // "2fa_events" would otherwise produce the invalid identifier + // "2faEvents"). Prefix with "X" to keep it a valid, exported name. + if unicode.IsDigit(rune(result[0])) { // digits are ASCII; no rune-slice needed + result = "X" + result + } + return result +} + +func sortedKeys(m map[string]tableSchema) []string { + names := make([]string, 0, len(m)) + for name := range m { + names = append(names, name) + } + slices.Sort(names) + return names +} + +func generate(schemas map[string]tableSchema, pkg string) (string, error) { + var sb strings.Builder + + names := sortedKeys(schemas) + + // json.Number fields (128/256-bit integer columns) need the import. + needsJSON := false + for _, name := range names { + for _, col := range schemas[name].Columns { + if strings.Contains(chTypeToGo(col.Type), "json.") { + needsJSON = true + } + } + } + fmt.Fprintf(&sb, "// Code generated by wavehouse-codegen. DO NOT EDIT.\n\npackage %s\n\n", pkg) + if needsJSON { + sb.WriteString("import \"encoding/json\"\n\n") + } + + // pascalCase is not injective ("user_id" and "userId" both yield + // "UserId"), and format.Source only parses — it doesn't type-check — so + // a duplicate identifier would be written as a non-compiling file with a + // success message. Fail loudly instead. + seenTypes := make(map[string]string, len(names)) + for _, name := range names { + schema := schemas[name] + typeName := pascalCase(name) + "Row" + if prev, dup := seenTypes[typeName]; dup { + return "", fmt.Errorf("tables %q and %q both map to type %q; rename one or generate separately", prev, name, typeName) + } + seenTypes[typeName] = name + fmt.Fprintf(&sb, "// %s represents a row in the %q table.\ntype %s struct {\n", typeName, name, typeName) + seenFields := make(map[string]string, len(schema.Columns)) + for _, col := range schema.Columns { + goType := chTypeToGo(col.Type) + fieldName := pascalCase(col.Name) + if prev, dup := seenFields[fieldName]; dup { + return "", fmt.Errorf("table %q: columns %q and %q both map to field %q", name, prev, col.Name, fieldName) + } + seenFields[fieldName] = col.Name + jsonTag := col.Name + if col.HasDefault { + // Pointer + omitempty is the Go spelling of the TS codegen's + // `field?: T`: nil omits the field (server default applies), + // while a pointer to the zero value still sends an explicit + // 0/false/"" instead of silently dropping it. + jsonTag += ",omitempty" + if !strings.HasPrefix(goType, "*") { + goType = "*" + goType + } + } + fmt.Fprintf(&sb, "\t%s %s `json:%q`\n", fieldName, goType, jsonTag) + } + sb.WriteString("}\n\n") + } + + return sb.String(), nil +} + +func main() { + args := parseArgs() + fmt.Printf("Fetching schema from %s...\n", args.url) + + schemas, err := fetchSchemas(context.Background(), args.url, args.auth) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + if len(schemas) == 0 { + fmt.Fprintln(os.Stderr, "No tables found. Is WaveHouse running with tables in ClickHouse?") + os.Exit(1) + } + + names := sortedKeys(schemas) + fmt.Printf("Found %d table(s): %s\n", len(schemas), strings.Join(names, ", ")) + + output, err := generate(schemas, args.pkg) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + os.Exit(1) + } + + // gofmt the output. A failure here means the generated source is not + // valid Go (e.g. a table/column name produced an invalid identifier); + // don't write unusable output and claim success. + formatted, err := format.Source([]byte(output)) + if err != nil { + fmt.Fprintf(os.Stderr, "Error: generated code is not valid Go: %v\n", err) + os.Exit(1) + } + + if err := os.WriteFile(args.out, formatted, 0o600); err != nil { + fmt.Fprintf(os.Stderr, "Error writing %s: %v\n", args.out, err) + os.Exit(1) + } + + fmt.Printf("✓ Types written to %s\n", args.out) +} diff --git a/clients/go/cmd/wavehouse-codegen/main_test.go b/clients/go/cmd/wavehouse-codegen/main_test.go new file mode 100644 index 00000000..f6207ddc --- /dev/null +++ b/clients/go/cmd/wavehouse-codegen/main_test.go @@ -0,0 +1,182 @@ +package main + +import ( + "encoding/json" + "go/format" + "strings" + "testing" +) + +func TestChTypeToGo(t *testing.T) { + tests := []struct { + ch string + want string + }{ + {"String", "string"}, + {"FixedString(16)", "string"}, + {"UUID", "string"}, + {"DateTime64(3, 'UTC')", "string"}, + {"Date", "string"}, + {"Time64(3)", "string"}, + {"Enum8('a' = 1)", "string"}, + {"IPv4", "string"}, + {"Bool", "bool"}, + {"Boolean", "bool"}, + {"UInt8", "uint8"}, + {"UInt16", "uint16"}, + {"UInt32", "uint32"}, + {"Int8", "int8"}, + {"Int32", "int32"}, + {"Float32", "float32"}, + {"BFloat16", "float32"}, + {"Float64", "float64"}, + // /v1/query re-marshals server-side: 64-bit ints arrive unquoted. + {"UInt64", "uint64"}, + {"Int64", "int64"}, + {"UInt128", "json.Number"}, + {"Int256", "json.Number"}, + {"Decimal(18, 4)", "string"}, + {"Nullable(Int32)", "*int32"}, + {"Nullable(Int64)", "*int64"}, + {"LowCardinality(String)", "string"}, + {"LowCardinality(Nullable(String))", "*string"}, + {"SimpleAggregateFunction(sum, UInt32)", "uint32"}, + {"SimpleAggregateFunction(any)", "any"}, + {"Array(String)", "[]string"}, + {"Array(Nullable(Int32))", "[]*int32"}, + // []uint8 is []byte → base64 on marshal; RawMessage round-trips both + // the ingest array form and the (currently base64) query response. + {"Array(UInt8)", "json.RawMessage"}, + {"Map(String, UInt32)", "map[string]uint32"}, + {"Map(String, Map(UInt32, String))", "map[string]map[uint32]string"}, + {"Tuple(String, UInt8)", "any"}, + {"SomethingNew", "any"}, + } + for _, tt := range tests { + if got := chTypeToGo(tt.ch); got != tt.want { + t.Errorf("chTypeToGo(%q) = %q, want %q", tt.ch, got, tt.want) + } + } +} + +func TestPascalCase(t *testing.T) { + tests := []struct { + in string + want string + }{ + {"clicks", "Clicks"}, + {"user_id", "UserId"}, + {"received_timestamp", "ReceivedTimestamp"}, + {"multi-part.name here", "MultiPartNameHere"}, + {"2fa_events", "X2faEvents"}, // leading digit gets the X prefix + {"", ""}, + } + for _, tt := range tests { + if got := pascalCase(tt.in); got != tt.want { + t.Errorf("pascalCase(%q) = %q, want %q", tt.in, got, tt.want) + } + } +} + +func TestFindTopLevelComma(t *testing.T) { + tests := []struct { + in string + want int + }{ + {"String, UInt32", 6}, + {"Map(String, String), UInt8", 19}, + {"NoComma", -1}, + } + for _, tt := range tests { + if got := findTopLevelComma(tt.in); got != tt.want { + t.Errorf("findTopLevelComma(%q) = %d, want %d", tt.in, got, tt.want) + } + } +} + +func TestGenerate_Basic(t *testing.T) { + out, err := generate(map[string]tableSchema{ + "clicks": {Name: "clicks", Columns: []column{ + {Name: "page", Type: "String"}, + {Name: "score", Type: "Float64"}, + {Name: "received_timestamp", Type: "DateTime64(3, 'UTC')", HasDefault: true}, + }}, + }, "myapp") + if err != nil { + t.Fatal(err) + } + for _, want := range []string{ + "package myapp", + "type ClicksRow struct {", + "Page string `json:\"page\"`", + "Score float64 `json:\"score\"`", + // Defaulted column: pointer + omitempty so an explicit zero still sends. + "ReceivedTimestamp *string `json:\"received_timestamp,omitempty\"`", + } { + if !strings.Contains(out, want) { + t.Errorf("generated output missing %q:\n%s", want, out) + } + } +} + +func TestGenerate_FieldCollisionFails(t *testing.T) { + _, err := generate(map[string]tableSchema{ + "t": {Name: "t", Columns: []column{ + {Name: "user_id", Type: "String"}, + {Name: "userId", Type: "String"}, + }}, + }, "main") + if err == nil || !strings.Contains(err.Error(), "UserId") { + t.Fatalf("want field-collision error naming UserId, got %v", err) + } +} + +func TestGenerate_TypeCollisionFails(t *testing.T) { + _, err := generate(map[string]tableSchema{ + "2fa": {Name: "2fa", Columns: []column{{Name: "a", Type: "String"}}}, + "x2fa": {Name: "x2fa", Columns: []column{{Name: "a", Type: "String"}}}, + }, "main") + if err == nil || !strings.Contains(err.Error(), "X2faRow") { + t.Fatalf("want type-collision error naming X2faRow, got %v", err) + } +} + +// TestGeneratedShapeDecodesStructuredQueryPayload asserts the mapping choices +// actually decode what /v1/query emits: the server scans ClickHouse values +// into Go types and re-marshals, so 64-bit ints are unquoted numbers, +// 128/256-bit ints are unquoted arbitrary-width numbers, and Decimals are +// quoted strings. +func TestGeneratedShapeDecodesStructuredQueryPayload(t *testing.T) { + type row struct { + ID uint64 `json:"id"` + Delta int64 `json:"delta"` + Big json.Number `json:"big"` + Price string `json:"price"` + } + payload := `[{"id":18446744073709551615,"delta":-9007199254740993,"big":170141183460469231731687303715884105727,"price":"12.3400"}]` + var rows []row + if err := json.Unmarshal([]byte(payload), &rows); err != nil { + t.Fatalf("generated shape failed to decode /v1/query payload: %v", err) + } + if rows[0].ID != 18446744073709551615 || rows[0].Delta != -9007199254740993 { + t.Fatalf("64-bit values corrupted: %+v", rows[0]) + } + if rows[0].Big.String() != "170141183460469231731687303715884105727" { + t.Fatalf("128-bit value corrupted: %s", rows[0].Big) + } +} + +func TestGenerate_JSONNumberImport(t *testing.T) { + out, err := generate(map[string]tableSchema{ + "t": {Name: "t", Columns: []column{{Name: "big", Type: "UInt128"}}}, + }, "main") + if err != nil { + t.Fatal(err) + } + if !strings.Contains(out, `import "encoding/json"`) { + t.Fatalf("json.Number field without encoding/json import:\n%s", out) + } + if _, err := format.Source([]byte(out)); err != nil { + t.Fatalf("generated output is not valid Go: %v", err) + } +} diff --git a/clients/go/conformance_test.go b/clients/go/conformance_test.go new file mode 100644 index 00000000..c8af2899 --- /dev/null +++ b/clients/go/conformance_test.go @@ -0,0 +1,432 @@ +package wavehouse + +import ( + "context" + _ "embed" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "net/url" + "reflect" + "strings" + "sync" + "testing" +) + +// logCallErr surfaces SDK-call errors that the conformance harness otherwise +// ignores — the assertions only inspect the captured request, but when a call +// fails before sending, the failure message should name the real cause. +func logCallErr(t *testing.T, err error) { + t.Helper() + if err != nil { + t.Logf("SDK call returned error (request may still be valid): %v", err) + } +} + +// wireCasesJSON embeds the shared wire-format conformance fixture so the +// test binary is self-contained: it works from a module archive or a +// standalone checkout without depending on paths outside the Go module. +// +//go:embed testdata/wire_cases.json +var wireCasesJSON []byte + +// wireCase is one entry in the shared wire_cases.json fixture. +type wireCase struct { + Name string `json:"name"` + Endpoint string `json:"endpoint"` + Table string `json:"table"` + Operations []wireOp `json:"operations"` + PipeName string `json:"pipe_name"` + PipeParams map[string]any `json:"pipe_params"` + PipeDefBody json.RawMessage `json:"pipe_def"` + PolicyBody json.RawMessage `json:"policy_body"` + SQL string `json:"sql"` + ExpectedPath string `json:"expected_path"` + ExpectedMethod string `json:"expected_method"` + ExpectedContentType string `json:"expected_content_type"` + ExpectedBody json.RawMessage `json:"expected_body"` + ExpectedRawBody *string `json:"expected_raw_body"` +} + +type wireOp struct { + Method string `json:"method"` + Args []any `json:"args"` +} + +func loadWireCases(t *testing.T) []wireCase { + t.Helper() + var cases []wireCase + if err := json.Unmarshal(wireCasesJSON, &cases); err != nil { + t.Fatalf("parse wire_cases.json: %v", err) + } + return cases +} + +// captured holds the HTTP request details from a single SDK call. +type captured struct { + method string + path string // path + query string + contentType string + body string +} + +func TestConformance_WireFormat(t *testing.T) { + cases := loadWireCases(t) + + for _, tc := range cases { + t.Run(tc.Name, func(t *testing.T) { + // capt is written on the server goroutine and read on the test + // goroutine; the mutex is what makes that visible under -race. + var mu sync.Mutex + var capt captured + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + capt.method = r.Method + capt.path = r.URL.RequestURI() + capt.contentType = r.Header.Get("Content-Type") + raw, _ := io.ReadAll(r.Body) + capt.body = string(raw) + mu.Unlock() + + // Return valid JSON so the SDK doesn't error on decode. + w.Header().Set("Content-Type", "application/json") + switch { + case strings.HasPrefix(r.URL.Path, "/v1/ops/dlq"): + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{}, Total: 0}) + case strings.HasPrefix(r.URL.Path, "/v1/ops/schema") && r.Method == "GET": + _ = json.NewEncoder(w).Encode([]TableSchema{}) + case r.URL.Path == "/v1/ops/policy/validate" && r.Method == "POST": + _ = json.NewEncoder(w).Encode(ValidationResult{Valid: true}) + case strings.HasPrefix(r.URL.Path, "/v1/ops/policy") && r.Method == "GET": + _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) + case strings.HasPrefix(r.URL.Path, "/v1/ops/pipes/") && r.Method == "GET": + _ = json.NewEncoder(w).Encode(Pipe{Name: "test", SQL: "SELECT 1"}) + case r.URL.Path == "/v1/ops/pipes" && r.Method == "GET": + _ = json.NewEncoder(w).Encode([]Pipe{}) + default: + _ = json.NewEncoder(w).Encode([]map[string]any{}) + } + })) + defer srv.Close() + + c := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{MaxRetries: 0}, + }) + ctx := context.Background() + + // Execute the case. + switch tc.Endpoint { + case "query": + q := applyOps(t, tc.Table, c, tc.Operations) + _, err := q.FetchUntyped(ctx) + logCallErr(t, err) + + case "ingest": + if len(tc.Operations) == 0 || tc.Operations[0].Method != "insert" { + t.Fatalf("ingest case %q has no insert operation", tc.Name) + } + if len(tc.Operations[0].Args) == 0 { + t.Fatal("insert needs 1 arg, got 0") + } + data := tc.Operations[0].Args[0] + _, err := c.From(tc.Table).Insert(ctx, data) + logCallErr(t, err) + + case "ingest_batch": + if len(tc.Operations) == 0 || tc.Operations[0].Method != "insert" { + t.Fatalf("ingest_batch case %q has no insert operation", tc.Name) + } + if len(tc.Operations[0].Args) == 0 { + t.Fatal("insert needs 1 arg, got 0") + } + rawArr, ok := tc.Operations[0].Args[0].([]any) + if !ok { + t.Fatalf("batch insert args[0] is not an array") + } + rows := make([]map[string]any, len(rawArr)) + for i, r := range rawArr { + rows[i] = toStringMap(t, r) + } + _, batchErr := c.From(tc.Table).Insert(ctx, rows) + logCallErr(t, batchErr) + + case "pipe": + p := c.Pipe(tc.PipeName, tc.PipeParams) + _, err := p.FetchUntyped(ctx) + logCallErr(t, err) + + case "sql": + _, err := SQL[map[string]any](ctx, c, tc.SQL) + logCallErr(t, err) + + case "health": + logCallErr(t, c.Sys.Health(ctx)) + + case "schema_list": + _, err := c.Schema.List(ctx) + logCallErr(t, err) + + case "schema_refresh": + logCallErr(t, c.Schema.Refresh(ctx)) + + case "policy_get": + _, err := c.Policy.Get(ctx) + logCallErr(t, err) + + case "policy_set": + var pol Policy + if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { + t.Fatalf("parse policy_body: %v", err) + } + logCallErr(t, c.Policy.Set(ctx, &pol)) + + case "policy_validate": + var pol Policy + if err := json.Unmarshal(tc.PolicyBody, &pol); err != nil { + t.Fatalf("parse policy_body: %v", err) + } + _, err := c.Policy.Validate(ctx, &pol) + logCallErr(t, err) + + case "dlq_list": + _, err := c.DLQ.List(ctx) + logCallErr(t, err) + + case "dlq_table": + _, err := c.DLQ.Table(ctx, tc.Table) + logCallErr(t, err) + + case "pipes_list": + _, err := c.Pipes.List(ctx) + logCallErr(t, err) + + case "pipes_get": + _, err := c.Pipes.Get(ctx, tc.PipeName) + logCallErr(t, err) + + case "pipes_set": + var def PipeDef + if err := json.Unmarshal(tc.PipeDefBody, &def); err != nil { + t.Fatalf("parse pipe_def: %v", err) + } + logCallErr(t, c.Pipes.Set(ctx, tc.PipeName, def)) + + case "pipes_delete": + logCallErr(t, c.Pipes.Delete(ctx, tc.PipeName)) + + default: + // Hard failure, matching the TS runner: skipped cases break + // cross-SDK parity. + t.Fatalf("unhandled endpoint %q — wire it up in the dispatch switch", tc.Endpoint) + } + + // Verify method. + mu.Lock() + defer mu.Unlock() + if tc.ExpectedMethod != "" && capt.method != tc.ExpectedMethod { + t.Errorf("method: want %s, got %s", tc.ExpectedMethod, capt.method) + } + + // Verify path. + if tc.ExpectedPath != "" { + // Normalize: the SDK may use different encoding (+ vs %20). + wantPath := normalizePath(tc.ExpectedPath) + gotPath := normalizePath(capt.path) + if wantPath != gotPath { + t.Errorf("path: want %s, got %s", tc.ExpectedPath, capt.path) + } + } + + // Verify content type. + if tc.ExpectedContentType != "" && capt.contentType != tc.ExpectedContentType { + t.Errorf("content-type: want %s, got %s", tc.ExpectedContentType, capt.contentType) + } + + // Verify raw body (for NDJSON). + if tc.ExpectedRawBody != nil { + if capt.body != *tc.ExpectedRawBody { + t.Errorf("raw body:\n want: %s\n got: %s", *tc.ExpectedRawBody, capt.body) + } + return + } + + // Verify JSON body. + if tc.ExpectedBody != nil && string(tc.ExpectedBody) != "null" { + var want, got any + if err := json.Unmarshal(tc.ExpectedBody, &want); err != nil { + t.Fatalf("parse expected_body: %v", err) + } + if err := json.Unmarshal([]byte(capt.body), &got); err != nil { + t.Fatalf("parse captured body: %v (body: %s)", err, capt.body) + } + if !deepEqualJSON(want, got) { + wantJSON, _ := json.MarshalIndent(want, "", " ") + gotJSON, _ := json.MarshalIndent(got, "", " ") + t.Errorf("body mismatch:\n want: %s\n got: %s", wantJSON, gotJSON) + } + } + }) + } +} + +// applyOps replays the operation chain from the fixture onto a QueryBuilder. +// Fixtures always put select first (mirroring real usage), so rebuilding on +// select is safe and keeps this simple. +func applyOps(t *testing.T, table string, c *Client, ops []wireOp) *QueryBuilder { + t.Helper() + q := c.From(table).Select() + + for _, op := range ops { + switch op.Method { + case "select": + q = c.From(table).Select(toStringSlice(op.Args)...) + case "selectAll": + q = q.SelectAll() + case "where": + if len(op.Args) != 3 { + t.Fatalf("where needs 3 args, got %d", len(op.Args)) + } + col, ok := op.Args[0].(string) + if !ok { + t.Fatalf("where: column arg is %T, want string", op.Args[0]) + } + rawOp, ok := op.Args[1].(string) + if !ok { + t.Fatalf("where: operator arg is %T, want string", op.Args[1]) + } + opStr := FilterOp(rawOp) + val := op.Args[2] + q = q.Where(col, opStr, val) + case "count": + col, alias := stringArg(op.Args, 0, "*"), stringArg(op.Args, 1, "count") + q = q.Count(col, alias) + case "sum": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Sum(col, alias) + case "avg": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Avg(col, alias) + case "min": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Min(col, alias) + case "max": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.Max(col, alias) + case "countDistinct": + col, alias := stringArg(op.Args, 0, ""), stringArg(op.Args, 1, "") + q = q.CountDistinct(col, alias) + case "aggregate": + fn := stringArg(op.Args, 0, "") + col := stringArg(op.Args, 1, "") + alias := stringArg(op.Args, 2, "") + q = q.Aggregate(fn, col, alias) + case "groupBy": + cols := toStringSlice(op.Args) + q = q.GroupBy(cols...) + case "orderBy": + col := stringArg(op.Args, 0, "") + dir := stringArg(op.Args, 1, "asc") + q = q.OrderBy(col, dir) + case "limit": + n := intArg(op.Args, 0) + q = q.Limit(n) + case "timeRange": + col := stringArg(op.Args, 0, "") + since := stringArg(op.Args, 1, "") + until := stringArg(op.Args, 2, "") + q = q.TimeRange(col, since, until) + case "cacheTTL": + n := intArg(op.Args, 0) + q = q.CacheTTL(n) + } + } + return q +} + +func stringArg(args []any, i int, fallback string) string { + if i >= len(args) { + return fallback + } + s, ok := args[i].(string) + if !ok { + return fallback + } + return s +} + +func intArg(args []any, i int) int { + if i >= len(args) { + return 0 + } + switch v := args[i].(type) { + case float64: + return int(v) + case int: + return v + default: + return 0 + } +} + +func toStringSlice(args []any) []string { + out := make([]string, len(args)) + for i, a := range args { + out[i], _ = a.(string) + } + return out +} + +func toStringMap(t *testing.T, v any) map[string]any { + t.Helper() + m, ok := v.(map[string]any) + if !ok { + t.Fatalf("fixture row is not an object: %T", v) + } + return m +} + +// deepEqualJSON compares two JSON-decoded values, treating float64 ints as equal +// to ints (JSON numbers decode as float64 in Go). +func deepEqualJSON(a, b any) bool { + return reflect.DeepEqual(normalizeJSON(a), normalizeJSON(b)) +} + +func normalizeJSON(v any) any { + switch val := v.(type) { + case map[string]any: + m := make(map[string]any, len(val)) + for k, v := range val { + m[k] = normalizeJSON(v) + } + return m + case []any: + s := make([]any, len(val)) + for i, v := range val { + s[i] = normalizeJSON(v) + } + return s + case float64: + // Normalize integer-valued floats to int for comparison. + if val == float64(int64(val)) { + return int64(val) + } + return val + default: + return val + } +} + +// normalizePath compares request URIs by meaning: same path, same decoded +// query values regardless of + vs %20 spelling or parameter order. A raw +// string replace would also rewrite literal + characters and stop asserting +// the encoding at all. +func normalizePath(p string) string { + u, err := url.ParseRequestURI(p) + if err != nil { + return p + } + return u.Path + "?" + u.Query().Encode() +} diff --git a/clients/go/dlq.go b/clients/go/dlq.go new file mode 100644 index 00000000..65712765 --- /dev/null +++ b/clients/go/dlq.go @@ -0,0 +1,45 @@ +package wavehouse + +import ( + "context" + "fmt" + "net/url" +) + +// DLQNamespace provides admin-only dead-letter-queue statistics. +// +// The server registers /v1/ops/dlq/stats only when the DLQ is enabled, so on a +// deployment with dlq.enabled: false these calls return an [*Error] with +// Status 404 — "the DLQ is switched off", not "the DLQ is empty". Check +// Status before reading a zero DLQStats as a healthy result. +type DLQNamespace struct { + ctx httpContext + createStream func(table string, opts *StreamOptions) *StreamController +} + +// List returns DLQ statistics (message counts per table). Admin-only. +func (d *DLQNamespace) List(ctx context.Context) (*DLQStats, error) { + return d.stats(ctx, nil) +} + +// Table returns DLQ stats filtered by table name. Admin-only. +func (d *DLQNamespace) Table(ctx context.Context, name string) (*DLQStats, error) { + return d.stats(ctx, url.Values{"table": {name}}) +} + +func (d *DLQNamespace) stats(ctx context.Context, params url.Values) (*DLQStats, error) { + var stats DLQStats + if err := doRequest(ctx, d.ctx, requestOptions{ + method: "GET", + path: "/v1/ops/dlq/stats", + params: params, + }, &stats); err != nil { + return nil, fmt.Errorf("get dlq stats: %w", err) + } + return &stats, nil +} + +// Stream subscribes to live DLQ events. Not yet functional server-side (#197). +func (d *DLQNamespace) Stream(opts *StreamOptions) *StreamController { + return d.createStream("dlq", opts) +} diff --git a/clients/go/e2e_test.go b/clients/go/e2e_test.go new file mode 100644 index 00000000..3bce0341 --- /dev/null +++ b/clients/go/e2e_test.go @@ -0,0 +1,424 @@ +//go:build e2e + +package wavehouse + +import ( + "context" + "errors" + "fmt" + "net/http" + "os" + "slices" + "strings" + "testing" + "time" +) + +// e2eClient builds a Client pointing at the live WaveHouse instance. +// It reads WAVEHOUSE_URL (default http://localhost:8080) and the optional +// WAVEHOUSE_AUTH bearer token. The test is skipped when the server is +// unreachable — so `go test -tags e2e` degrades gracefully on a dev +// machine that isn't running the stack. +func e2eClient(t *testing.T) *Client { + t.Helper() + + base := os.Getenv("WAVEHOUSE_URL") + if base == "" { + base = "http://localhost:8080" + } + + cfg := Config{ + BaseURL: base, + Options: &ClientOptions{MaxRetries: 1}, + } + if tok := os.Getenv("WAVEHOUSE_AUTH"); tok != "" { + cfg.Auth = StaticToken(tok) + } + + // Probe the server before committing to the test. Bounded so a host that + // accepts the connection but never responds still yields the graceful skip. + probeCtx, cancel := context.WithTimeout(context.Background(), 3*time.Second) + defer cancel() + probe, err := http.NewRequestWithContext(probeCtx, "GET", base+"/v1/health", nil) + if err != nil { + t.Skipf("e2e: bad WAVEHOUSE_URL %q: %v", base, err) + } + resp, err := (&http.Client{Timeout: 3 * time.Second}).Do(probe) + if err != nil { + t.Skipf("e2e: server unreachable at %s: %v", base, err) + } + resp.Body.Close() + + return NewClient(cfg) +} + +// marker returns a unique string for the running test, useful for +// inserting distinguishable rows that won't collide across parallel runs. +func marker(t *testing.T) string { + t.Helper() + // Replace slashes in subtest names so it's a clean string value. + safe := strings.ReplaceAll(t.Name(), "/", "_") + return fmt.Sprintf("%s_%d", safe, time.Now().UnixNano()) +} + +// firstTable discovers a usable table from the schema list and returns its +// schema alongside the name. Many E2E tests need a real table to insert/query +// — this avoids hardcoding a name, and returning the schema avoids a second +// Schema.List whose result set might no longer contain the chosen table. +// Sorted so every run picks the same table (map iteration order is random). +func firstTable(t *testing.T, c *Client) (string, TableSchema) { + t.Helper() + schemas, err := c.Schema.List(context.Background()) + if err != nil { + t.Skipf("e2e: cannot list schemas (auth?): %v", err) + } + names := make([]string, 0, len(schemas)) + for name := range schemas { + names = append(names, name) + } + if len(names) == 0 { + t.Skip("e2e: no tables found — server has an empty schema") + } + slices.Sort(names) + return names[0], schemas[names[0]] +} + +// waitForRows polls the marker query until at least want rows are visible or +// the deadline expires. Ingestion is asynchronous — a fixed sleep fails on a +// loaded runner without any real defect. +func waitForRows(t *testing.T, c *Client, table, markerCol, mk string, want int) []map[string]any { + t.Helper() + deadline := time.Now().Add(10 * time.Second) + for { + page, err := c.From(table).Select(markerCol). + Where(markerCol, OpEq, mk). + Limit(max(want, 1)). + FetchUntyped(context.Background()) + if err != nil { + t.Fatalf("query for marker %q: %v", mk, err) + } + if len(page.Data) >= want || time.Now().After(deadline) { + return page.Data + } + time.Sleep(200 * time.Millisecond) + } +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +func TestE2E_HealthCheck(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + if err := c.Sys.Health(ctx); err != nil { + t.Fatalf("Health check failed: %v", err) + } +} + +func TestE2E_SchemaList(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + schemas, err := c.Schema.List(ctx) + if err != nil { + t.Fatalf("Schema.List failed: %v", err) + } + if len(schemas) == 0 { + t.Fatal("Schema.List returned zero tables — expected at least one") + } + // Quick sanity: every table should have columns. + for name, ts := range schemas { + if len(ts.Columns) == 0 { + t.Errorf("table %q has no columns", name) + } + } +} + +func TestE2E_InsertAndQuery(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table, ts := firstTable(t, c) + mk := marker(t) + + row, markerCol := buildMarkerRow(t, ts, mk) + + res, err := c.From(table).Insert(ctx, row) + if err != nil { + t.Fatalf("Insert into %s failed: %v", table, err) + } + if !res.OK { + t.Fatalf("Insert into %s: OK=false", table) + } + + rows := waitForRows(t, c, table, markerCol, mk, 1) + if len(rows) == 0 { + t.Fatal("Query returned zero rows — expected the inserted marker row") + } + got, _ := rows[0][markerCol].(string) + if got != mk { + t.Errorf("marker mismatch: want %q, got %q", mk, got) + } +} + +func TestE2E_BatchInsert(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table, ts := firstTable(t, c) + + mk := marker(t) + + // Build 3 rows, each with the same marker so we can count them. + rows := make([]map[string]any, 3) + markerCol := "" + for i := range rows { + rows[i], markerCol = buildMarkerRow(t, ts, mk) + } + + res, err := c.From(table).Insert(ctx, rows) + if err != nil { + t.Fatalf("Batch insert failed: %v", err) + } + if !res.OK { + t.Fatalf("Batch insert: OK=false") + } + + got := waitForRows(t, c, table, markerCol, mk, 3) + if len(got) < 3 { + t.Fatalf("expected >= 3 rows for marker %q, got %d", mk, len(got)) + } +} + +func TestE2E_QueryBuilder(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table, ts := firstTable(t, c) + + // Pick two columns for a minimal projection. + var cols []string + for _, col := range ts.Columns { + cols = append(cols, col.Name) + if len(cols) >= 2 { + break + } + } + if len(cols) == 0 { + t.Skipf("e2e: table %q has no columns", table) + } + + page, err := c.From(table). + Select(cols...). + OrderBy(cols[0], "asc"). + Limit(5). + FetchUntyped(ctx) + if err != nil { + t.Fatalf("QueryBuilder chain failed: %v", err) + } + // We can't assert exact data, but the chain should execute without error + // and return at most 5 rows. + if len(page.Data) > 5 { + t.Errorf("Limit(5) returned %d rows", len(page.Data)) + } +} + +func TestE2E_TypedFetch(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + table, _ := firstTable(t, c) + + q := c.From(table).SelectAll().Limit(3) + page, err := FetchTyped[map[string]any](ctx, q) + if err != nil { + t.Fatalf("FetchTyped failed: %v", err) + } + // If the table has data we should get rows; if it's empty that's still + // a valid result. The important thing is no error and correct type. + for i, row := range page.Data { + if row == nil { + t.Errorf("row %d is nil", i) + } + } +} + +func TestE2E_SQLQuery(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + rows, err := SQL[map[string]any](ctx, c, "SELECT 1 AS n") + if err != nil { + skipIfUnauthorized(t, err, "SQL query") + t.Fatalf("SQL query failed: %v", err) + } + if len(rows) != 1 { + t.Fatalf("expected 1 row, got %d", len(rows)) + } + // ClickHouse returns numbers as strings or floats depending on format; + // accept either. + n := rows[0]["n"] + switch v := n.(type) { + case float64: + if v != 1 { + t.Errorf("expected n=1, got %v", v) + } + case string: + if v != "1" { + t.Errorf("expected n=1, got %q", v) + } + default: + t.Errorf("unexpected type for n: %T = %v", n, n) + } +} + +func TestE2E_PolicyGetSet(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + pol, err := c.Policy.Get(ctx) + if err != nil { + skipIfUnauthorized(t, err, "Policy.Get") + t.Fatalf("Policy.Get failed: %v", err) + } + + // Round-trip: set the same policy back. + if err := c.Policy.Set(ctx, pol); err != nil { + t.Fatalf("Policy.Set (round-trip) failed: %v", err) + } + + // Read again and verify tables still match. + pol2, err := c.Policy.Get(ctx) + if err != nil { + t.Fatalf("Policy.Get (after set) failed: %v", err) + } + if len(pol2.Tables) != len(pol.Tables) { + t.Errorf("policy table count changed: %d -> %d", len(pol.Tables), len(pol2.Tables)) + } +} + +func TestE2E_PipesCRUD(t *testing.T) { + c := e2eClient(t) + ctx := context.Background() + + pipeName := fmt.Sprintf("e2e_test_%d", time.Now().UnixNano()) + + // Create + def := PipeDef{ + SQL: "SELECT 1 AS ok", + Description: "E2E test pipe — safe to delete", + } + if err := c.Pipes.Set(ctx, pipeName, def); err != nil { + skipIfUnauthorized(t, err, "Pipes.Set") + t.Fatalf("Pipes.Set (create) failed: %v", err) + } + + // Cleanup: always attempt delete so we don't litter. + t.Cleanup(func() { + _ = c.Pipes.Delete(context.Background(), pipeName) + }) + + // Get + pipe, err := c.Pipes.Get(ctx, pipeName) + if err != nil { + t.Fatalf("Pipes.Get failed: %v", err) + } + if pipe.SQL != def.SQL { + t.Errorf("pipe SQL mismatch: want %q, got %q", def.SQL, pipe.SQL) + } + + // List — verify it appears + pipes, err := c.Pipes.List(ctx) + if err != nil { + t.Fatalf("Pipes.List failed: %v", err) + } + found := false + for _, p := range pipes { + if p.Name == pipeName { + found = true + break + } + } + if !found { + t.Errorf("Pipes.List: created pipe %q not found in list of %d pipes", pipeName, len(pipes)) + } + + // Delete + if err := c.Pipes.Delete(ctx, pipeName); err != nil { + t.Fatalf("Pipes.Delete failed: %v", err) + } + + // Verify gone — Get should fail. + _, err = c.Pipes.Get(ctx, pipeName) + if err == nil { + t.Error("Pipes.Get after delete: expected error, got nil") + } +} + +// --------------------------------------------------------------------------- +// Helpers +// --------------------------------------------------------------------------- + +// buildMarkerRow constructs a minimal valid row for the table, injecting the +// marker into the first non-default string column and using sensible values +// for other required columns. It returns the row and the marker column, so +// callers query back the exact column the marker went into. +func buildMarkerRow(t *testing.T, ts TableSchema, mk string) (map[string]any, string) { + t.Helper() + row := make(map[string]any) + markerCol := "" + markerSet := false + for _, col := range ts.Columns { + if col.HasDefault { + continue // let the server fill defaults + } + ct := strings.ToLower(col.Type) + switch { + case strings.Contains(ct, "array(") || strings.Contains(ct, "map(") || + strings.Contains(ct, "tuple(") || strings.Contains(ct, "nested("): + // Container types must be gated BEFORE the substring cases below — + // "array(string)" contains "string" and would otherwise get a + // scalar marker injected into an array column. + t.Skipf("e2e: table %q requires container column %q of type %q", ts.Name, col.Name, col.Type) + case !markerSet && strings.Contains(ct, "string"): + row[col.Name] = mk + markerCol = col.Name + markerSet = true + case strings.Contains(ct, "string"): + row[col.Name] = "e2e" + case strings.Contains(ct, "int"): + row[col.Name] = 0 + case strings.Contains(ct, "float") || strings.Contains(ct, "decimal"): + row[col.Name] = 0.0 + case strings.Contains(ct, "date") || strings.Contains(ct, "datetime"): + row[col.Name] = time.Now().UTC().Format(time.RFC3339) + case strings.Contains(ct, "bool"): + row[col.Name] = false + default: + // No safe synthetic value for this type (UUID, IPv6, ...) — an + // empty string would make the insert fail with a type error that + // looks like an SDK defect. + t.Skipf("e2e: table %q requires column %q of unsupported type %q", ts.Name, col.Name, col.Type) + } + } + if !markerSet { + t.Skipf("e2e: table %q has no non-default string column for marker", ts.Name) + } + return row, markerCol +} + +// skipIfUnauthorized skips the test when err indicates a 401 or 403, +// meaning the operation requires admin auth the current token lacks. +func skipIfUnauthorized(t *testing.T, err error, op string) { + t.Helper() + if isHTTPStatus(err, 401) || isHTTPStatus(err, 403) { + t.Skipf("%s requires admin auth, skipping", op) + } +} + +// isHTTPStatus checks whether err wraps a wavehouse.Error with the given status. +func isHTTPStatus(err error, status int) bool { + var e *Error + if errors.As(err, &e) { + return e.Status == status + } + return false +} diff --git a/clients/go/errors.go b/clients/go/errors.go new file mode 100644 index 00000000..8ec3b2e8 --- /dev/null +++ b/clients/go/errors.go @@ -0,0 +1,78 @@ +package wavehouse + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "net/http" +) + +// Error is the structured error returned by all SDK operations. Use +// [errors.As] to extract it from wrapped errors. +type Error struct { + // Status is the HTTP status code (0 for network/abort errors). + Status int `json:"status"` + // Code is a machine-readable error code (e.g. "HTTP_400", "NETWORK_ERROR", "ABORTED"). + Code string `json:"code"` + // Message is a human-readable description. + Message string `json:"message"` + // Details contains the full parsed error body, if available. + Details map[string]any `json:"details,omitempty"` + // Retryable indicates whether the request can be retried. + Retryable bool `json:"retryable"` +} + +func (e *Error) Error() string { + if e.Status > 0 { + return fmt.Sprintf("wavehouse: %s (%d): %s", e.Code, e.Status, e.Message) + } + return fmt.Sprintf("wavehouse: %s: %s", e.Code, e.Message) +} + +// IsRetryable reports whether err wraps a retryable [*Error]. +func IsRetryable(err error) bool { + var e *Error + return errors.As(err, &e) && e.Retryable +} + +// parseErrorResponse creates an Error from an HTTP response. +func parseErrorResponse(res *http.Response) *Error { + var body map[string]any + if res.Body != nil { + raw, _ := io.ReadAll(io.LimitReader(res.Body, 1<<20)) // cap at 1 MiB + _ = json.Unmarshal(raw, &body) + } + + var msg string + if s, ok := body["error"].(string); ok { + msg = s + } else if s, ok := body["message"].(string); ok { + msg = s + } else { + msg = http.StatusText(res.StatusCode) + } + + retryable := res.StatusCode >= 500 || res.StatusCode == http.StatusTooManyRequests + return &Error{ + Status: res.StatusCode, + Code: fmt.Sprintf("HTTP_%d", res.StatusCode), + Message: msg, + Details: body, + Retryable: retryable, + } +} + +// networkError creates an Error from a transport-level failure. +func networkError(cause error) *Error { + msg := "unknown network error" + if cause != nil { + msg = cause.Error() + } + return &Error{ + Status: 0, + Code: "NETWORK_ERROR", + Message: msg, + Retryable: true, + } +} diff --git a/clients/go/errors_test.go b/clients/go/errors_test.go new file mode 100644 index 00000000..160cb5c4 --- /dev/null +++ b/clients/go/errors_test.go @@ -0,0 +1,155 @@ +package wavehouse + +import ( + "errors" + "io" + "net/http" + "strings" + "testing" +) + +func TestParseErrorResponse(t *testing.T) { + tests := []struct { + name string + status int + body string + wantMsg string + wantCode string + wantRetry bool + nilDetails bool + }{ + { + name: "JSONError", + status: 404, + body: `{"error":"unknown table: foo"}`, + wantMsg: "unknown table: foo", + wantCode: "HTTP_404", + }, + { + name: "MessageField", + status: 400, + body: `{"message":"bad request"}`, + wantMsg: "bad request", + }, + { + name: "FallsBackToStatusText", + status: 500, + body: `{"code":123}`, + wantMsg: "Internal Server Error", + wantRetry: true, + }, + { + name: "NonJSONBody", + status: 502, + body: "plain text", + wantMsg: "Bad Gateway", + wantRetry: true, + nilDetails: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + res := &http.Response{ + StatusCode: tt.status, + Body: io.NopCloser(strings.NewReader(tt.body)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Status != tt.status { + t.Fatalf("want status %d, got %d", tt.status, e.Status) + } + if tt.wantCode != "" && e.Code != tt.wantCode { + t.Fatalf("want code %s, got %s", tt.wantCode, e.Code) + } + if e.Message != tt.wantMsg { + t.Fatalf("want message %q, got %q", tt.wantMsg, e.Message) + } + if e.Retryable != tt.wantRetry { + t.Fatalf("want retryable=%v, got %v", tt.wantRetry, e.Retryable) + } + if tt.nilDetails && e.Details != nil { + t.Fatal("details should be nil") + } + }) + } +} + +func TestParseErrorResponse_5xxRetryable(t *testing.T) { + tests := []struct { + name string + status int + retryable bool + }{ + {"BadRequest", 400, false}, + {"Forbidden", 403, false}, + {"TooManyRequests", 429, true}, + {"InternalServerError", 500, true}, + {"ServiceUnavailable", 503, true}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + res := &http.Response{ + StatusCode: tt.status, + Body: io.NopCloser(strings.NewReader(`{"error":"test"}`)), + Header: http.Header{}, + } + e := parseErrorResponse(res) + if e.Retryable != tt.retryable { + t.Errorf("status %d: want retryable=%v, got %v", tt.status, tt.retryable, e.Retryable) + } + }) + } +} + +func TestNetworkError(t *testing.T) { + e := networkError(errors.New("connection refused")) + if e.Code != "NETWORK_ERROR" { + t.Fatalf("want NETWORK_ERROR, got %s", e.Code) + } + if e.Message != "connection refused" { + t.Fatalf("want 'connection refused', got %s", e.Message) + } + if !e.Retryable { + t.Fatal("network errors should be retryable") + } + if e.Status != 0 { + t.Fatalf("want status 0, got %d", e.Status) + } +} + +func TestError_ErrorMethod(t *testing.T) { + e := &Error{Status: 404, Code: "HTTP_404", Message: "not found"} + got := e.Error() + if !strings.Contains(got, "HTTP_404") || !strings.Contains(got, "not found") { + t.Fatalf("unexpected Error() output: %s", got) + } + + e2 := &Error{Status: 0, Code: "NETWORK_ERROR", Message: "timeout"} + got2 := e2.Error() + if !strings.Contains(got2, "NETWORK_ERROR") { + t.Fatalf("unexpected Error() output: %s", got2) + } +} + +func TestIsRetryable(t *testing.T) { + if !IsRetryable(&Error{Retryable: true}) { + t.Fatal("want true for retryable error") + } + if IsRetryable(&Error{Retryable: false}) { + t.Fatal("want false for non-retryable error") + } + if IsRetryable(errors.New("plain error")) { + t.Fatal("want false for non-wavehouse error") + } +} + +func TestErrorsAs(t *testing.T) { + err := error(&Error{Status: 403, Code: "HTTP_403", Message: "forbidden"}) + var e *Error + if !errors.As(err, &e) { + t.Fatal("errors.As should find *Error") + } + if e.Status != 403 { + t.Fatalf("want 403, got %d", e.Status) + } +} diff --git a/clients/go/example_test.go b/clients/go/example_test.go new file mode 100644 index 00000000..b8e566c7 --- /dev/null +++ b/clients/go/example_test.go @@ -0,0 +1,75 @@ +package wavehouse_test + +import ( + "context" + "fmt" + "log" + + wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +) + +// ExampleNewClient demonstrates creating an unauthenticated client and +// performing a health check. The Output assertion is omitted because the +// example needs a running server. +func ExampleNewClient() { + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + }) + + // Health check — returns nil when the server is reachable. + if err := client.Sys.Health(context.Background()); err != nil { + log.Fatal(err) + } +} + +func ExampleNewClient_withAuth() { + _ = wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("my-jwt-token"), + }) +} + +func ExampleClient_From() { + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + }) + + // Insert a row. + _, _ = client.From("clicks").Insert(context.Background(), map[string]any{ + "page": "/home", + "button": "cta", + }) + + // Query with the builder. + page, err := client.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + OrderBy("page", "asc"). + Limit(10). + FetchUntyped(context.Background()) + if err != nil { + log.Fatal(err) + } + + for _, row := range page.Data { + fmt.Println(row["page"]) + } +} + +func ExampleSQL() { + client := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("admin-token"), + }) + + rows, err := wavehouse.SQL[map[string]any]( + context.Background(), client, + "SELECT page, count() as views FROM clicks GROUP BY page LIMIT 5", + ) + if err != nil { + log.Fatal(err) + } + for _, row := range rows { + fmt.Println(row["page"], row["views"]) + } +} diff --git a/clients/go/go.mod b/clients/go/go.mod new file mode 100644 index 00000000..b78eaa50 --- /dev/null +++ b/clients/go/go.mod @@ -0,0 +1,6 @@ +module github.com/Wave-RF/WaveHouse/clients/go + +// Library floor, deliberately lower than the server's pinned toolchain: +// the newest things this module uses are range-over-int and math/rand/v2 +// (Go 1.22). Keep it a supported-releases floor, not a patch pin. +go 1.24 diff --git a/clients/go/http.go b/clients/go/http.go new file mode 100644 index 00000000..eaab0e94 --- /dev/null +++ b/clients/go/http.go @@ -0,0 +1,224 @@ +package wavehouse + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "math" + "math/rand/v2" + "net/http" + "net/url" + "strconv" + "time" +) + +var errAborted = &Error{Status: 0, Code: "ABORTED", Message: "Request aborted", Retryable: false} + +// maxRetryAfter caps server-supplied Retry-After delays so a hostile or +// misconfigured server can't park the calling goroutine for hours. +const maxRetryAfter = 30 * time.Second + +// httpContext carries per-client state needed by every request. +type httpContext struct { + baseURL string + auth func(ctx context.Context) (string, error) + maxRetries int + httpClient *http.Client + headers map[string]string +} + +// applyConfiguredHeaders writes the client's configured headers onto a request. +// Call it *before* the SDK sets its own headers: Set replaces, so whatever the +// SDK writes afterwards wins a collision. http.Header canonicalizes names, so +// "x-tenant" and "X-Tenant" are the same entry. +func applyConfiguredHeaders(h http.Header, configured map[string]string) { + for k, v := range configured { + h.Set(k, v) + } +} + +// requestOptions describes a single HTTP request. +type requestOptions struct { + method string + path string + body any // JSON-serialized if non-nil + rawBody string // sent verbatim if non-empty (takes precedence over body) + contentType string // overrides Content-Type (default "application/json") + params url.Values +} + +// doRequest is the internal fetch wrapper with auth, retry, and backoff. +// It decodes the response body into dst (unless dst is nil). +func doRequest(ctx context.Context, hctx httpContext, opts requestOptions, dst any) error { + reqURL := buildURL(hctx.baseURL, opts.path, opts.params) + ct := opts.contentType + if ct == "" { + ct = "application/json" + } + + // Serialize body once so every retry sends identical bytes. + var bodyBytes []byte + if opts.rawBody != "" { + bodyBytes = []byte(opts.rawBody) + } else if opts.body != nil { + var err error + bodyBytes, err = json.Marshal(opts.body) + if err != nil { + return fmt.Errorf("wavehouse: marshal request body: %w", err) + } + } + + // Resolve auth once per request (not per attempt). + var authHeader string + if hctx.auth != nil { + token, err := hctx.auth(ctx) + if err != nil { + return fmt.Errorf("wavehouse: auth provider: %w", err) + } + if token != "" { + authHeader = "Bearer " + token + } + } + + var lastErr error + maxAttempts := hctx.maxRetries + 1 + + // Retries all methods including POST. For /v1/ingest, at-least-once delivery + // is the documented contract; dedup is the server-side safety net. + for attempt := range maxAttempts { + var bodyReader io.Reader + if bodyBytes != nil { + bodyReader = bytes.NewReader(bodyBytes) + } + + req, err := http.NewRequestWithContext(ctx, opts.method, reqURL, bodyReader) + if err != nil { + return fmt.Errorf("wavehouse: build request: %w", err) + } + applyConfiguredHeaders(req.Header, hctx.headers) + req.Header.Set("Content-Type", ct) + req.Header.Set("Accept", "application/json") + if authHeader != "" { + req.Header.Set("Authorization", authHeader) + } + + res, err := hctx.httpClient.Do(req) + if err != nil { + // Context cancellation — return immediately, no retry. + if ctx.Err() != nil { + return errAborted + } + lastErr = networkError(err) + if attempt < maxAttempts-1 { + if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { + return errAborted + } + } + continue + } + + if res.StatusCode >= 200 && res.StatusCode < 300 { + defer func() { _ = res.Body.Close() }() + if dst == nil { + _, _ = io.Copy(io.Discard, res.Body) + return nil + } + raw, readErr := io.ReadAll(res.Body) + if readErr != nil { + return networkError(readErr) + } + if len(raw) == 0 { + return nil + } + if err := json.Unmarshal(raw, dst); err != nil { + return &Error{ + Status: 0, + Code: "NETWORK_ERROR", + Message: fmt.Errorf("decode response: %w", err).Error(), + Retryable: false, + } + } + return nil + } + + apiErr := parseErrorResponse(res) + _ = res.Body.Close() + + // 503/429 with Retry-After: wait the specified duration (capped). + if res.StatusCode == http.StatusServiceUnavailable || res.StatusCode == http.StatusTooManyRequests { + if ra := res.Header.Get("Retry-After"); ra != "" && attempt < maxAttempts-1 { + if sleepErr := sleepWithContext(ctx, retryAfterDelay(ra, attempt)); sleepErr != nil { + return errAborted + } + lastErr = apiErr + continue + } + } + + // Retryable server errors (5xx). + if apiErr.Retryable && attempt < maxAttempts-1 { + if sleepErr := sleepWithContext(ctx, backoff(attempt)); sleepErr != nil { + return errAborted + } + lastErr = apiErr + continue + } + + return apiErr + } + + return lastErr +} + +func buildURL(base, path string, params url.Values) string { + u := base + path + if len(params) > 0 { + u += "?" + params.Encode() + } + return u +} + +// retryAfterDelay resolves a Retry-After header (delta-seconds or HTTP-date) +// into a wait, clamped to maxRetryAfter. An unparseable header falls back to +// the ordinary backoff for this attempt, not the maximum. +func retryAfterDelay(ra string, attempt int) time.Duration { + delay := backoff(attempt) + if secs, err := strconv.Atoi(ra); err == nil && secs > 0 { + // Compare before converting: time.Duration(secs) * time.Second wraps + // negative past ~9.2e9 seconds, and min() below would then pick the + // negative value, firing the retry timer instantly. + if secs > int(maxRetryAfter/time.Second) { + return maxRetryAfter + } + delay = time.Duration(secs) * time.Second + } else if parsed, err := http.ParseTime(ra); err == nil { + if d := time.Until(parsed); d > 0 { + delay = d + } + } + return min(delay, maxRetryAfter) +} + +func backoff(attempt int) time.Duration { + ms := 1000 * math.Pow(2, float64(attempt)) + // ±20% jitter so clients failing at the same moment don't retry in + // lockstep; capped after jitter so the documented 30s max holds. + ms *= 0.8 + 0.4*rand.Float64() //nolint:gosec // retry jitter, not cryptographic + return time.Duration(min(ms, 30000)) * time.Millisecond +} + +func sleepWithContext(ctx context.Context, d time.Duration) error { + if ctx.Err() != nil { + return ctx.Err() + } + t := time.NewTimer(d) + defer t.Stop() + select { + case <-t.C: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} diff --git a/clients/go/http_test.go b/clients/go/http_test.go new file mode 100644 index 00000000..52212c16 --- /dev/null +++ b/clients/go/http_test.go @@ -0,0 +1,432 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "errors" + "io" + "net/http" + "net/http/httptest" + "sync/atomic" + "testing" + "time" +) + +// errIs checks if err wraps a *Error with the given code. +func errIs(err error, code string) bool { + var e *Error + if errors.As(err, &e) { + return e.Code == code + } + return false +} + +func testCtx(t *testing.T, handler http.Handler) httpContext { + t.Helper() + srv := httptest.NewServer(handler) + t.Cleanup(srv.Close) + return httpContext{ + baseURL: srv.URL, + maxRetries: 0, + httpClient: srv.Client(), + } +} + +func TestDoRequest_SuccessfulGET(t *testing.T) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + })) + + var result map[string]string + err := doRequest(context.Background(), hctx, requestOptions{ + method: "GET", + path: "/health", + }, &result) + if err != nil { + t.Fatal(err) + } + if result["status"] != "ok" { + t.Fatalf("want ok, got %v", result) + } +} + +func TestDoRequest_POSTWithBody(t *testing.T) { + var gotBody map[string]string + var gotCT string + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotCT = r.Header.Get("Content-Type") + _ = json.NewDecoder(r.Body).Decode(&gotBody) + w.WriteHeader(200) + })) + + err := doRequest(context.Background(), hctx, requestOptions{ + method: "POST", + path: "/v1/ingest", + body: map[string]string{"page": "/home"}, + }, nil) + if err != nil { + t.Fatal(err) + } + if gotCT != "application/json" { + t.Fatalf("want application/json, got %s", gotCT) + } + if gotBody["page"] != "/home" { + t.Fatalf("want /home, got %v", gotBody) + } +} + +func TestDoRequest_RawBody(t *testing.T) { + var gotBody string + var gotCT string + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotCT = r.Header.Get("Content-Type") + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) + _ = json.NewEncoder(w).Encode(map[string]int{"total": 1}) + })) + + err := doRequest(context.Background(), hctx, requestOptions{ + method: "POST", + path: "/v1/ingest", + rawBody: `{"page":"/a"}`, + contentType: "application/x-ndjson", + }, nil) + if err != nil { + t.Fatal(err) + } + if gotCT != "application/x-ndjson" { + t.Fatalf("want ndjson content type, got %s", gotCT) + } + if gotBody != `{"page":"/a"}` { + t.Fatalf("want raw body, got %s", gotBody) + } +} + +func TestDoRequest_AuthInjection(t *testing.T) { + var gotAuth string + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotAuth = r.Header.Get("Authorization") + w.WriteHeader(200) + })) + hctx.auth = StaticToken("my-token") + + err := doRequest(context.Background(), hctx, requestOptions{ + method: "GET", + path: "/v1/ops/schema", + }, nil) + if err != nil { + t.Fatal(err) + } + if gotAuth != "Bearer my-token" { + t.Fatalf("want 'Bearer my-token', got %s", gotAuth) + } +} + +func TestDoRequest_4xxNotRetried(t *testing.T) { + var count atomic.Int32 + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + count.Add(1) + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(404) + _ = json.NewEncoder(w).Encode(map[string]string{"error": "not found"}) + })) + hctx.maxRetries = 2 + + err := doRequest(context.Background(), hctx, requestOptions{ + method: "GET", + path: "/v1/ops/schema", + }, nil) + + if !errIs(err, "HTTP_404") { + t.Fatalf("want HTTP_404 error, got %v", err) + } + if count.Load() != 1 { + t.Fatalf("4xx should not retry, got %d attempts", count.Load()) + } +} + +func TestDoRequest_5xxRetried(t *testing.T) { + var count atomic.Int32 + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + n := count.Add(1) + if n < 3 { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(500) + _ = json.NewEncoder(w).Encode(map[string]string{"error": "internal"}) + return + } + _ = json.NewEncoder(w).Encode(map[string]string{"ok": "true"}) + })) + hctx.maxRetries = 2 + + var result map[string]string + err := doRequest(context.Background(), hctx, requestOptions{ + method: "GET", + path: "/health", + }, &result) + if err != nil { + t.Fatalf("want success after retries, got %v", err) + } + if count.Load() != 3 { + t.Fatalf("want 3 attempts, got %d", count.Load()) + } +} + +func TestDoRequest_AbortedOnCancel(t *testing.T) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + time.Sleep(5 * time.Second) + })) + + ctx, cancel := context.WithCancel(context.Background()) + cancel() // cancel immediately + + err := doRequest(ctx, hctx, requestOptions{ + method: "GET", + path: "/health", + }, nil) + + if !errIs(err, "ABORTED") { + t.Fatalf("want ABORTED, got %v", err) + } +} + +func TestDoRequest_EmptyResponse(t *testing.T) { + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.WriteHeader(200) + })) + + var result map[string]string + err := doRequest(context.Background(), hctx, requestOptions{ + method: "POST", + path: "/v1/ops/schema/refresh", + }, &result) + if err != nil { + t.Fatal(err) + } + // Empty body = no decode, result stays zero value. + if result != nil { + t.Fatalf("want nil, got %v", result) + } +} + +func TestBackoff(t *testing.T) { + tests := []struct { + name string + attempt int + base time.Duration + }{ + {"Attempt0", 0, 1 * time.Second}, + {"Attempt1", 1, 2 * time.Second}, + {"Attempt2", 2, 4 * time.Second}, + {"Attempt3", 3, 8 * time.Second}, + {"CappedAt30s", 10, 30 * time.Second}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := backoff(tt.attempt) + if tt.name == "CappedAt30s" { + // The cap is applied *after* jitter, so this is exact — a ±20% + // window here would also accept capping before the jitter, + // which lets the documented 30s max drift to 36s. + if got != 30*time.Second { + t.Errorf("backoff(%d) = %v, want exactly 30s", tt.attempt, got) + } + return + } + // backoff applies ±20% jitter around the exponential base. + lo := time.Duration(float64(tt.base) * 0.8) + hi := time.Duration(float64(tt.base) * 1.2) + if got < lo || got > hi { + t.Errorf("backoff(%d) = %v, want within [%v, %v]", tt.attempt, got, lo, hi) + } + }) + } +} + +func TestRetryAfterDelay(t *testing.T) { + tests := []struct { + name string + ra string + want time.Duration + }{ + {"DeltaSeconds", "5", 5 * time.Second}, + {"ClampedToMax", "3600", maxRetryAfter}, + // time.Duration(secs) * time.Second wraps negative past ~9.2e9s; an + // unguarded min() then picks the negative and retries instantly. + {"OverflowClamped", "10000000000", maxRetryAfter}, + {"MaxIntClamped", "9223372036854775807", maxRetryAfter}, + {"HTTPDateFuture", time.Now().Add(10 * time.Second).UTC().Format(http.TimeFormat), 0}, // range-checked below + {"Garbage", "not-a-delay", 0}, // range-checked below + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := retryAfterDelay(tt.ra, 0) + switch tt.name { + case "HTTPDateFuture": + // Lower bound well clear of the backoff(0) fallback (~1s), so a + // broken HTTP-date branch can't pass by falling through to it. + if got < 5*time.Second || got > 10*time.Second { + t.Fatalf("want ~10s, got %v", got) + } + case "Garbage": + // Falls back to backoff(0): 1s ±20% jitter. + if got < 800*time.Millisecond || got > 1200*time.Millisecond { + t.Fatalf("want backoff(0) fallback, got %v", got) + } + default: + if got != tt.want { + t.Fatalf("want %v, got %v", tt.want, got) + } + } + }) + } +} + +func TestDoRequest_429RetriesWithRetryAfter(t *testing.T) { + var calls atomic.Int64 + hctx := testCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + if calls.Add(1) == 1 { + w.Header().Set("Retry-After", "1") + w.WriteHeader(http.StatusTooManyRequests) + return + } + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + })) + hctx.maxRetries = 1 + + start := time.Now() + var result map[string]string + err := doRequest(context.Background(), hctx, requestOptions{method: "GET", path: "/x"}, &result) + if err != nil { + t.Fatalf("want success after 429 retry, got %v", err) + } + if got := calls.Load(); got != 2 { + t.Fatalf("want 2 attempts, got %d", got) + } + if elapsed := time.Since(start); elapsed < 900*time.Millisecond { + t.Fatalf("Retry-After: 1 not honored — retried after only %v", elapsed) + } +} + +// A BaseURL carrying a path prefix must survive on both transports — the bug +// #428 fixed in the TS client, which Go avoids by concatenating rather than +// resolving. Guards against a future switch to url.JoinPath/ResolveReference. +func TestBaseURLPathPrefixIsPreserved(t *testing.T) { + mux := http.NewServeMux() + mux.HandleFunc("/api/warehouse/v1/query", func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(map[string]string{"status": "ok"}) + }) + srv := httptest.NewServer(mux) // anything off-prefix 404s + t.Cleanup(srv.Close) + + for _, base := range []string{srv.URL + "/api/warehouse", srv.URL + "/api/warehouse/"} { + client := NewClient(Config{BaseURL: base, Options: &ClientOptions{}, HTTPClient: srv.Client()}) + + var result map[string]string + if err := doRequest(context.Background(), client.ctx, requestOptions{ + method: "POST", + path: "/v1/query", + }, &result); err != nil { + t.Fatalf("base %q: %v", base, err) + } + if result["status"] != "ok" { + t.Fatalf("base %q: want ok, got %v", base, result) + } + } +} + +// TestConfiguredHeadersOnRESTRequests: ClientOptions.Headers apply to every +// REST call, are matched case-insensitively, and always lose to the SDK's own +// headers rather than appending alongside them. +func TestConfiguredHeadersOnRESTRequests(t *testing.T) { + tests := []struct { + name string + configured map[string]string + auth func(context.Context) (string, error) + header string + want string + }{ + { + name: "custom header is forwarded", + configured: map[string]string{"X-Operator-Key": "op-secret"}, + header: "X-Operator-Key", + want: "op-secret", + }, + { + name: "name matching is case-insensitive", + configured: map[string]string{"x-tenant-id": "acme"}, + header: "X-Tenant-Id", + want: "acme", + }, + { + name: "SDK Accept outranks a configured one", + configured: map[string]string{"Accept": "text/plain"}, + header: "Accept", + want: "application/json", + }, + { + name: "SDK Authorization outranks a configured one", + configured: map[string]string{"Authorization": "Bearer configured"}, + auth: StaticToken("real-token"), + header: "Authorization", + want: "Bearer real-token", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + var got http.Header + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Clone() + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `[]`) + })) + defer srv.Close() + + client := NewClient(Config{ + BaseURL: srv.URL, + Auth: tc.auth, + HTTPClient: srv.Client(), + Options: &ClientOptions{Headers: tc.configured}, + }) + if _, err := client.Schema.List(context.Background()); err != nil { + t.Fatalf("schema list: %v", err) + } + if v := got.Values(tc.header); len(v) != 1 { + t.Fatalf("want exactly one %s header, got %v", tc.header, v) + } + if v := got.Get(tc.header); v != tc.want { + t.Fatalf("want %s: %q, got %q", tc.header, tc.want, v) + } + }) + } +} + +// TestConfiguredHeadersAreCopied: mutating the caller's map after NewClient +// must not change what later requests send. +func TestConfiguredHeadersAreCopied(t *testing.T) { + var got http.Header + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + got = r.Header.Clone() + w.Header().Set("Content-Type", "application/json") + _, _ = io.WriteString(w, `[]`) + })) + defer srv.Close() + + headers := map[string]string{"X-Tenant-Id": "acme"} + client := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{Headers: headers}, + }) + headers["X-Tenant-Id"] = "attacker" + delete(headers, "X-Tenant-Id") + + if _, err := client.Schema.List(context.Background()); err != nil { + t.Fatalf("schema list: %v", err) + } + if v := got.Get("X-Tenant-Id"); v != "acme" { + t.Fatalf("want the value captured at construction, got %q", v) + } +} diff --git a/clients/go/live_query.go b/clients/go/live_query.go new file mode 100644 index 00000000..d47ce618 --- /dev/null +++ b/clients/go/live_query.go @@ -0,0 +1,161 @@ +package wavehouse + +import ( + "context" + "sync" + "time" +) + +// LiveQueryHandle controls a live query that combines historical backfill +// with a real-time stream. +type LiveQueryHandle struct { + stream *StreamController + cancel context.CancelFunc + unsub func() + closeOnce sync.Once + + mu sync.Mutex + buffer []StreamEvent + buffering bool + closed bool +} + +// newLiveQuery starts a live query: opens the stream immediately, fetches +// historical data, deduplicates buffered events, then goes live. +func newLiveQuery( + stream *StreamController, + fetchFn func(ctx context.Context) ([]map[string]any, error), + sub *StreamSubscriber, +) *LiveQueryHandle { + ctx, cancel := context.WithCancel(context.Background()) //nolint:gosec // cancel is called in Close() + lq := &LiveQueryHandle{ + stream: stream, + cancel: cancel, + buffering: true, + } + + // Step 1: Subscribe to live events and buffer them. User callbacks are + // invoked outside lq.mu so a subscriber may call Close() without + // deadlocking. + lq.unsub = stream.Subscribe(&StreamSubscriber{ + Next: func(event StreamEvent) { + lq.mu.Lock() + if lq.closed { + lq.mu.Unlock() + return + } + if lq.buffering { + lq.buffer = append(lq.buffer, event) + lq.mu.Unlock() + return + } + lq.mu.Unlock() + if sub.Next != nil { + sub.Next(event) + } + }, + Status: func(s StreamStatus) { + if !lq.isClosed() && sub.Status != nil { + sub.Status(s) + } + }, + Error: func(err error) { + if !lq.isClosed() && sub.Error != nil { + sub.Error(err) + } + }, + }) + + // Step 2–5: Fetch historical and flush. + go func() { + rows, err := fetchFn(ctx) + if ctx.Err() != nil || lq.isClosed() { + return + } + + // Step 3: Deliver initial snapshot. + if sub.Initial != nil { + sub.Initial(rows, err) + } + + if err != nil { + lq.mu.Lock() + lq.buffering = false + lq.buffer = nil + lq.mu.Unlock() + return + } + + // Step 4: Dedup bound — the maximum backfilled timestamp, compared as + // parsed times. OrderBy(..., "desc") makes the *last* row the oldest, + // and RFC3339 strings with varying fractional digits don't sort + // lexically, so neither "last row" nor raw string compare is safe. + var lastTS time.Time + for _, row := range rows { + if s, ok := row["received_timestamp"].(string); ok { + if ts, perr := time.Parse(time.RFC3339Nano, s); perr == nil && ts.After(lastTS) { + lastTS = ts + } + } + } + + // Step 5: Flush buffered events. buffering stays true until the + // buffer is provably empty under the lock — prevents concurrent + // sub.Next calls and preserves delivery order. + for { + lq.mu.Lock() + if lq.closed { + lq.mu.Unlock() + return + } + pending := lq.buffer + lq.buffer = nil + if len(pending) == 0 { + lq.buffering = false + lq.mu.Unlock() + break + } + lq.mu.Unlock() + + for _, event := range pending { + if lq.isClosed() { + return + } + // Skip events already delivered in the backfill. + // Sub-millisecond received_timestamp precision makes + // boundary collisions rare. + if !lastTS.IsZero() { + if ts, perr := time.Parse(time.RFC3339Nano, event.Timestamp); perr == nil && !ts.After(lastTS) { + continue + } + } + if sub.Next != nil { + sub.Next(event) + } + } + } + }() + + return lq +} + +func (lq *LiveQueryHandle) isClosed() bool { + lq.mu.Lock() + defer lq.mu.Unlock() + return lq.closed +} + +// Close shuts down the live query and the underlying stream. The close state +// is applied synchronously: no new subscriber callbacks start after Close +// returns (a callback already in flight may still complete). +func (lq *LiveQueryHandle) Close() { + lq.closeOnce.Do(func() { + lq.mu.Lock() + lq.closed = true + lq.buffer = nil + lq.mu.Unlock() + lq.unsub() + lq.cancel() + lq.stream.Close() + }) +} diff --git a/clients/go/live_query_test.go b/clients/go/live_query_test.go new file mode 100644 index 00000000..f703a68e --- /dev/null +++ b/clients/go/live_query_test.go @@ -0,0 +1,173 @@ +package wavehouse + +import ( + "context" + "errors" + "sync/atomic" + "testing" + "time" +) + +// bareStream builds a StreamController that never dials anything — events are +// injected with emitEvent, exactly how the run loop feeds real ones. +func bareStream() *StreamController { + return &StreamController{ + status: StatusLive, + eventCh: make(chan StreamEvent, 16), + done: make(chan struct{}), + cancel: func() {}, + } +} + +func liveEvent(ts string) StreamEvent { + return StreamEvent{Table: "clicks", Timestamp: ts, Data: map[string]any{"page": "/home"}} +} + +func awaitInitial(t *testing.T, ch <-chan []map[string]any) []map[string]any { + t.Helper() + select { + case rows := <-ch: + return rows + case <-time.After(5 * time.Second): + t.Fatal("Initial never fired") + return nil + } +} + +func TestLiveQuery_InitialThenLiveWithDedup(t *testing.T) { + sc := bareStream() + fetched := []map[string]any{ + {"page": "/a", "received_timestamp": "2026-01-01T00:00:05Z"}, + // Descending order: the max timestamp is NOT the last row. + {"page": "/b", "received_timestamp": "2026-01-01T00:00:03Z"}, + } + gate := make(chan struct{}) + initialCh := make(chan []map[string]any, 1) + nextCh := make(chan StreamEvent, 8) + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { + <-gate + return fetched, nil + }, + &StreamSubscriber{ + Initial: func(rows []map[string]any, err error) { + if err != nil { + t.Errorf("Initial err: %v", err) + } + initialCh <- rows + }, + Next: func(e StreamEvent) { nextCh <- e }, + }) + defer lq.Close() + + // Buffered while the backfill is in flight; deduped against the *max* + // backfilled timestamp (5Z despite descending order) on flush. + sc.emitEvent(liveEvent("2026-01-01T00:00:04Z")) // ≤ max backfill → skipped + sc.emitEvent(liveEvent("2026-01-01T00:00:06Z")) // newer → delivered + close(gate) + + rows := awaitInitial(t, initialCh) + if len(rows) != 2 { + t.Fatalf("want 2 backfill rows, got %d", len(rows)) + } + + select { + case e := <-nextCh: + if e.Timestamp != "2026-01-01T00:00:06Z" { + t.Fatalf("want the newer event only, got %s", e.Timestamp) + } + case <-time.After(5 * time.Second): + t.Fatal("live event never delivered") + } + select { + case e := <-nextCh: + t.Fatalf("stale event delivered despite dedup: %s", e.Timestamp) + case <-time.After(100 * time.Millisecond): + } +} + +func TestLiveQuery_BuffersDuringBackfill(t *testing.T) { + sc := bareStream() + gate := make(chan struct{}) + initialCh := make(chan []map[string]any, 1) + nextCh := make(chan StreamEvent, 8) + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { + <-gate + return []map[string]any{{"received_timestamp": "2026-01-01T00:00:01Z"}}, nil + }, + &StreamSubscriber{ + Initial: func(rows []map[string]any, _ error) { initialCh <- rows }, + Next: func(e StreamEvent) { nextCh <- e }, + }) + defer lq.Close() + + // Events arriving mid-backfill are buffered, then flushed post-Initial. + sc.emitEvent(liveEvent("2026-01-01T00:00:02Z")) + sc.emitEvent(liveEvent("2026-01-01T00:00:00Z")) // older than backfill → dropped in flush + close(gate) + + awaitInitial(t, initialCh) + select { + case e := <-nextCh: + if e.Timestamp != "2026-01-01T00:00:02Z" { + t.Fatalf("want buffered 02Z event, got %s", e.Timestamp) + } + case <-time.After(5 * time.Second): + t.Fatal("buffered event never flushed") + } + select { + case e := <-nextCh: + t.Fatalf("pre-backfill event should have been deduped: %s", e.Timestamp) + case <-time.After(100 * time.Millisecond): + } +} + +func TestLiveQuery_FetchErrorReportedOnce(t *testing.T) { + sc := bareStream() + errCh := make(chan error, 1) + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { return nil, errors.New("boom") }, + &StreamSubscriber{ + Initial: func(_ []map[string]any, err error) { errCh <- err }, + }) + defer lq.Close() + + select { + case err := <-errCh: + if err == nil || err.Error() != "boom" { + t.Fatalf("want boom, got %v", err) + } + case <-time.After(5 * time.Second): + t.Fatal("Initial never fired on fetch error") + } +} + +func TestLiveQuery_NoCallbacksAfterClose(t *testing.T) { + sc := bareStream() + initialCh := make(chan []map[string]any, 1) + var delivered atomic.Int64 + + lq := newLiveQuery(sc, + func(context.Context) ([]map[string]any, error) { return nil, nil }, + &StreamSubscriber{ + Initial: func(rows []map[string]any, _ error) { initialCh <- rows }, + Next: func(StreamEvent) { delivered.Add(1) }, + Status: func(StreamStatus) { delivered.Add(1) }, + Error: func(error) { delivered.Add(1) }, + }) + awaitInitial(t, initialCh) + + lq.Close() + before := delivered.Load() + sc.emitEvent(liveEvent("2026-01-01T00:00:09Z")) + sc.emitError(errors.New("late")) + sc.setStatus(StatusReconnecting) + time.Sleep(50 * time.Millisecond) + if got := delivered.Load(); got != before { + t.Fatalf("callbacks fired after Close: before=%d after=%d", before, got) + } +} diff --git a/clients/go/namespaces_test.go b/clients/go/namespaces_test.go new file mode 100644 index 00000000..27453e6b --- /dev/null +++ b/clients/go/namespaces_test.go @@ -0,0 +1,201 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "net/http" + "sync" + "testing" +) + +func TestSysNamespace_Health(t *testing.T) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/health" { + t.Errorf("want /v1/health, got %s", r.URL.Path) + } + w.WriteHeader(200) + })) + err := c.Sys.Health(context.Background()) + if err != nil { + t.Fatal(err) + } +} + +func TestSchemaNamespace_List(t *testing.T) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/v1/ops/schema" { + t.Errorf("want /v1/ops/schema, got %s", r.URL.Path) + } + _ = json.NewEncoder(w).Encode([]TableSchema{ + {Name: "clicks", Columns: []Column{{Name: "page", Type: "String"}}}, + }) + })) + schemas, err := c.Schema.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if _, ok := schemas["clicks"]; !ok { + t.Fatal("want clicks in schemas") + } +} + +func TestSchemaNamespace_Refresh(t *testing.T) { + var mu sync.Mutex + var gotMethod string + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + gotMethod = r.Method + mu.Unlock() + w.WriteHeader(200) + })) + err := c.Schema.Refresh(context.Background()) + if err != nil { + t.Fatal(err) + } + mu.Lock() + defer mu.Unlock() + if gotMethod != "POST" { + t.Fatalf("want POST, got %s", gotMethod) + } +} + +func TestPolicyNamespace_GetSetValidate(t *testing.T) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.Method { + case "GET": + _ = json.NewEncoder(w).Encode(Policy{Tables: map[string]TablePolicy{}}) + case "PUT": + w.WriteHeader(200) + case "POST": + _ = json.NewEncoder(w).Encode(ValidationResult{Valid: true}) + } + })) + + pol, err := c.Policy.Get(context.Background()) + if err != nil { + t.Fatal(err) + } + if pol.Tables == nil { + t.Fatal("want tables map") + } + + err = c.Policy.Set(context.Background(), pol) + if err != nil { + t.Fatal(err) + } + + v, err := c.Policy.Validate(context.Background(), pol) + if err != nil { + t.Fatal(err) + } + if !v.Valid { + t.Fatal("want valid=true") + } +} + +func TestDLQNamespace(t *testing.T) { + t.Run("List", func(t *testing.T) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 3}, Total: 3}) + })) + stats, err := c.DLQ.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if stats.Total != 3 { + t.Fatalf("want total=3, got %d", stats.Total) + } + }) + + t.Run("Table", func(t *testing.T) { + var mu sync.Mutex + var gotParam string + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + gotParam = r.URL.Query().Get("table") + mu.Unlock() + _ = json.NewEncoder(w).Encode(DLQStats{Tables: map[string]int{"clicks": 2}, Total: 2}) + })) + _, err := c.DLQ.Table(context.Background(), "clicks") + if err != nil { + t.Fatal(err) + } + mu.Lock() + defer mu.Unlock() + if gotParam != "clicks" { + t.Fatalf("want table=clicks, got %s", gotParam) + } + }) +} + +func TestPipesNamespace_CRUD(t *testing.T) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.Method { + case "GET": + if r.URL.Path == "/v1/ops/pipes" { + _ = json.NewEncoder(w).Encode([]Pipe{{Name: "p1", SQL: "SELECT 1"}}) + } else { + _ = json.NewEncoder(w).Encode(Pipe{Name: "p1", SQL: "SELECT 1"}) + } + case "PUT": + w.WriteHeader(200) + case "DELETE": + w.WriteHeader(200) + } + })) + + pipes, err := c.Pipes.List(context.Background()) + if err != nil { + t.Fatal(err) + } + if len(pipes) != 1 || pipes[0].Name != "p1" { + t.Fatalf("want [p1], got %v", pipes) + } + + p, err := c.Pipes.Get(context.Background(), "p1") + if err != nil { + t.Fatal(err) + } + if p.Name != "p1" { + t.Fatalf("want p1, got %s", p.Name) + } + + err = c.Pipes.Set(context.Background(), "p1", PipeDef{SQL: "SELECT 1"}) + if err != nil { + t.Fatal(err) + } + + err = c.Pipes.Delete(context.Background(), "p1") + if err != nil { + t.Fatal(err) + } +} + +func TestPipeRef_Fetch(t *testing.T) { + var mu sync.Mutex + var gotPath, gotMethod string + var gotBody map[string]any + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + gotPath = r.URL.Path + gotMethod = r.Method + _ = json.NewDecoder(r.Body).Decode(&gotBody) + mu.Unlock() + _ = json.NewEncoder(w).Encode([]map[string]any{{"count": 42}}) + })) + rows, err := Fetch[map[string]any](context.Background(), c.Pipe("top_pages", map[string]any{"limit": 10})) + if err != nil { + t.Fatal(err) + } + if len(rows) != 1 { + t.Fatalf("want 1 row, got %d", len(rows)) + } + mu.Lock() + defer mu.Unlock() + if gotPath != "/v1/pipes/top_pages" { + t.Fatalf("want /v1/pipes/top_pages, got %s", gotPath) + } + if gotMethod != "POST" { + t.Fatalf("want POST, got %s", gotMethod) + } +} diff --git a/clients/go/pipes.go b/clients/go/pipes.go new file mode 100644 index 00000000..5501fa86 --- /dev/null +++ b/clients/go/pipes.go @@ -0,0 +1,106 @@ +package wavehouse + +import ( + "context" + "fmt" + "net/url" +) + +// PipesNamespace provides admin-only named-pipe management. +type PipesNamespace struct { + ctx httpContext +} + +// List returns all registered pipes. Admin-only. +func (p *PipesNamespace) List(ctx context.Context) ([]Pipe, error) { + var pipes []Pipe + if err := doRequest(ctx, p.ctx, requestOptions{ + method: "GET", + path: "/v1/ops/pipes", + }, &pipes); err != nil { + return nil, fmt.Errorf("list pipes: %w", err) + } + return pipes, nil +} + +// Get returns a single pipe definition by name. Admin-only. +func (p *PipesNamespace) Get(ctx context.Context, name string) (*Pipe, error) { + var pipe Pipe + if err := doRequest(ctx, p.ctx, requestOptions{ + method: "GET", + path: "/v1/ops/pipes/" + url.PathEscape(name), + }, &pipe); err != nil { + return nil, fmt.Errorf("get pipe %q: %w", name, err) + } + return &pipe, nil +} + +// Set creates or updates a pipe. Admin-only. +func (p *PipesNamespace) Set(ctx context.Context, name string, def PipeDef) error { + if err := doRequest(ctx, p.ctx, requestOptions{ + method: "PUT", + path: "/v1/ops/pipes/" + url.PathEscape(name), + body: def, + }, nil); err != nil { + return fmt.Errorf("set pipe %q: %w", name, err) + } + return nil +} + +// Delete removes a pipe by name. Admin-only. +func (p *PipesNamespace) Delete(ctx context.Context, name string) error { + if err := doRequest(ctx, p.ctx, requestOptions{ + method: "DELETE", + path: "/v1/ops/pipes/" + url.PathEscape(name), + }, nil); err != nil { + return fmt.Errorf("delete pipe %q: %w", name, err) + } + return nil +} + +// PipeDef is the definition body for creating/updating a pipe (Pipe minus name). +type PipeDef struct { + SQL string `json:"sql"` + Parameters []ParamDef `json:"parameters,omitempty"` + Description string `json:"description,omitempty"` + AllowedRoles []string `json:"allowed_roles,omitempty"` +} + +// PipeRef is a reference to a named query pipe. Use Fetch to execute it. +type PipeRef struct { + ctx httpContext + name string + params map[string]any + createStream func(table string, opts *StreamOptions) *StreamController +} + +// Fetch executes the pipe and returns the result rows decoded into []T. +func Fetch[Row any](ctx context.Context, p *PipeRef) ([]Row, error) { + body := p.params + if body == nil { + body = map[string]any{} + } + var rows []Row + if err := doRequest(ctx, p.ctx, requestOptions{ + method: "POST", + path: "/v1/pipes/" + url.PathEscape(p.name), + body: body, + }, &rows); err != nil { + return nil, fmt.Errorf("execute pipe %q: %w", p.name, err) + } + return rows, nil +} + +// FetchUntyped executes the pipe and returns rows as []map[string]any. +func (p *PipeRef) FetchUntyped(ctx context.Context) ([]map[string]any, error) { + return Fetch[map[string]any](ctx, p) +} + +// Stream opens a live event stream from the pipe's underlying query. +// +// This streams by table name, using the pipe's own name as the table — it +// only works when the pipe name is also a valid table name. This matches +// the TS SDK's PipeRef.stream(), which has the same limitation. +func (p *PipeRef) Stream(opts *StreamOptions) *StreamController { + return p.createStream(p.name, opts) +} diff --git a/clients/go/policy.go b/clients/go/policy.go new file mode 100644 index 00000000..7e356bee --- /dev/null +++ b/clients/go/policy.go @@ -0,0 +1,48 @@ +package wavehouse + +import ( + "context" + "fmt" +) + +// PolicyNamespace provides admin-only access-control policy management. +type PolicyNamespace struct { + ctx httpContext +} + +// Get returns the current access-control policy. Admin-only. +func (p *PolicyNamespace) Get(ctx context.Context) (*Policy, error) { + var pol Policy + if err := doRequest(ctx, p.ctx, requestOptions{ + method: "GET", + path: "/v1/ops/policy", + }, &pol); err != nil { + return nil, fmt.Errorf("get policy: %w", err) + } + return &pol, nil +} + +// Set replaces the entire access-control policy. Admin-only. +func (p *PolicyNamespace) Set(ctx context.Context, pol *Policy) error { + if err := doRequest(ctx, p.ctx, requestOptions{ + method: "PUT", + path: "/v1/ops/policy", + body: pol, + }, nil); err != nil { + return fmt.Errorf("set policy: %w", err) + } + return nil +} + +// Validate checks a policy without applying it (dry run). Admin-only. +func (p *PolicyNamespace) Validate(ctx context.Context, pol *Policy) (*ValidationResult, error) { + var result ValidationResult + if err := doRequest(ctx, p.ctx, requestOptions{ + method: "POST", + path: "/v1/ops/policy/validate", + body: pol, + }, &result); err != nil { + return nil, fmt.Errorf("validate policy: %w", err) + } + return &result, nil +} diff --git a/clients/go/query_builder.go b/clients/go/query_builder.go new file mode 100644 index 00000000..8d0e0212 --- /dev/null +++ b/clients/go/query_builder.go @@ -0,0 +1,331 @@ +package wavehouse + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "net/url" +) + +// DefaultLimit is applied when no explicit limit is set — deliberately tighter +// than the backend's DefaultMaxRows (10000) safety cap. +const DefaultLimit = 1000 + +// queryState is the immutable core of a QueryBuilder. +type queryState struct { + table string + columns []string + selectAll bool + aggregations []Aggregation + filters []QueryFilter + groupBy []string + orderBy []OrderClause + limit *int + timeRange *TimeRange + cacheTTL *int // client-side only, not sent to server (#280) +} + +// QueryBuilder builds structured queries. Immutable — every chain method +// returns a new builder. Use Fetch or FetchUntyped to execute. +type QueryBuilder struct { + ctx httpContext + createStream func(table string, opts *StreamOptions) *StreamController + state queryState +} + +func (q *QueryBuilder) clone(mutate func(*queryState)) *QueryBuilder { + s := q.state + // Deep-copy slices so mutations don't alias. + s.columns = append([]string(nil), s.columns...) + s.aggregations = append([]Aggregation(nil), s.aggregations...) + s.filters = append([]QueryFilter(nil), s.filters...) + s.groupBy = append([]string(nil), s.groupBy...) + s.orderBy = append([]OrderClause(nil), s.orderBy...) + mutate(&s) + return &QueryBuilder{ctx: q.ctx, createStream: q.createStream, state: s} +} + +// Select appends columns to the projection. +func (q *QueryBuilder) Select(columns ...string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.columns = append(s.columns, columns...) + }) +} + +// SelectAll requests every column the caller's role may read. +func (q *QueryBuilder) SelectAll() *QueryBuilder { + return q.clone(func(s *queryState) { + s.selectAll = true + }) +} + +// Where adds a filter condition. +func (q *QueryBuilder) Where(column string, op FilterOp, value any) *QueryBuilder { + wireOp, ok := opMap[op] + if !ok { + wireOp = string(op) + } + return q.clone(func(s *queryState) { + s.filters = append(s.filters, QueryFilter{Column: column, Op: wireOp, Value: value}) + }) +} + +// Count adds a COUNT aggregation. +func (q *QueryBuilder) Count(column, alias string) *QueryBuilder { + if column == "" { + column = "*" + } + if alias == "" { + alias = "count" + } + return q.addAgg("count", column, alias) +} + +// Sum adds a SUM aggregation. +func (q *QueryBuilder) Sum(column, alias string) *QueryBuilder { + return q.aggDefault("sum", "sum_", column, alias) +} + +// Avg adds an AVG aggregation. +func (q *QueryBuilder) Avg(column, alias string) *QueryBuilder { + return q.aggDefault("avg", "avg_", column, alias) +} + +// Min adds a MIN aggregation. +func (q *QueryBuilder) Min(column, alias string) *QueryBuilder { + return q.aggDefault("min", "min_", column, alias) +} + +// Max adds a MAX aggregation. +func (q *QueryBuilder) Max(column, alias string) *QueryBuilder { + return q.aggDefault("max", "max_", column, alias) +} + +// CountDistinct adds a COUNT DISTINCT aggregation. +func (q *QueryBuilder) CountDistinct(column, alias string) *QueryBuilder { + return q.aggDefault("countDistinct", "count_distinct_", column, alias) +} + +// Aggregate adds a custom aggregation function. +func (q *QueryBuilder) Aggregate(fn, column, alias string) *QueryBuilder { + return q.addAgg(fn, column, alias) +} + +// GroupBy appends columns to the GROUP BY clause. +func (q *QueryBuilder) GroupBy(columns ...string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.groupBy = append(s.groupBy, columns...) + }) +} + +// OrderBy appends an ORDER BY clause. dir defaults to "asc". +func (q *QueryBuilder) OrderBy(column, dir string) *QueryBuilder { + if dir == "" { + dir = "asc" + } + return q.clone(func(s *queryState) { + s.orderBy = append(s.orderBy, OrderClause{Column: column, Dir: dir}) + }) +} + +// Limit sets the maximum number of rows to return. +func (q *QueryBuilder) Limit(n int) *QueryBuilder { + return q.clone(func(s *queryState) { + s.limit = &n + }) +} + +// TimeRange filters by a time window. since and until accept RFC3339 timestamps +// or relative durations ("1h", "30m", "7d", "2w"). +func (q *QueryBuilder) TimeRange(column, since, until string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.timeRange = &TimeRange{Column: column, Since: since, Until: until} + }) +} + +// CacheTTL records a desired result-cache TTL. Currently client-side only — +// the server derives TTLs adaptively (#280). +func (q *QueryBuilder) CacheTTL(seconds int) *QueryBuilder { + return q.clone(func(s *queryState) { + s.cacheTTL = &seconds + }) +} + +// FetchTyped executes the query and decodes rows into []T. +func FetchTyped[Row any](ctx context.Context, q *QueryBuilder) (*Page[Row], error) { + limit := DefaultLimit + if q.state.limit != nil { + limit = *q.state.limit + } + ast := q.buildAST(limit) + + var rows []Row + if err := doRequest(ctx, q.ctx, requestOptions{ + method: "POST", + path: "/v1/query", + params: url.Values{"table": {q.state.table}}, + body: ast, + }, &rows); err != nil { + return nil, err + } + + hasMore := limit > 0 && len(rows) >= limit + page := &Page[Row]{Data: rows, HasMore: hasMore} + + // Attach Next whenever we have an order column to build a cursor from. + // This doesn't check that the order column is present in the row + // projection — a Select() that omits it means fetchNextTyped can't find + // a cursor value and will quietly return an empty page (matches the TS + // SDK's QueryBuilder.fetch()/_fetchNext(), which has the same limitation). + if hasMore && len(q.state.orderBy) > 0 { + page.Next = func(ctx context.Context) (*Page[Row], error) { + return fetchNextTyped(ctx, q, rows) + } + } + + return page, nil +} + +// FetchUntyped executes the query and returns rows as []map[string]any. +func (q *QueryBuilder) FetchUntyped(ctx context.Context) (*Page[map[string]any], error) { + return FetchTyped[map[string]any](ctx, q) +} + +// Stream opens a live SSE event stream for this query's table. +// Filters and column projections are applied client-side. +func (q *QueryBuilder) Stream(opts *StreamOptions) *StreamController { + raw := q.createStream(q.state.table, opts) + if len(q.state.filters) == 0 && len(q.state.columns) == 0 { + return raw + } + return newFilteredStreamController(raw, q.state.filters, q.state.columns) +} + +// LiveQuery starts a live query: fetches historical data, then streams live +// updates. The subscriber's Initial is called once, then Next fires for each +// live event. Returns a LiveQuery handle with a Close method. +func (q *QueryBuilder) LiveQuery(sub *StreamSubscriber, opts *StreamOptions) *LiveQueryHandle { + stream := q.Stream(opts) + fetchFn := func(ctx context.Context) ([]map[string]any, error) { + page, err := q.FetchUntyped(ctx) + if err != nil { + return nil, err + } + return page.Data, nil + } + return newLiveQuery(stream, fetchFn, sub) +} + +func (q *QueryBuilder) aggDefault(fn, prefix, column, alias string) *QueryBuilder { + if alias == "" { + alias = prefix + column + } + return q.addAgg(fn, column, alias) +} + +func (q *QueryBuilder) addAgg(fn, column, alias string) *QueryBuilder { + return q.clone(func(s *queryState) { + s.aggregations = append(s.aggregations, Aggregation{Fn: fn, Column: column, Alias: alias}) + }) +} + +func (q *QueryBuilder) buildAST(effectiveLimit int) *StructuredQuery { + ast := &StructuredQuery{} + hasColumns := len(q.state.columns) > 0 + hasAggs := len(q.state.aggregations) > 0 + + // Projection: explicit select_all, then explicit columns, else — for a bare + // query with no projection and no aggregations — default to select_all so + // from(t).fetch() returns rows. + switch { + case q.state.selectAll: + ast.SelectAll = true + case hasColumns: + ast.Columns = q.state.columns + case !hasAggs: + ast.SelectAll = true + } + + if hasAggs { + ast.Aggregations = q.state.aggregations + } + if len(q.state.filters) > 0 { + ast.Filters = q.state.filters + } + if len(q.state.groupBy) > 0 { + ast.GroupBy = q.state.groupBy + } + if len(q.state.orderBy) > 0 { + ast.OrderBy = q.state.orderBy + } + ast.Limit = &effectiveLimit + if q.state.timeRange != nil { + ast.TimeRange = q.state.timeRange + } + return ast +} + +func fetchNextTyped[Row any](ctx context.Context, q *QueryBuilder, prevRows []Row) (*Page[Row], error) { + if len(q.state.orderBy) == 0 { + return &Page[Row]{}, nil + } + cursor := q.state.orderBy[0] + + // Extract the last row's value for the cursor column. + lastRow := any(prevRows[len(prevRows)-1]) + m, ok := lastRow.(map[string]any) + if !ok { + // TODO: marshal/unmarshal round-trip to get a map — optimize with reflect if perf matters. + // UseNumber keeps typed int64 cursor values exact past 2^53. The + // untyped path (FetchUntyped / TableRef.Fetch) doesn't get this + // protection: its rows were already decoded to float64 by + // encoding/json, so precision above 2^53 is gone before we get here — + // the same ceiling the TS SDK has with JS numbers. Use FetchTyped (or + // codegen structs — their 64-bit int columns are int64/uint64, and + // 128/256-bit are json.Number) when paging on >2^53 integer cursors. + raw, err := json.Marshal(lastRow) + if err != nil { + // Row itself is unmarshalable (e.g. a func field absent from the + // response). Silently truncating the result set would look like + // normal end-of-pagination, so surface it. + return nil, fmt.Errorf("wavehouse: marshal cursor row: %w", err) + } + m = make(map[string]any) + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.UseNumber() + // Decode error is deliberate: a Row that marshals to a non-object + // (FetchTyped[[]any], a scalar row type) leaves m empty and ends + // pagination below, same as an absent cursor column. Tracked in #452. + _ = dec.Decode(&m) + } + lastValue, exists := m[cursor.Column] + if !exists { + // Cursor column wasn't in the projection (e.g. Select() omitted it) — + // no cursor value to page from, so end pagination quietly rather than + // erroring. Matches the TS SDK's _fetchNext(). + return &Page[Row]{}, nil + } + + cursorOp := "gt" + if cursor.Dir == "desc" { + cursorOp = "lt" + } + + next := q.clone(func(s *queryState) { + // Replace an existing cursor filter instead of appending — otherwise + // page N carries N stacked filters on the cursor column. + for i := range s.filters { + if s.filters[i].Column == cursor.Column && s.filters[i].Op == cursorOp { + s.filters[i].Value = lastValue + return + } + } + s.filters = append(s.filters, QueryFilter{ + Column: cursor.Column, + Op: cursorOp, + Value: lastValue, + }) + }) + return FetchTyped[Row](ctx, next) +} diff --git a/clients/go/query_builder_test.go b/clients/go/query_builder_test.go new file mode 100644 index 00000000..7ccb9027 --- /dev/null +++ b/clients/go/query_builder_test.go @@ -0,0 +1,481 @@ +package wavehouse + +import ( + "bytes" + "context" + "encoding/json" + "fmt" + "io" + "net/http" + "net/http/httptest" + "sync" + "testing" +) + +func queryTestCtx(t *testing.T, handler http.Handler) *Client { + t.Helper() + srv := httptest.NewServer(handler) + t.Cleanup(srv.Close) + return NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{MaxRetries: 0}, + }) +} + +func captureQueryBody(t *testing.T, handler http.Handler) (*Client, func() map[string]any) { + t.Helper() + // body is written on the server goroutine and read on the test goroutine; + // the mutex is what makes that visible under -race. + var mu sync.Mutex + var body []byte + wrapper := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, err := io.ReadAll(r.Body) + if err != nil { + t.Errorf("read request body: %v", err) + } + mu.Lock() + body = raw + mu.Unlock() + handler.ServeHTTP(w, r) + }) + c := queryTestCtx(t, wrapper) + return c, func() map[string]any { + mu.Lock() + defer mu.Unlock() + var m map[string]any + _ = json.Unmarshal(body, &m) + return m + } +} + +var emptyRows = http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode([]map[string]any{{"page": "/home"}}) +}) + +func TestQueryBuilder_Immutability(t *testing.T) { + c := queryTestCtx(t, emptyRows) + b1 := c.From("clicks").Select("page") + b2 := b1.Where("score", OpGt, 10) + if b1 == b2 { + t.Fatal("builder should be immutable — chain methods return new instances") + } +} + +func TestQueryBuilder_SelectColumns(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page", "button").FetchUntyped(context.Background()) + + body := getBody() + cols, ok := body["columns"].([]any) + if !ok || len(cols) != 2 { + t.Fatalf("want [page, button], got %v", body["columns"]) + } +} + +func TestQueryBuilder_SelectAll(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").SelectAll().FetchUntyped(context.Background()) + + body := getBody() + if body["select_all"] != true { + t.Fatalf("want select_all=true, got %v", body) + } +} + +func TestQueryBuilder_BareQueryDefaultsToSelectAll(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select().FetchUntyped(context.Background()) + + body := getBody() + if body["select_all"] != true { + t.Fatalf("bare query should default to select_all, got %v", body) + } +} + +func TestQueryBuilder_AggregationOnlyNoSelectAll(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select().Count("*", "n").FetchUntyped(context.Background()) + + body := getBody() + if body["select_all"] != nil { + t.Fatalf("aggregation-only query should not set select_all, got %v", body) + } +} + +func TestQueryBuilder_Where(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").Where("score", OpGt, 10).FetchUntyped(context.Background()) + + body := getBody() + filters, ok := body["filters"].([]any) + if !ok || len(filters) != 1 { + t.Fatalf("want 1 filter, got %v", body["filters"]) + } + f := filters[0].(map[string]any) + if f["column"] != "score" || f["op"] != "gt" { + t.Fatalf("want score/gt filter, got %v", f) + } +} + +func TestQueryBuilder_AllOperators(t *testing.T) { + ops := []struct { + sdk FilterOp + wire string + }{ + {OpEq, "eq"}, + {OpNeq, "neq"}, + {OpGt, "gt"}, + {OpGte, "gte"}, + {OpLt, "lt"}, + {OpLte, "lte"}, + {OpIn, "in"}, + {OpLike, "like"}, + {OpNotLike, "not_like"}, + } + for _, tt := range ops { + t.Run(tt.wire, func(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("x").Where("col", tt.sdk, "v").FetchUntyped(context.Background()) + body := getBody() + filters := body["filters"].([]any) + f := filters[0].(map[string]any) + if f["op"] != tt.wire { + t.Errorf("want wire op %s, got %s", tt.wire, f["op"]) + } + }) + } +} + +func TestQueryBuilder_Aggregations(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select(). + Count("*", "total"). + Sum("score", ""). + Avg("score", ""). + Min("score", ""). + Max("score", ""). + CountDistinct("page", ""). + Aggregate("uniqExact", "user_id", "unique_users"). + FetchUntyped(context.Background()) + + body := getBody() + aggs, ok := body["aggregations"].([]any) + if !ok || len(aggs) != 7 { + t.Fatalf("want 7 aggregations, got %v", body["aggregations"]) + } +} + +func TestQueryBuilder_GroupBy(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").GroupBy("page").FetchUntyped(context.Background()) + + body := getBody() + gb, ok := body["group_by"].([]any) + if !ok || len(gb) != 1 || gb[0] != "page" { + t.Fatalf("want [page], got %v", body["group_by"]) + } +} + +func TestQueryBuilder_OrderBy(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").OrderBy("page", "desc").FetchUntyped(context.Background()) + + body := getBody() + ob := body["order_by"].([]any) + o := ob[0].(map[string]any) + if o["column"] != "page" || o["dir"] != "desc" { + t.Fatalf("want page/desc, got %v", o) + } +} + +func TestQueryBuilder_Limit(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page").Limit(50).FetchUntyped(context.Background()) + + body := getBody() + if body["limit"] != float64(50) { + t.Fatalf("want 50, got %v", body["limit"]) + } +} + +func TestQueryBuilder_TimeRange(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks").Select("page"). + TimeRange("received_timestamp", "1h", ""). + FetchUntyped(context.Background()) + + body := getBody() + tr := body["time_range"].(map[string]any) + if tr["column"] != "received_timestamp" || tr["since"] != "1h" { + t.Fatalf("want received_timestamp/1h, got %v", tr) + } +} + +func TestQueryBuilder_Pagination_HasMore(t *testing.T) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) + })) + + page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if !page.HasMore { + t.Fatal("want hasMore=true") + } + if page.Next == nil { + t.Fatal("want next function") + } +} + +func TestQueryBuilder_Pagination_NoOrderNoNext(t *testing.T) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode([]map[string]any{{"id": "a"}, {"id": "b"}}) + })) + + page, err := c.From("clicks").Select("id").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if !page.HasMore { + t.Fatal("want hasMore=true") + } + if page.Next != nil { + t.Fatal("want nil next — no order column for cursor") + } +} + +func TestQueryBuilder_ComplexQuery(t *testing.T) { + c, getBody := captureQueryBody(t, emptyRows) + _, _ = c.From("clicks"). + Select("page"). + Where("score", OpGt, 10). + Count("*", "total"). + GroupBy("page"). + OrderBy("total", "desc"). + Limit(50). + TimeRange("received_timestamp", "1h", ""). + CacheTTL(60). + FetchUntyped(context.Background()) + + body := getBody() + if body["columns"].([]any)[0] != "page" { + t.Fatal("missing page column") + } + if body["limit"] != float64(50) { + t.Fatal("wrong limit") + } + if body["group_by"].([]any)[0] != "page" { + t.Fatal("wrong group_by") + } +} + +// pagingServer returns limit-sized pages of rows and captures each request +// body, so tests can walk page.Next and inspect the cursor filters sent. +func pagingServer(t *testing.T, pages [][]map[string]any) (*Client, func() []map[string]any) { + t.Helper() + var mu sync.Mutex + var bodies []map[string]any + call := 0 + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + raw, _ := io.ReadAll(r.Body) + var body map[string]any + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.UseNumber() // keep int64 cursor values exact on the capture side too + _ = dec.Decode(&body) + mu.Lock() + bodies = append(bodies, body) + idx := call + call++ + mu.Unlock() + page := []map[string]any{} + if idx < len(pages) { + page = pages[idx] + } + _ = json.NewEncoder(w).Encode(page) + })) + return c, func() []map[string]any { + mu.Lock() + defer mu.Unlock() + return append([]map[string]any(nil), bodies...) + } +} + +func filtersOf(t *testing.T, body map[string]any) []map[string]any { + t.Helper() + raw, ok := body["filters"].([]any) + if !ok { + return nil + } + out := make([]map[string]any, len(raw)) + for i, f := range raw { + out[i] = f.(map[string]any) + } + return out +} + +func TestQueryBuilder_Pagination_NextWalksPages(t *testing.T) { + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": "a"}, {"id": "b"}}, + {{"id": "c"}, {"id": "d"}}, + {{"id": "e"}}, + }) + + page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + page2, err := page.Next(context.Background()) + if err != nil { + t.Fatal(err) + } + if page2.Data[0]["id"] != "c" || !page2.HasMore || page2.Next == nil { + t.Fatalf("unexpected page 2: %+v", page2) + } + page3, err := page2.Next(context.Background()) + if err != nil { + t.Fatal(err) + } + if len(page3.Data) != 1 || page3.HasMore { + t.Fatalf("unexpected page 3: %+v", page3) + } + + bodies := getBodies() + if len(bodies) != 3 { + t.Fatalf("want 3 requests, got %d", len(bodies)) + } + if f := filtersOf(t, bodies[0]); len(f) != 0 { + t.Fatalf("page 1 must have no cursor filter, got %v", f) + } + // Page 2 and 3: exactly ONE cursor filter (replaced, not stacked), with + // the ascending op and the previous page's last cursor value. + for i, want := range []string{"b", "d"} { + f := filtersOf(t, bodies[i+1]) + if len(f) != 1 { + t.Fatalf("page %d: want exactly 1 cursor filter, got %v", i+2, f) + } + if f[0]["column"] != "id" || f[0]["op"] != "gt" || f[0]["value"] != want { + t.Fatalf("page %d: unexpected cursor filter %v", i+2, f[0]) + } + } +} + +func TestQueryBuilder_Pagination_DescUsesLt(t *testing.T) { + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": "z"}, {"id": "y"}}, + {{"id": "x"}}, + }) + + page, err := c.From("clicks").Select("id").OrderBy("id", "desc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if _, err := page.Next(context.Background()); err != nil { + t.Fatal(err) + } + + f := filtersOf(t, getBodies()[1]) + if len(f) != 1 || f[0]["op"] != "lt" || f[0]["value"] != "y" { + t.Fatalf("desc cursor filter wrong: %v", f) + } +} + +func TestQueryBuilder_Pagination_CursorColumnMissingEndsQuietly(t *testing.T) { + c, _ := pagingServer(t, [][]map[string]any{ + {{"other": "1"}, {"other": "2"}}, // projection omits the order column + }) + + page, err := c.From("clicks").Select("other").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + next, err := page.Next(context.Background()) + if err != nil { + t.Fatal(err) + } + if len(next.Data) != 0 || next.HasMore || next.Next != nil { + t.Fatalf("want quiet empty page, got %+v", next) + } +} + +func TestQueryBuilder_Pagination_TypedInt64CursorKeepsPrecision(t *testing.T) { + type idRow struct { + ID int64 `json:"id"` + } + const bigID = int64(9007199254740993) // 2^53 + 1: float64 round-trip corrupts it + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": 1}, {"id": bigID}}, + {}, + }) + + q := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2) + page, err := FetchTyped[idRow](context.Background(), q) + if err != nil { + t.Fatal(err) + } + if _, err := page.Next(context.Background()); err != nil { + t.Fatal(err) + } + + f := filtersOf(t, getBodies()[1]) + if len(f) != 1 { + t.Fatalf("want 1 cursor filter, got %v", f) + } + // json.Number survives the round-trip; float64 would have sent ...992. + if got := fmt.Sprint(f[0]["value"]); got != "9007199254740993" { + t.Fatalf("cursor value lost precision: %s", got) + } +} + +// TestQueryBuilder_Pagination_UntypedCursorFloat64Ceiling documents the known +// ceiling on the untyped path: rows decode to float64, so an integer cursor +// past 2^53 loses precision before pagination sees it (same as the TS SDK's +// JS-number ceiling). Use FetchTyped or codegen structs past 2^53. +func TestQueryBuilder_Pagination_UntypedCursorFloat64Ceiling(t *testing.T) { + c, getBodies := pagingServer(t, [][]map[string]any{ + {{"id": 1}, {"id": int64(9007199254740993)}}, + {}, + }) + + page, err := c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2).FetchUntyped(context.Background()) + if err != nil { + t.Fatal(err) + } + if _, err := page.Next(context.Background()); err != nil { + t.Fatal(err) + } + + f := filtersOf(t, getBodies()[1]) + if len(f) != 1 { + t.Fatalf("want 1 cursor filter, got %v", f) + } + // float64 rounds 2^53+1 down to 2^53 — the documented untyped ceiling. + if got := fmt.Sprint(f[0]["value"]); got != "9007199254740992" { + t.Fatalf("untyped ceiling changed (update docs if intentional): %s", got) + } +} + +// The cursor round-trip re-marshals the last row to read its cursor value. A +// Row that unmarshals cleanly but can't be marshaled back (an exported func +// field, absent from the response) must surface an error rather than an empty +// page, which is indistinguishable from real exhaustion. +func TestQueryBuilder_Pagination_UnmarshalableRowErrors(t *testing.T) { + type row struct { + ID string `json:"id"` + Cb func() `json:"cb"` + } + c, _ := pagingServer(t, [][]map[string]any{{{"id": "a"}, {"id": "b"}}}) + page, err := FetchTyped[row](context.Background(), + c.From("clicks").Select("id").OrderBy("id", "asc").Limit(2)) + if err != nil { + t.Fatal(err) + } + if page.Next == nil { + t.Fatal("want a Next cursor") + } + if _, err := page.Next(context.Background()); err == nil { + t.Fatal("want a marshal error, got a silently empty page") + } +} diff --git a/clients/go/schema.go b/clients/go/schema.go new file mode 100644 index 00000000..c79dcb11 --- /dev/null +++ b/clients/go/schema.go @@ -0,0 +1,39 @@ +package wavehouse + +import ( + "context" + "fmt" +) + +// SchemaNamespace provides admin-only schema introspection. +type SchemaNamespace struct { + ctx httpContext +} + +// List returns all table schemas discovered from ClickHouse. Admin-only. +func (s *SchemaNamespace) List(ctx context.Context) (Schemas, error) { + // The backend returns []TableSchema; transform to map[string]TableSchema. + var raw []TableSchema + if err := doRequest(ctx, s.ctx, requestOptions{ + method: "GET", + path: "/v1/ops/schema", + }, &raw); err != nil { + return nil, fmt.Errorf("list schemas: %w", err) + } + schemas := make(Schemas, len(raw)) + for _, t := range raw { + schemas[t.Name] = t + } + return schemas, nil +} + +// Refresh forces a schema re-discovery from ClickHouse. Admin-only. +func (s *SchemaNamespace) Refresh(ctx context.Context) error { + if err := doRequest(ctx, s.ctx, requestOptions{ + method: "POST", + path: "/v1/ops/schema/refresh", + }, nil); err != nil { + return fmt.Errorf("refresh schema: %w", err) + } + return nil +} diff --git a/clients/go/stream.go b/clients/go/stream.go new file mode 100644 index 00000000..a9388896 --- /dev/null +++ b/clients/go/stream.go @@ -0,0 +1,799 @@ +package wavehouse + +import ( + "bufio" + "context" + "encoding/json" + "errors" + "fmt" + "log" + "mime" + "net/http" + "net/url" + "reflect" + "regexp" + "strings" + "sync" + "time" +) + +// StreamController manages a live SSE event stream. Use Subscribe for +// callback-based consumption or Events for channel-based consumption. +type StreamController struct { + mu sync.Mutex + status StreamStatus + subscribers []*StreamSubscriber + eventCh chan StreamEvent // single buffered channel for Go-native consumption + dropLogOnce sync.Once + cancel context.CancelFunc + done chan struct{} + closed bool +} + +// newStreamController opens an SSE connection for the given table. +func newStreamController(hctx httpContext, table string, opts *StreamOptions) *StreamController { + ctx, cancel := context.WithCancel(context.Background()) + sc := &StreamController{ + status: StatusConnecting, + eventCh: make(chan StreamEvent, 256), + cancel: cancel, + done: make(chan struct{}), + } + go sc.run(ctx, hctx, table, opts) + return sc +} + +// Status returns the current connection status. +func (sc *StreamController) Status() StreamStatus { + sc.mu.Lock() + defer sc.mu.Unlock() + return sc.status +} + +// Subscribe registers callbacks for stream events. Returns an unsubscribe +// function. The subscriber's Status callback fires immediately with the +// current status. +func (sc *StreamController) Subscribe(sub *StreamSubscriber) func() { + sc.mu.Lock() + sc.subscribers = append(sc.subscribers, sub) + currentStatus := sc.status + sc.mu.Unlock() + + // Benign race: setStatus also calls the subscriber, so a stale + // status here is immediately followed by the correct one. + if sub.Status != nil { + sub.Status(currentStatus) + } + + return func() { + sc.mu.Lock() + defer sc.mu.Unlock() + for i, s := range sc.subscribers { + if s == sub { + sc.subscribers = append(sc.subscribers[:i], sc.subscribers[i+1:]...) + break + } + } + } +} + +// Events returns a read-only channel that receives stream events. +// The channel is closed when the stream closes. Events buffer into it from +// stream construction (matching the TS SDK), so events that arrive before +// the first Events() call are not lost. A Subscribe-only consumer that never +// calls Events() at most fills the 256-slot buffer and trips the one-time +// drop log. +func (sc *StreamController) Events() <-chan StreamEvent { + return sc.eventCh +} + +// Connected blocks until the stream reaches "live" status or the context +// expires. Returns an error if the stream closes before connecting. +func (sc *StreamController) Connected(ctx context.Context) error { + sc.mu.Lock() + if sc.status == StatusLive { + sc.mu.Unlock() + return nil + } + if sc.status == StatusClosed || sc.closed { + sc.mu.Unlock() + return fmt.Errorf("stream is closed") + } + sc.mu.Unlock() + + // Poll — simple and correct. + // TODO: switch to a condition variable if polling shows up in profiles. + ticker := time.NewTicker(50 * time.Millisecond) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-sc.done: + return fmt.Errorf("stream closed before connecting") + case <-ticker.C: + sc.mu.Lock() + s := sc.status + sc.mu.Unlock() + if s == StatusLive { + return nil + } + if s == StatusClosed { + return fmt.Errorf("stream closed before connecting") + } + } + } +} + +// Close shuts down the stream and releases resources. Non-blocking so it is +// safe to call from subscriber callbacks (which run on the stream goroutine). +func (sc *StreamController) Close() { + sc.mu.Lock() + if sc.closed { + sc.mu.Unlock() + return + } + sc.closed = true + sc.mu.Unlock() + + sc.cancel() + // Don't block on <-sc.done: callbacks execute on the stream goroutine, + // so waiting here would deadlock if Close is called from a callback. +} + +func (sc *StreamController) setStatus(s StreamStatus) { + sc.mu.Lock() + // StatusClosed is terminal: a filtered wrapper's inner controller can + // have copied its subscriber slice before unsub, so a stale Status + // callback may land after Close — it must not resurrect the status. + if s == sc.status || sc.status == StatusClosed { + sc.mu.Unlock() + return + } + sc.status = s + subs := append([]*StreamSubscriber(nil), sc.subscribers...) + sc.mu.Unlock() + + for _, sub := range subs { + if sub.Status != nil { + sub.Status(s) + } + } +} + +// snapshotSubs copies the subscriber list under mu so callbacks run unlocked. +// setStatus keeps its own inline copy: there the snapshot must share the +// critical section with the status write to keep callback order consistent. +func (sc *StreamController) snapshotSubs() []*StreamSubscriber { + sc.mu.Lock() + defer sc.mu.Unlock() + return append([]*StreamSubscriber(nil), sc.subscribers...) +} + +func (sc *StreamController) emitEvent(event StreamEvent) { + for _, sub := range sc.snapshotSubs() { + if sub.Next != nil { + sub.Next(event) + } + } + + // Non-blocking send to the channel, which buffers from construction (TS + // parity) so events emitted before the first Events() call survive. + // Guarded by mu so the send and closeEventCh serialize — a late event can + // never hit a closed channel. + sc.mu.Lock() + defer sc.mu.Unlock() + if sc.closed { + return + } + select { + case sc.eventCh <- event: + default: + sc.dropLogOnce.Do(func() { + log.Printf("[wavehouse] stream event dropped: Events() channel buffer full (further drops not logged)") + }) + } +} + +func (sc *StreamController) emitError(err error) { + for _, sub := range sc.snapshotSubs() { + if sub.Error != nil { + sub.Error(err) + } + } +} + +// closeEventCh marks the controller closed and closes the events channel. +// Must serialize with emitEvent's send via mu. +func (sc *StreamController) closeEventCh() { + sc.mu.Lock() + sc.closed = true + close(sc.eventCh) + sc.mu.Unlock() +} + +// run is the SSE connection loop with reconnect/backoff. +func (sc *StreamController) run(ctx context.Context, hctx httpContext, table string, opts *StreamOptions) { + defer func() { + sc.setStatus(StatusClosed) + sc.closeEventCh() + close(sc.done) + }() + + since := "" + if opts != nil { + since = opts.Since + } + + attempt := 0 + for { + if ctx.Err() != nil { + return + } + + lastID, live, err := sc.connect(ctx, hctx, table, since) + // Persist the last event ID so the next reconnect resumes from it. + if lastID != "" { + since = lastID + } + if ctx.Err() != nil { + return + } + + // A connection that reached "live" resets the backoff so a long-lived + // stream doesn't inherit a maxed-out delay on its first drop. + if live { + attempt = 0 + } + + if err != nil { + // connect classifies its own failures (SSE_AUTH_ERROR, + // SSE_NETWORK_ERROR, SSE_REDIRECT, SSE_BAD_CONTENT_TYPE, + // SSE_READ_ERROR, HTTP_nnn), so pass the typed error straight + // through and let Retryable decide whether to reconnect. A + // non-retryable error is terminal: reconnecting can't fix a bad + // token, a missing table, or a proxy answering with HTML. + var apiErr *Error + if errors.As(err, &apiErr) { + sc.emitError(apiErr) + if !apiErr.Retryable { + return + } + } else { + // Unclassified — retry, but keep the generic code so callers + // can still match on it. + sc.emitError(&Error{ + Status: 0, + Code: "SSE_ERROR", + Message: err.Error(), + Retryable: true, + }) + } + } + + sc.setStatus(StatusReconnecting) + delay := backoff(attempt) + attempt++ + + select { + case <-ctx.Done(): + return + case <-time.After(delay): + } + } +} + +// connect opens a single SSE connection and reads events until it closes. +// Returns the last seen event ID (empty if none), whether the connection +// reached the live state, and any error. +func (sc *StreamController) connect(ctx context.Context, hctx httpContext, table, since string) (string, bool, error) { + u, err := url.Parse(hctx.baseURL + "/v1/stream") + if err != nil { + return "", false, &Error{ + Status: 0, + Code: "SSE_CONNECT_ERROR", + Message: fmt.Sprintf("invalid baseURL: %v", err), + Retryable: false, + } + } + // A non-HTTP scheme can never carry SSE. Terminal, not retryable: retrying + // a ws:// or file:// baseURL just spins. + if u.Scheme != "http" && u.Scheme != "https" { + return "", false, &Error{ + Status: 0, + Code: "SSE_CONNECT_ERROR", + Message: fmt.Sprintf("baseURL scheme %q is not http or https", u.Scheme), + Retryable: false, + } + } + q := u.Query() + q.Set("table", table) + if since != "" { + q.Set("since", since) + } + + // Auth: Go SDK uses Authorization header (not ?token= like browser EventSource). + var authHeader string + if hctx.auth != nil { + token, err := hctx.auth(ctx) + if err != nil { + // Retryable: a token endpoint having a bad minute shouldn't tear + // down a healthy long-lived stream. + return "", false, &Error{ + Status: 0, + Code: "SSE_AUTH_ERROR", + Message: err.Error(), + Retryable: true, + } + } + if token != "" { + authHeader = "Bearer " + token + } + } + + u.RawQuery = q.Encode() + + req, err := http.NewRequestWithContext(ctx, "GET", u.String(), nil) + if err != nil { + return "", false, err + } + applyConfiguredHeaders(req.Header, hctx.headers) + req.Header.Set("Accept", "text/event-stream") + req.Header.Set("Cache-Control", "no-cache") + if authHeader != "" { + req.Header.Set("Authorization", authHeader) + } + + client := hctx.httpClient + credentialed := authHeader != "" || len(hctx.headers) > 0 + if credentialed { + // Refuse to follow a redirect while carrying a credential. net/http + // drops Authorization on a cross-host hop but forwards custom headers + // verbatim, so following one would either downgrade the stream to + // default_role without saying so, or hand configured secrets to + // wherever the redirect points. Copy the client so a caller-supplied + // one keeps its own CheckRedirect for every other request. + c := *client + c.CheckRedirect = func(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse + } + client = &c + } + + resp, err := client.Do(req) + if err != nil { + if ctx.Err() != nil { + return "", false, errAborted + } + return "", false, &Error{ + Status: 0, + Code: "SSE_NETWORK_ERROR", + Message: err.Error(), + Retryable: true, + } + } + defer func() { _ = resp.Body.Close() }() + + if credentialed && resp.StatusCode >= 300 && resp.StatusCode < 400 { + return "", false, &Error{ + Status: resp.StatusCode, + Code: "SSE_REDIRECT", + Message: fmt.Sprintf( + "stream endpoint redirected to %q and the SDK did not follow it; redirects are refused while the request carries a credential", + resp.Header.Get("Location")), + Retryable: false, + } + } + + if resp.StatusCode != http.StatusOK { + return "", false, parseErrorResponse(resp) + } + + // A 200 that isn't an event stream means something between the caller and + // WaveHouse answered — a captive portal or an auth gateway's login page. + // Without this check the stream sits in StatusLive and silently delivers + // nothing. + if ct := resp.Header.Get("Content-Type"); !isEventStream(ct) { + shown := ct + if shown == "" { + shown = "(none)" + } + return "", false, &Error{ + Status: resp.StatusCode, + Code: "SSE_BAD_CONTENT_TYPE", + Message: fmt.Sprintf("expected Content-Type text/event-stream, got %s", shown), + Retryable: false, + } + } + + sc.setStatus(StatusLive) + + // Parse SSE frames. + scanner := bufio.NewScanner(resp.Body) + // 16 MiB max line: generous headroom over the ~1 MiB NATS MaxPayload + // ceiling on a single event envelope (oversized records are rejected at + // ingest publish and never reach the stream). + scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024) + var eventID, dataLine string + lastID := since + + for scanner.Scan() { + if ctx.Err() != nil { + return lastID, true, nil + } + + line := scanner.Text() + + if line == "" { + // Empty line = end of event frame. + if dataLine != "" { + sc.handleSSEData(dataLine) + // Track last event ID for reconnect gap-fill. + if eventID != "" { + lastID = eventID + } + } + eventID = "" + dataLine = "" + continue + } + + if strings.HasPrefix(line, ":") { + // Comment (keepalive or connected). Skip. + continue + } + + if strings.HasPrefix(line, "id:") { + eventID = strings.TrimSpace(strings.TrimPrefix(line, "id:")) + } else if strings.HasPrefix(line, "data:") { + trimmed := strings.TrimSpace(strings.TrimPrefix(line, "data:")) + if dataLine == "" { + dataLine = trimmed + } else { + dataLine = dataLine + "\n" + trimmed + } + } + } + + if scanErr := scanner.Err(); scanErr != nil { + return lastID, true, &Error{ + Status: 0, + Code: "SSE_READ_ERROR", + Message: scanErr.Error(), + Retryable: true, + } + } + return lastID, true, nil +} + +// isEventStream reports whether a Content-Type header names text/event-stream, +// ignoring any parameters (charset, boundary) and case. +func isEventStream(contentType string) bool { + mediaType, _, err := mime.ParseMediaType(contentType) + if err != nil { + return false + } + return mediaType == "text/event-stream" +} + +// sseMessage matches the server's SSE event JSON shape. +type sseMessage struct { + TableName string `json:"table_name"` + ReceivedTimestamp string `json:"received_timestamp"` + Data map[string]any `json:"data"` +} + +func (sc *StreamController) handleSSEData(data string) { + var msg sseMessage + if err := json.Unmarshal([]byte(data), &msg); err != nil { + // Delivered via the subscriber Error callback rather than the + // process-global logger, so consumers control visibility and a + // malformed-frame flood can't spam host-application logs. Payload + // deliberately omitted: event data can carry tenant/PII fields. + sc.emitError(&Error{ + Status: 0, + Code: "SSE_PARSE_ERROR", + Message: fmt.Sprintf("malformed SSE message (%d bytes): %v", len(data), err), + Retryable: true, + }) + return + } + + event := StreamEvent{ + Table: msg.TableName, + Timestamp: msg.ReceivedTimestamp, + Data: msg.Data, + } + sc.emitEvent(event) +} + +// newFilteredStreamController wraps a StreamController with client-side +// filtering and column projection. +func newFilteredStreamController(inner *StreamController, filters []QueryFilter, columns []string) *StreamController { + compiled := compileFilters(filters) + ctx, cancel := context.WithCancel(context.Background()) + sc := &StreamController{ + status: inner.Status(), + eventCh: make(chan StreamEvent, 256), + cancel: cancel, + done: make(chan struct{}), + } + + go func() { + defer func() { + sc.setStatus(StatusClosed) + // closeEventCh serializes with any in-flight emitEvent (which + // runs on the inner controller's goroutine), so the channel is + // never closed under a pending send. + sc.closeEventCh() + close(sc.done) + }() + + unsub := inner.Subscribe(&StreamSubscriber{ + Next: func(event StreamEvent) { + if !matchesFilters(event.Data, compiled) { + return + } + if len(columns) > 0 { + event.Data = projectColumns(event.Data, columns) + } + sc.emitEvent(event) + }, + Status: func(s StreamStatus) { + sc.setStatus(s) + }, + Error: func(err error) { + sc.emitError(err) + }, + }) + + select { + case <-ctx.Done(): + unsub() + inner.Close() + case <-inner.done: + // Inner closed on its own — still unsubscribe so the closed + // controller doesn't retain a reference to this wrapper. + unsub() + } + }() + + return sc +} + +// compiledFilter pairs a filter with its precompiled LIKE regex (nil for +// every other operator, or when the pattern isn't a string / doesn't compile). +type compiledFilter struct { + QueryFilter + re *regexp.Regexp +} + +// compileFilters precompiles LIKE/NOT LIKE patterns once per stream. A +// controller's filters never change after construction, so this replaces a +// per-event compile (and avoids any process-global pattern cache). +func compileFilters(filters []QueryFilter) []compiledFilter { + out := make([]compiledFilter, len(filters)) + for i, f := range filters { + out[i] = compiledFilter{QueryFilter: f} + if f.Op == "like" || f.Op == "not_like" { + if pattern, ok := f.Value.(string); ok { + out[i].re = compileLike(pattern) + } + } + } + return out +} + +// compileLike converts a SQL LIKE pattern to a case-insensitive anchored +// regex (matching the TS SDK). Returns nil if the pattern doesn't compile. +func compileLike(pattern string) *regexp.Regexp { + escaped := regexp.QuoteMeta(pattern) + escaped = strings.ReplaceAll(escaped, "%", ".*") + escaped = strings.ReplaceAll(escaped, "_", ".") + re, err := regexp.Compile("(?i)^" + escaped + "$") + if err != nil { + return nil + } + return re +} + +// matchesFilters evaluates all filters against a data row (AND). +func matchesFilters(row map[string]any, filters []compiledFilter) bool { + for _, f := range filters { + val := row[f.Column] + if !evaluateFilter(val, f.Op, f.Value, f.re) { + return false + } + } + return true +} + +func evaluateFilter(actual any, op string, expected any, re *regexp.Regexp) bool { + switch op { + case "eq": + return equalValues(actual, expected) + case "neq": + return !equalValues(actual, expected) + case "gt": + c, ok := compareOrdered(actual, expected) + return ok && c > 0 + case "gte": + c, ok := compareOrdered(actual, expected) + return ok && c >= 0 + case "lt": + c, ok := compareOrdered(actual, expected) + return ok && c < 0 + case "lte": + c, ok := compareOrdered(actual, expected) + return ok && c <= 0 + case "in": + return evaluateIn(actual, expected) + case "like", "not_like": + aStr, ok := actual.(string) + if !ok || re == nil { + return false + } + return (op == "like") == re.MatchString(aStr) + default: + return false + } +} + +// equalValues compares two values for equality, normalizing numeric types +// (JSON decodes numbers as float64, but callers may pass int) and comparing +// timestamps as instants rather than as text — see asInstant. +func equalValues(a, b any) bool { + // nil only equals nil: without this the fmt.Sprint fallback would match a + // missing column against the literal string "". + if a == nil || b == nil { + return a == nil && b == nil + } + if af, aOK := toFloat64(a); aOK { + if bf, bOK := toFloat64(b); bOK { + return af == bf + } + } + if at, aOK := asInstant(a); aOK { + if bt, bOK := asInstant(b); bOK { + return at.Equal(bt) + } + } + // fmt.Sprint is safe for all types (no panic on maps/slices). + return fmt.Sprint(a) == fmt.Sprint(b) +} + +// maxTimeOperandChars mirrors the server's row-filter pre-gate: the longest +// spelling the ingest grammar accepts (RFC 3339 with nanoseconds and a numeric +// offset) is 35 bytes, so 64 is generous slack while keeping a megabyte +// "timestamp" from being scanned once per filter per event. +const maxTimeOperandChars = 64 + +// asInstant reports the instant a value denotes, but only for spellings that +// name one unambiguously — RFC 3339 with an explicit offset or `Z`. +// +// This exists because the server canonicalizes every top-level DateTime value +// to RFC 3339 UTC before publishing (#402), so a payload reads `...T04:00:00Z` +// while a caller's filter constant may name the same instant as +// `...T06:00:00+02:00`. Comparing those as text is wrong in both directions: +// lexically the payload sorts *below* the constant, so `gte` misses a row that +// is chronologically equal. The server compares DateTime columns as instants +// for exactly this reason; this is the client-side twin of that rule. +// +// Deliberately narrow. A zone-less spelling ("2026-06-21 04:00:00") names an +// instant only relative to the column's declared timezone, which the server +// reads from the schema and a stream subscriber does not have. Guessing UTC +// would move the instant, so those fail to parse here and fall through to text +// comparison rather than being silently reinterpreted. +func asInstant(v any) (time.Time, bool) { + s, ok := v.(string) + if !ok || len(s) > maxTimeOperandChars { + return time.Time{}, false + } + // ClickHouse has no ',' decimal separator, but Go's RFC3339Nano accepts one + // per ISO 8601. Reject it so the client can't admit a spelling the server + // would refuse. + if strings.ContainsRune(s, ',') { + return time.Time{}, false + } + t, err := time.Parse(time.RFC3339Nano, s) + if err != nil { + return time.Time{}, false + } + return t, true +} + +// evaluateIn checks whether actual is contained in the expected slice. +// Reflection handles []any and typed slices (e.g., []string, []int) alike. +func evaluateIn(actual, expected any) bool { + rv := reflect.ValueOf(expected) + if rv.Kind() != reflect.Slice { + return false + } + for i := range rv.Len() { + if equalValues(actual, rv.Index(i).Interface()) { + return true + } + } + return false +} + +// compareOrdered returns (-1, 0, or 1) and true for comparable ordered types, +// or (0, false) when the types cannot be compared. +func compareOrdered(actual, expected any) (int, bool) { + if a, aOK := toFloat64(actual); aOK { + if b, bOK := toFloat64(expected); bOK { + switch { + case a < b: + return -1, true + case a > b: + return 1, true + default: + return 0, true + } + } + } + // Timestamps compare chronologically, not lexically. If either side names + // an instant the other must too: ordering a canonicalized payload against a + // spelling that isn't a provable instant is meaningless, and text + // comparison there would admit rows the query path excludes. Fail closed, + // as the server's row filter does for a DateTime column. + aTime, aIsTime := asInstant(actual) + bTime, bIsTime := asInstant(expected) + if aIsTime || bIsTime { + if !aIsTime || !bIsTime { + return 0, false + } + switch { + case aTime.Before(bTime): + return -1, true + case aTime.After(bTime): + return 1, true + default: + return 0, true + } + } + if aStr, ok := actual.(string); ok { + if bStr, ok := expected.(string); ok { + switch { + case aStr < bStr: + return -1, true + case aStr > bStr: + return 1, true + default: + return 0, true + } + } + } + return 0, false +} + +func toFloat64(v any) (float64, bool) { + switch n := v.(type) { + case float64: + return n, true + case float32: + return float64(n), true + case json.Number: + f, err := n.Float64() + return f, err == nil + } + // All int/uint widths in two cases (codegen structs use the narrow ones). + rv := reflect.ValueOf(v) + switch { + case rv.CanInt(): + return float64(rv.Int()), true + case rv.CanUint(): + return float64(rv.Uint()), true + } + return 0, false +} + +func projectColumns(row map[string]any, columns []string) map[string]any { + result := make(map[string]any, len(columns)) + for _, col := range columns { + if v, ok := row[col]; ok { + result[col] = v + } + } + return result +} diff --git a/clients/go/stream_test.go b/clients/go/stream_test.go new file mode 100644 index 00000000..533c14f2 --- /dev/null +++ b/clients/go/stream_test.go @@ -0,0 +1,770 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/http/httptest" + "strings" + "sync" + "sync/atomic" + "testing" + "time" +) + +// sseServer serves the given SSE frames on any request, then holds the +// connection open until the client disconnects. +func sseServer(t *testing.T, frames []string) *httptest.Server { + t.Helper() + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + fl, ok := w.(http.Flusher) + if !ok { + t.Error("response writer is not a flusher") + return + } + w.WriteHeader(200) + fl.Flush() + for _, f := range frames { + _, _ = io.WriteString(w, f) + fl.Flush() + } + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + return srv +} + +func sseFrame(ts, page string) string { + return "id: " + ts + "\n" + + `data: {"table_name":"clicks","received_timestamp":"` + ts + `","data":{"page":"` + page + `"}}` + + "\n\n" +} + +func streamClient(t *testing.T, srv *httptest.Server) *Client { + t.Helper() + return NewClient(Config{BaseURL: srv.URL, HTTPClient: srv.Client()}) +} + +func TestStream_SubscribeReceivesEvents(t *testing.T) { + srv := sseServer(t, []string{ + sseFrame("2026-01-01T00:00:01Z", "/home"), + sseFrame("2026-01-01T00:00:02Z", "/about"), + }) + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + got := make(chan StreamEvent, 8) + stream.Subscribe(&StreamSubscriber{ + Next: func(e StreamEvent) { got <- e }, + }) + + e1 := recvEvent(t, got) + if e1.Table != "clicks" || e1.Data["page"] != "/home" { + t.Fatalf("unexpected first event: %+v", e1) + } + e2 := recvEvent(t, got) + if e2.Data["page"] != "/about" || e2.Timestamp != "2026-01-01T00:00:02Z" { + t.Fatalf("unexpected second event: %+v", e2) + } +} + +func recvEvent(t *testing.T, ch <-chan StreamEvent) StreamEvent { + t.Helper() + select { + case e := <-ch: + return e + case <-time.After(5 * time.Second): + t.Fatal("timed out waiting for stream event") + return StreamEvent{} + } +} + +func TestStream_EventsChannel(t *testing.T) { + srv := sseServer(t, []string{sseFrame("2026-01-01T00:00:01Z", "/home")}) + stream := streamClient(t, srv).From("clicks").Stream(nil) + + ch := stream.Events() + e := recvEvent(t, ch) + if e.Data["page"] != "/home" { + t.Fatalf("unexpected event: %+v", e) + } + + stream.Close() + select { + case _, open := <-ch: + if open { + // A buffered event may arrive before close; drain once more. + if _, open2 := <-ch; open2 { + t.Fatal("events channel not closed after Close") + } + } + case <-time.After(5 * time.Second): + t.Fatal("events channel never closed after Close") + } +} + +func TestStream_Connected(t *testing.T) { + srv := sseServer(t, nil) + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := stream.Connected(ctx); err != nil { + t.Fatalf("Connected: %v", err) + } + if s := stream.Status(); s != StatusLive { + t.Fatalf("want live, got %s", s) + } +} + +func TestStream_FilteredDeliversMatchesAndProjects(t *testing.T) { + srv := sseServer(t, []string{ + sseFrame("2026-01-01T00:00:01Z", "/home"), + sseFrame("2026-01-01T00:00:02Z", "/miss"), + sseFrame("2026-01-01T00:00:03Z", "/home"), + }) + stream := streamClient(t, srv).From("clicks"). + Select("page"). + Where("page", OpEq, "/home"). + Stream(nil) + defer stream.Close() + + got := make(chan StreamEvent, 8) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { got <- e }}) + + for range 2 { + e := recvEvent(t, got) + if e.Data["page"] != "/home" { + t.Fatalf("filter leaked event: %+v", e) + } + if len(e.Data) != 1 { + t.Fatalf("projection kept extra columns: %+v", e.Data) + } + } + select { + case e := <-got: + t.Fatalf("unexpected third event: %+v", e) + case <-time.After(100 * time.Millisecond): + } +} + +// TestStream_FilteredCloseUnderLoad exercises the wrapper-close path while the +// inner stream is still delivering — the send-on-closed-channel regression. +func TestStream_FilteredCloseUnderLoad(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Content-Type", "text/event-stream") + fl := w.(http.Flusher) + w.WriteHeader(200) + fl.Flush() + for i := 0; ; i++ { + select { + case <-r.Context().Done(): + return + default: + } + _, err := io.WriteString(w, sseFrame(fmt.Sprintf("2026-01-01T00:00:%02dZ", i%60), "/home")) + if err != nil { + return + } + fl.Flush() + } + })) + t.Cleanup(srv.Close) + + stream := streamClient(t, srv).From("clicks"). + Select(). + Where("page", OpEq, "/home"). + Stream(nil) + + var n atomic.Int64 + stream.Subscribe(&StreamSubscriber{Next: func(StreamEvent) { n.Add(1) }}) + // Also exercise the Events() channel feed path during close. + _ = stream.Events() + + deadline := time.Now().Add(5 * time.Second) + for n.Load() < 10 && time.Now().Before(deadline) { + time.Sleep(5 * time.Millisecond) + } + if n.Load() == 0 { + t.Fatal("no events delivered before close") + } + stream.Close() // must not panic or race with in-flight emits + + select { + case <-stream.done: + case <-time.After(5 * time.Second): + t.Fatal("filtered stream goroutine never exited") + } +} + +func TestStream_HandleMalformedSSEData(t *testing.T) { + sc := &StreamController{eventCh: make(chan StreamEvent, 1)} + var gotErr error + sc.subscribers = []*StreamSubscriber{{Error: func(err error) { gotErr = err }}} + sc.handleSSEData("not json") // must not panic or emit + select { + case e := <-sc.eventCh: + t.Fatalf("malformed data emitted event: %+v", e) + default: + } + if gotErr == nil || !strings.Contains(gotErr.Error(), "malformed SSE message") { + t.Fatalf("want malformed-SSE error via subscriber, got %v", gotErr) + } +} + +// TestStream_EventsBufferBeforeFirstEventsCall pins TS parity: the channel +// buffers from construction, so events emitted before the first Events() call +// are still delivered once the consumer starts reading. +func TestStream_EventsBufferBeforeFirstEventsCall(t *testing.T) { + sc := &StreamController{eventCh: make(chan StreamEvent, 256)} + sc.emitEvent(StreamEvent{Table: "clicks", Data: map[string]any{"page": "/"}}) + + select { + case e := <-sc.Events(): + if e.Table != "clicks" { + t.Fatalf("want event for clicks, got %+v", e) + } + default: + t.Fatal("event emitted before Events() was not buffered") + } +} + +// --------------------------------------------------------------------------- +// Client-side filter engine +// --------------------------------------------------------------------------- + +func TestEvaluateFilter(t *testing.T) { + tests := []struct { + name string + actual any + op string + expected any + want bool + }{ + {"EqNumericCrossType", float64(10), "eq", 10, true}, + {"EqString", "a", "eq", "a", true}, + {"EqMismatch", "a", "eq", "b", false}, + {"Neq", "a", "neq", "b", true}, + {"GtTrue", float64(11), "gt", 10, true}, + {"GtFalse", float64(10), "gt", 10, false}, + {"Gte", float64(10), "gte", 10, true}, + {"Lt", float64(9), "lt", 10, true}, + {"LteString", "a", "lte", "b", true}, + {"GtIncomparable", "a", "gt", 10, false}, + // Narrow/unsigned codegen-struct fields must compare, not silently drop. + {"GtUnsignedOperand", float64(10), "gt", uint32(5), true}, + {"InAnySlice", "b", "in", []any{"a", "b"}, true}, + {"InTypedSlice", float64(2), "in", []int{1, 2}, true}, + {"InMiss", "c", "in", []any{"a", "b"}, false}, + {"InNotASlice", "a", "in", "a", false}, + {"Like", "hello world", "like", "hello%", true}, + {"LikeCaseInsensitive", "HELLO", "like", "hello", true}, + {"LikeUnderscore", "cat", "like", "c_t", true}, + {"LikeAnchored", "xhello", "like", "hello%", false}, + {"NotLike", "abc", "not_like", "x%", true}, + {"LikeNonString", 5, "like", "5", false}, + {"UnknownOp", "a", "regex", "a", false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + cf := compileFilters([]QueryFilter{{Column: "c", Op: tt.op, Value: tt.expected}})[0] + if got := evaluateFilter(tt.actual, tt.op, tt.expected, cf.re); got != tt.want { + t.Errorf("evaluateFilter(%v, %q, %v) = %v, want %v", tt.actual, tt.op, tt.expected, got, tt.want) + } + }) + } +} + +func TestMatchesFilters_AllMustMatch(t *testing.T) { + row := map[string]any{"page": "/home", "score": float64(10)} + both := []QueryFilter{ + {Column: "page", Op: "eq", Value: "/home"}, + {Column: "score", Op: "gt", Value: 5}, + } + if !matchesFilters(row, compileFilters(both)) { + t.Fatal("want match when every filter passes") + } + oneFails := append(append([]QueryFilter(nil), both...), QueryFilter{Column: "score", Op: "gt", Value: 99}) + if matchesFilters(row, compileFilters(oneFails)) { + t.Fatal("want no match when any filter fails") + } + if !matchesFilters(row, nil) { + t.Fatal("want match with no filters") + } +} + +func TestCompareOrdered(t *testing.T) { + if c, ok := compareOrdered(float64(1), 2); !ok || c != -1 { + t.Fatalf("numeric compare: got (%d, %v)", c, ok) + } + if c, ok := compareOrdered("b", "a"); !ok || c != 1 { + t.Fatalf("string compare: got (%d, %v)", c, ok) + } + if _, ok := compareOrdered(map[string]any{}, 1); ok { + t.Fatal("incomparable types must return ok=false") + } +} + +func TestToFloat64(t *testing.T) { + for _, v := range []any{ + float64(1), float32(1), + int(1), int8(1), int16(1), int32(1), int64(1), + uint(1), uint8(1), uint16(1), uint32(1), uint64(1), + json.Number("1"), + } { + if f, ok := toFloat64(v); !ok || f != 1 { + t.Fatalf("toFloat64(%T) = (%v, %v)", v, f, ok) + } + } + if _, ok := toFloat64("1"); ok { + t.Fatal("strings must not convert") + } +} + +func TestProjectColumns(t *testing.T) { + row := map[string]any{"a": 1, "b": 2, "c": 3} + got := projectColumns(row, []string{"a", "c", "missing"}) + if len(got) != 2 || got["a"] != 1 || got["c"] != 3 { + t.Fatalf("unexpected projection: %+v", got) + } +} + +// TestStream_NonRetryableConnectErrorIsTerminal: a 403 must close the stream +// (no infinite reconnect) and surface the API error to Error subscribers. +func TestStream_NonRetryableConnectErrorIsTerminal(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + http.Error(w, `{"error":"forbidden"}`, http.StatusForbidden) + })) + t.Cleanup(srv.Close) + + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + errCh := make(chan error, 4) + stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) || apiErr.Status != http.StatusForbidden || apiErr.Retryable { + t.Fatalf("want non-retryable HTTP_403, got %v", err) + } + case <-time.After(5 * time.Second): + t.Fatal("error never surfaced") + } + + select { + case <-stream.done: + case <-time.After(5 * time.Second): + t.Fatal("stream never closed after non-retryable connect error") + } + if s := stream.Status(); s != StatusClosed { + t.Fatalf("want closed, got %s", s) + } + + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := stream.Connected(ctx); err == nil { + t.Fatal("Connected must fail on a terminally-closed stream") + } +} + +// TestStream_ReconnectResumesFromLastEventID: the gap-fill contract. The +// initial request carries StreamOptions.Since; after the connection drops, +// the reconnect carries ?since=. +func TestStream_ReconnectResumesFromLastEventID(t *testing.T) { + var mu sync.Mutex + var sinceParams []string + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + sinceParams = append(sinceParams, r.URL.Query().Get("since")) + n := len(sinceParams) + mu.Unlock() + + w.Header().Set("Content-Type", "text/event-stream") + fl := w.(http.Flusher) + w.WriteHeader(200) + fl.Flush() + if n == 1 { + _, _ = io.WriteString(w, sseFrame("2026-01-01T00:00:01Z", "/home")) + fl.Flush() + return // server closes → client must reconnect with since= + } + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + + stream := streamClient(t, srv).From("clicks").Stream(&StreamOptions{Since: "seed-id"}) + defer stream.Close() + + got := make(chan StreamEvent, 4) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { got <- e }}) + recvEvent(t, got) + + // Reconnect happens after ~backoff(0) (≈1s with jitter). + deadline := time.Now().Add(10 * time.Second) + for { + mu.Lock() + n := len(sinceParams) + mu.Unlock() + if n >= 2 { + break + } + if time.Now().After(deadline) { + t.Fatal("stream never reconnected") + } + time.Sleep(20 * time.Millisecond) + } + + mu.Lock() + defer mu.Unlock() + if sinceParams[0] != "seed-id" { + t.Fatalf("initial request: want since=seed-id, got %q", sinceParams[0]) + } + if sinceParams[1] != "2026-01-01T00:00:01Z" { + t.Fatalf("reconnect: want since=, got %q", sinceParams[1]) + } +} + +// The SSE transport builds its URL separately from buildURL (stream.go), so a +// BaseURL path prefix needs its own guard. See TestBaseURLPathPrefixIsPreserved. +func TestStreamBaseURLPathPrefixIsPreserved(t *testing.T) { + gotPath := make(chan string, 1) + mux := http.NewServeMux() + mux.HandleFunc("/api/warehouse/v1/stream", func(w http.ResponseWriter, r *http.Request) { + select { + case gotPath <- r.URL.Path: + default: + } + w.Header().Set("Content-Type", "text/event-stream") + w.WriteHeader(200) + w.(http.Flusher).Flush() + <-r.Context().Done() + }) + srv := httptest.NewServer(mux) // anything off-prefix 404s + t.Cleanup(srv.Close) + + client := NewClient(Config{ + BaseURL: srv.URL + "/api/warehouse", + Options: &ClientOptions{}, + HTTPClient: srv.Client(), + }) + sc := client.From("clicks").Stream(nil) + t.Cleanup(sc.Close) + + select { + case p := <-gotPath: + if p != "/api/warehouse/v1/stream" { + t.Fatalf("want prefixed stream path, got %q", p) + } + case <-time.After(3 * time.Second): + t.Fatal("stream never reached the prefixed path") + } +} + +// TestStream_TerminalConnectFailures: every way a connection can fail in a way +// reconnecting cannot fix. Each case must surface a specific, non-retryable +// code and close the stream — the generic retryable SSE_ERROR would spin here. +func TestStream_TerminalConnectFailures(t *testing.T) { + tests := []struct { + name string + handler http.HandlerFunc + baseURL string // overrides the test server URL when non-empty + auth func(context.Context) (string, error) + wantCode string + }{ + { + name: "200 that is not an event stream", + handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", "text/html; charset=utf-8") + w.WriteHeader(http.StatusOK) + _, _ = io.WriteString(w, "Please sign in") + }, + wantCode: "SSE_BAD_CONTENT_TYPE", + }, + { + name: "200 with no Content-Type at all", + handler: func(w http.ResponseWriter, _ *http.Request) { + w.Header()["Content-Type"] = nil + w.WriteHeader(http.StatusOK) + }, + wantCode: "SSE_BAD_CONTENT_TYPE", + }, + { + name: "credentialed request is redirected", + handler: func(w http.ResponseWriter, _ *http.Request) { + http.Redirect(w, &http.Request{}, "https://elsewhere.example/v1/stream", http.StatusFound) + }, + auth: StaticToken("secret-token"), + wantCode: "SSE_REDIRECT", + }, + { + name: "baseURL scheme cannot carry SSE", + handler: func(http.ResponseWriter, *http.Request) {}, + baseURL: "ws://example.invalid", + wantCode: "SSE_CONNECT_ERROR", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + srv := httptest.NewServer(tc.handler) + t.Cleanup(srv.Close) + + base := srv.URL + if tc.baseURL != "" { + base = tc.baseURL + } + client := NewClient(Config{BaseURL: base, Auth: tc.auth, HTTPClient: srv.Client()}) + + stream := client.From("clicks").Stream(nil) + defer stream.Close() + + errCh := make(chan error, 4) + stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) { + t.Fatalf("want *Error, got %T: %v", err, err) + } + if apiErr.Code != tc.wantCode { + t.Fatalf("want code %s, got %s (%v)", tc.wantCode, apiErr.Code, err) + } + if apiErr.Retryable { + t.Fatalf("%s must not be retryable", apiErr.Code) + } + case <-time.After(5 * time.Second): + t.Fatal("error never surfaced") + } + + select { + case <-stream.done: + case <-time.After(5 * time.Second): + t.Fatalf("stream never closed after terminal %s", tc.wantCode) + } + }) + } +} + +// TestStream_RedirectFollowedWhenUncredentialed: the refusal is scoped to +// requests carrying a credential. Without one there is nothing to leak or +// silently downgrade, so the redirect is followed as usual. +func TestStream_RedirectFollowedWhenUncredentialed(t *testing.T) { + target := sseServer(t, []string{sseFrame("2026-01-01T00:00:01Z", "/home")}) + + front := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + http.Redirect(w, r, target.URL+"/v1/stream?table=clicks", http.StatusFound) + })) + t.Cleanup(front.Close) + + stream := streamClient(t, front).From("clicks").Stream(nil) + defer stream.Close() + + events := make(chan StreamEvent, 4) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { events <- e }}) + + select { + case e := <-events: + if e.Table != "clicks" { + t.Fatalf("want table clicks, got %s", e.Table) + } + case <-time.After(5 * time.Second): + t.Fatal("redirect was not followed for an uncredentialed stream") + } +} + +// TestStream_MalformedFrameIsTypedAndRetryable: a bad frame must arrive as an +// *Error so errors.As and IsRetryable work on it — a bare fmt.Errorf would +// leave callers string-matching. +func TestStream_MalformedFrameIsTypedAndRetryable(t *testing.T) { + srv := sseServer(t, []string{"id: 1\ndata: {not json\n\n"}) + stream := streamClient(t, srv).From("clicks").Stream(nil) + defer stream.Close() + + errCh := make(chan error, 4) + stream.Subscribe(&StreamSubscriber{Error: func(err error) { errCh <- err }}) + + select { + case err := <-errCh: + var apiErr *Error + if !errors.As(err, &apiErr) { + t.Fatalf("want *Error, got %T: %v", err, err) + } + if apiErr.Code != "SSE_PARSE_ERROR" { + t.Fatalf("want SSE_PARSE_ERROR, got %s", apiErr.Code) + } + if !IsRetryable(err) { + t.Fatal("a malformed frame must stay retryable") + } + if strings.Contains(apiErr.Message, "not json") { + t.Fatal("payload must not be echoed into the error message") + } + case <-time.After(5 * time.Second): + t.Fatal("error never surfaced") + } +} + +// TestStream_ConfiguredHeadersReachTheStream: ClientOptions.Headers apply to +// SSE, not just REST — and the SDK's own headers still win a collision. +func TestStream_ConfiguredHeadersReachTheStream(t *testing.T) { + seen := make(chan http.Header, 1) + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + select { + case seen <- r.Header.Clone(): + default: + } + w.Header().Set("Content-Type", "text/event-stream") + w.WriteHeader(http.StatusOK) + if fl, ok := w.(http.Flusher); ok { + fl.Flush() + } + <-r.Context().Done() + })) + t.Cleanup(srv.Close) + + client := NewClient(Config{ + BaseURL: srv.URL, + HTTPClient: srv.Client(), + Options: &ClientOptions{Headers: map[string]string{ + "X-Operator-Key": "op-secret", + "accept": "application/json", // must lose to the SDK's own Accept + }}, + }) + stream := client.From("clicks").Stream(nil) + defer stream.Close() + + select { + case h := <-seen: + if got := h.Get("X-Operator-Key"); got != "op-secret" { + t.Fatalf("want configured header on the stream request, got %q", got) + } + if got := h.Get("Accept"); got != "text/event-stream" { + t.Fatalf("SDK Accept must win, got %q", got) + } + case <-time.After(5 * time.Second): + t.Fatal("stream request never arrived") + } +} + +// TestEvaluateFilter_TimestampsCompareAsInstants: the server canonicalizes +// every top-level DateTime value to RFC 3339 UTC before publishing (#402), so +// a payload and a caller's filter constant routinely spell the same instant +// differently. Comparing those as text disagrees with the server's row filter, +// which compares DateTime columns chronologically. +func TestEvaluateFilter_TimestampsCompareAsInstants(t *testing.T) { + // The canonicalized payload value, and the same instant in +02:00 — which + // sorts ABOVE it lexically ("06" > "04") while being chronologically equal. + const canonical = "2026-06-21T04:00:00Z" + const sameInstantOffset = "2026-06-21T06:00:00+02:00" + const oneSecondLater = "2026-06-21T06:00:01+02:00" + + tests := []struct { + name string + actual any + op string + expected any + want bool + }{ + {"equal across offsets", canonical, "eq", sameInstantOffset, true}, + {"neq is false across offsets", canonical, "neq", sameInstantOffset, false}, + {"gte holds at the same instant", canonical, "gte", sameInstantOffset, true}, + {"lte holds at the same instant", canonical, "lte", sameInstantOffset, true}, + {"gt is false at the same instant", canonical, "gt", sameInstantOffset, false}, + {"lt sees a later offset instant", canonical, "lt", oneSecondLater, true}, + {"gt is false against a later instant", canonical, "gt", oneSecondLater, false}, + {"in matches across offsets", canonical, "in", []any{"2020-01-01T00:00:00Z", sameInstantOffset}, true}, + + // Same-offset spellings must keep working exactly as before. + {"gt within UTC", "2026-06-21T04:00:01Z", "gt", canonical, true}, + {"lt within UTC", "2026-06-21T03:59:59Z", "lt", canonical, true}, + {"eq identical text", canonical, "eq", canonical, true}, + + // Sub-second precision survives the round trip. + {"fractional seconds order correctly", "2026-06-21T04:00:00.500Z", "gt", canonical, true}, + + // A zone-less constant names an instant only relative to the column's + // declared timezone, which a stream subscriber does not have. It must + // not be silently read as UTC — ordering fails closed. + {"zone-less constant fails closed on gt", canonical, "gt", "2026-06-21 03:00:00", false}, + {"zone-less constant fails closed on lt", canonical, "lt", "2026-06-21 05:00:00", false}, + + // A ',' fraction is ISO 8601 but not ClickHouse, so it is not an instant. + {"comma fraction is not an instant", canonical, "eq", "2026-06-21T04:00:00,000Z", false}, + + // Non-timestamp strings keep lexicographic ordering. + {"plain strings still order lexically", "banana", "gt", "apple", true}, + {"plain strings still compare equal", "apple", "eq", "apple", true}, + + // Numbers are untouched by any of this. + {"numbers still order numerically", 100.0, "gt", 9.0, true}, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := evaluateFilter(tc.actual, tc.op, tc.expected, nil); got != tc.want { + t.Fatalf("evaluateFilter(%v, %q, %v) = %v, want %v", + tc.actual, tc.op, tc.expected, got, tc.want) + } + }) + } +} + +// TestEqualValues_NilOnlyEqualsNil: a column missing from the payload must not +// match the literal string "" through the fmt.Sprint fallback. +func TestEqualValues_NilOnlyEqualsNil(t *testing.T) { + tests := []struct { + name string + a, b any + want bool + }{ + {"nil equals nil", nil, nil, true}, + {"nil does not equal the string ", nil, "", false}, + {"the string does not equal nil", "", nil, false}, + {"nil does not equal empty string", nil, "", false}, + {"nil does not equal zero", nil, 0, false}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := equalValues(tc.a, tc.b); got != tc.want { + t.Fatalf("equalValues(%v, %v) = %v, want %v", tc.a, tc.b, got, tc.want) + } + }) + } +} + +// TestStream_FilterMatchesCanonicalizedPayload: the end-to-end shape of the +// same bug — a caller filters on a non-UTC spelling and the server delivers the +// canonicalized one. +func TestStream_FilterMatchesCanonicalizedPayload(t *testing.T) { + frame := `event: message +id: 2026-06-21T04:00:00Z +data: {"table_name":"clicks","received_timestamp":"2026-06-21T04:00:00Z","data":{"page":"/home","event_ts":"2026-06-21T04:00:00Z"}} + +` + srv := sseServer(t, []string{frame}) + + stream := streamClient(t, srv).From("clicks"). + SelectAll(). + Where("event_ts", OpGte, "2026-06-21T06:00:00+02:00"). + Stream(nil) + defer stream.Close() + + events := make(chan StreamEvent, 4) + stream.Subscribe(&StreamSubscriber{Next: func(e StreamEvent) { events <- e }}) + + select { + case e := <-events: + if e.Data["page"] != "/home" { + t.Fatalf("unexpected row: %v", e.Data) + } + case <-time.After(5 * time.Second): + t.Fatal("a row chronologically equal to the filter constant was withheld") + } +} diff --git a/clients/go/sys.go b/clients/go/sys.go new file mode 100644 index 00000000..ed875cdb --- /dev/null +++ b/clients/go/sys.go @@ -0,0 +1,23 @@ +package wavehouse + +import ( + "context" + "fmt" +) + +// SysNamespace provides system health checks. +type SysNamespace struct { + ctx httpContext +} + +// Health pings the server's public /v1/health endpoint. Returns nil when the +// server is reachable and past boot, or an error describing the failure. +func (s *SysNamespace) Health(ctx context.Context) error { + if err := doRequest(ctx, s.ctx, requestOptions{ + method: "GET", + path: "/v1/health", + }, nil); err != nil { + return fmt.Errorf("health check: %w", err) + } + return nil +} diff --git a/clients/go/table.go b/clients/go/table.go new file mode 100644 index 00000000..73c2ea3f --- /dev/null +++ b/clients/go/table.go @@ -0,0 +1,199 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "fmt" + "net/url" + "reflect" + "strings" +) + +// TableRef is a reference to a table. Use it for queries, inserts, schema, +// and streams. Safe for concurrent use: it holds no mutable state, and every +// builder method returns a fresh value. +type TableRef struct { + ctx httpContext + table string + createStream func(table string, opts *StreamOptions) *StreamController +} + +// Fetch is a SELECT * shortcut with a default limit of 1000. +func (t *TableRef) Fetch(ctx context.Context) (*Page[map[string]any], error) { + return t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx) +} + +// Select starts building a typed query with the given column projection. +func (t *TableRef) Select(columns ...string) *QueryBuilder { + return &QueryBuilder{ + ctx: t.ctx, + createStream: t.createStream, + state: queryState{ + table: t.table, + columns: columns, + }, + } +} + +// SelectAll starts a query that selects every column the caller's role may read. +func (t *TableRef) SelectAll() *QueryBuilder { + return t.Select().SelectAll() +} + +// Insert inserts one or more rows into this table. A single map or struct is +// sent as JSON; any slice — []map[string]any, a generated/user-defined row +// type such as []ClickRow, etc. — is serialized to NDJSON for batch ingest. +func (t *TableRef) Insert(ctx context.Context, data any) (*InsertResult, error) { + if rows, ok := data.([]map[string]any); ok { + return t.insertBatch(ctx, rows) + } + if rv, ok := sliceValue(data); ok { + return t.insertBatchReflect(ctx, rv) + } + return t.insertSingle(ctx, data) +} + +// sliceValue reports whether data is a slice type, returning its +// reflect.Value for iteration. []byte is excluded and treated as an opaque +// single value (matching encoding/json's special-cased handling of byte +// slices) rather than a batch of numbers. +func sliceValue(data any) (reflect.Value, bool) { + if data == nil { + return reflect.Value{}, false + } + if _, isBytes := data.([]byte); isBytes { + return reflect.Value{}, false + } + v := reflect.ValueOf(data) + if v.Kind() != reflect.Slice { + return reflect.Value{}, false + } + return v, true +} + +// InsertNDJSON inserts pre-formatted NDJSON (one record per line). +func (t *TableRef) InsertNDJSON(ctx context.Context, ndjson string) (*InsertResult, error) { + return t.sendNDJSON(ctx, ndjson) +} + +// Schema returns the table's column definitions from ClickHouse. Admin-only. +func (t *TableRef) Schema(ctx context.Context) (*TableSchema, error) { + var schema TableSchema + if err := doRequest(ctx, t.ctx, requestOptions{ + method: "GET", + path: "/v1/ops/schema", + params: url.Values{"table": {t.table}}, + }, &schema); err != nil { + return nil, fmt.Errorf("get schema for table %q: %w", t.table, err) + } + return &schema, nil +} + +// Stream opens a live SSE event stream for this table. +func (t *TableRef) Stream(opts *StreamOptions) *StreamController { + return t.createStream(t.table, opts) +} + +func (t *TableRef) insertSingle(ctx context.Context, data any) (*InsertResult, error) { + var res struct { + OK *bool `json:"ok"` + Duplicate *bool `json:"duplicate"` + } + if err := doRequest(ctx, t.ctx, requestOptions{ + method: "POST", + path: "/v1/ingest", + params: url.Values{"table": {t.table}}, + body: data, + }, &res); err != nil { + return nil, fmt.Errorf("insert into %q: %w", t.table, err) + } + ok := true + if res.OK != nil { + ok = *res.OK + } + result := &InsertResult{OK: ok} + if res.Duplicate != nil { + result.Duplicate = res.Duplicate + } + return result, nil +} + +func emptyInsertResult() *InsertResult { + // Separate vars, not one aliased &z: the fields are exported *int, so a + // caller writing through one would otherwise mutate all four. + total, succeeded, failed, duplicates := 0, 0, 0, 0 + return &InsertResult{OK: true, Total: &total, Succeeded: &succeeded, Failed: &failed, Duplicates: &duplicates} +} + +func marshalNDJSON(n int, elem func(int) any) (string, error) { + var sb strings.Builder + for i := range n { + if i > 0 { + sb.WriteByte('\n') + } + raw, err := json.Marshal(elem(i)) + if err != nil { + return "", fmt.Errorf("wavehouse: marshal row %d: %w", i, err) + } + sb.Write(raw) + } + return sb.String(), nil +} + +func (t *TableRef) insertBatch(ctx context.Context, rows []map[string]any) (*InsertResult, error) { + if len(rows) == 0 { + return emptyInsertResult(), nil + } + ndjson, err := marshalNDJSON(len(rows), func(i int) any { return rows[i] }) + if err != nil { + return nil, err + } + return t.sendNDJSON(ctx, ndjson) +} + +// insertBatchReflect is the fallback batch path for any slice type other +// than []map[string]any (the fast path in insertBatch above) — e.g. a +// generated or user-defined row type such as []ClickRow. Each element is +// marshaled to JSON individually and joined as NDJSON, exactly like +// insertBatch, so the server's per-record batch summary (failed, results, +// etc.) is preserved instead of being silently dropped by insertSingle. +func (t *TableRef) insertBatchReflect(ctx context.Context, rows reflect.Value) (*InsertResult, error) { + if rows.Len() == 0 { + return emptyInsertResult(), nil + } + ndjson, err := marshalNDJSON(rows.Len(), func(i int) any { return rows.Index(i).Interface() }) + if err != nil { + return nil, err + } + return t.sendNDJSON(ctx, ndjson) +} + +func (t *TableRef) sendNDJSON(ctx context.Context, ndjson string) (*InsertResult, error) { + var res struct { + Total int `json:"total"` + Succeeded int `json:"succeeded"` + Failed int `json:"failed"` + Duplicates int `json:"duplicates"` + Results []InsertRecordResult `json:"results"` + } + if err := doRequest(ctx, t.ctx, requestOptions{ + method: "POST", + path: "/v1/ingest", + params: url.Values{"table": {t.table}}, + rawBody: ndjson, + contentType: "application/x-ndjson", + }, &res); err != nil { + return nil, fmt.Errorf("ingest into %q: %w", t.table, err) + } + result := &InsertResult{ + OK: res.Failed == 0, + Total: &res.Total, + Succeeded: &res.Succeeded, + Failed: &res.Failed, + Duplicates: &res.Duplicates, + } + if len(res.Results) > 0 { + result.Results = res.Results + } + return result, nil +} diff --git a/clients/go/table_test.go b/clients/go/table_test.go new file mode 100644 index 00000000..652daf88 --- /dev/null +++ b/clients/go/table_test.go @@ -0,0 +1,243 @@ +package wavehouse + +import ( + "context" + "encoding/json" + "io" + "net/http" + "sync" + "testing" +) + +func TestTableRef_InsertSingle(t *testing.T) { + // mu guards handler captures throughout this file: the handler runs on the + // server goroutine and no happens-before edge exists via the TCP socket. + var mu sync.Mutex + var gotBody map[string]any + var gotPath string + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() + gotPath = r.URL.Path + _ = json.NewDecoder(r.Body).Decode(&gotBody) + _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) + })) + result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/home"}) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + mu.Lock() + defer mu.Unlock() + if gotPath != "/v1/ingest" { + t.Fatalf("want /v1/ingest, got %s", gotPath) + } + if gotBody["page"] != "/home" { + t.Fatalf("want page=/home, got %v", gotBody) + } +} + +func TestTableRef_InsertBatch(t *testing.T) { + var mu sync.Mutex + var gotCT string + var gotBody string + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() + gotCT = r.Header.Get("Content-Type") + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) + _ = json.NewEncoder(w).Encode(map[string]any{ + "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, + }) + })) + result, err := c.From("clicks").Insert(context.Background(), []map[string]any{ + {"page": "/a"}, + {"page": "/b"}, + }) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + mu.Lock() + defer mu.Unlock() + if gotCT != "application/x-ndjson" { + t.Fatalf("want ndjson content type, got %s", gotCT) + } + if gotBody != `{"page":"/a"}`+"\n"+`{"page":"/b"}` { + t.Fatalf("want NDJSON body, got %s", gotBody) + } +} + +// TestTableRef_InsertTypedSlice covers the P1 finding: a typed slice (e.g. a +// generated or user-defined row type such as []ClickRow) must take the batch +// NDJSON path — not fall through to insertSingle, which would send the slice +// as a single JSON body and silently ignore any per-record failures the +// server reports. +func TestTableRef_InsertTypedSlice(t *testing.T) { + type ClickRow struct { + Page string `json:"page"` + } + + var mu sync.Mutex + var gotCT string + var gotBody string + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() + gotCT = r.Header.Get("Content-Type") + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) + _ = json.NewEncoder(w).Encode(map[string]any{ + "total": 2, "succeeded": 1, "failed": 1, "duplicates": 0, + }) + })) + result, err := c.From("clicks").Insert(context.Background(), []ClickRow{ + {Page: "/a"}, + {Page: "/b"}, + }) + if err != nil { + t.Fatal(err) + } + mu.Lock() + defer mu.Unlock() + if gotCT != "application/x-ndjson" { + t.Fatalf("want ndjson content type, got %s", gotCT) + } + if gotBody != `{"page":"/a"}`+"\n"+`{"page":"/b"}` { + t.Fatalf("want NDJSON body, got %s", gotBody) + } + if result.OK { + t.Fatal("want ok=false when a batch record fails") + } + if result.Failed == nil || *result.Failed != 1 { + t.Fatalf("want failed=1, got %v", result.Failed) + } + if result.Total == nil || *result.Total != 2 { + t.Fatalf("want total=2, got %v", result.Total) + } +} + +// TestTableRef_InsertByteSliceNotBatch ensures []byte keeps going through +// insertSingle rather than being (mis)treated as a slice of per-byte rows. +func TestTableRef_InsertByteSliceNotBatch(t *testing.T) { + var mu sync.Mutex + var gotPath, gotCT, gotBody string + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() + gotPath = r.URL.Path + gotCT = r.Header.Get("Content-Type") + b, _ := io.ReadAll(r.Body) + gotBody = string(b) + _ = json.NewEncoder(w).Encode(map[string]any{"ok": true}) + })) + result, err := c.From("clicks").Insert(context.Background(), []byte(`{"page":"/home"}`)) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + mu.Lock() + defer mu.Unlock() + if gotPath != "/v1/ingest" { + t.Fatalf("want /v1/ingest, got %s", gotPath) + } + // The batch path posts to the same URL and also yields ok=true, so the + // wire format is the only thing that distinguishes them: one opaque JSON + // value vs. NDJSON of 16 per-byte rows. + if gotCT != "application/json" { + t.Fatalf("want application/json (single insert), got %q", gotCT) + } + // encoding/json base64s a []byte — documented in queries.md as a value the + // server rejects (use InsertNDJSON for raw bytes). Pinned here because it + // proves the batch path wasn't taken. + if gotBody != `"eyJwYWdlIjoiL2hvbWUifQ=="` { + t.Fatalf("want single base64 value, got %q", gotBody) + } +} + +func TestTableRef_InsertEmptyBatch(t *testing.T) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + t.Fatal("should not make a request for empty batch") + })) + result, err := c.From("clicks").Insert(context.Background(), []map[string]any{}) + if err != nil { + t.Fatal(err) + } + if !result.OK { + t.Fatal("want ok=true") + } + if result.Total == nil || *result.Total != 0 { + t.Fatal("want total=0") + } +} + +func TestTableRef_InsertNDJSON(t *testing.T) { + var mu sync.Mutex + var gotBody string + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() + raw, _ := io.ReadAll(r.Body) + gotBody = string(raw) + _ = json.NewEncoder(w).Encode(map[string]any{ + "total": 2, "succeeded": 2, "failed": 0, "duplicates": 0, + }) + })) + ndjson := `{"page":"/a"}` + "\n" + `{"page":"/b"}` + result, err := c.From("clicks").InsertNDJSON(context.Background(), ndjson) + if err != nil { + t.Fatal(err) + } + if result.Total == nil || *result.Total != 2 { + t.Fatalf("want total=2, got %v", result.Total) + } + mu.Lock() + defer mu.Unlock() + if gotBody != ndjson { + t.Fatalf("want raw NDJSON, got %s", gotBody) + } +} + +func TestTableRef_Schema(t *testing.T) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Query().Get("table") != "clicks" { + t.Errorf("want table=clicks") + } + _ = json.NewEncoder(w).Encode(TableSchema{ + Name: "clicks", + Columns: []Column{ + {Name: "page", Type: "String"}, + }, + }) + })) + schema, err := c.From("clicks").Schema(context.Background()) + if err != nil { + t.Fatal(err) + } + if schema.Name != "clicks" { + t.Fatalf("want clicks, got %s", schema.Name) + } + if len(schema.Columns) != 1 || schema.Columns[0].Name != "page" { + t.Fatalf("unexpected columns: %v", schema.Columns) + } +} + +func TestTableRef_InsertDuplicate(t *testing.T) { + c := queryTestCtx(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _ = json.NewEncoder(w).Encode(map[string]any{"duplicate": true}) + })) + result, err := c.From("clicks").Insert(context.Background(), map[string]any{"page": "/dup"}) + if err != nil { + t.Fatal(err) + } + if result.Duplicate == nil || !*result.Duplicate { + t.Fatal("want duplicate=true") + } +} diff --git a/clients/go/testdata/wire_cases.json b/clients/go/testdata/wire_cases.json new file mode 100644 index 00000000..d715b3bc --- /dev/null +++ b/clients/go/testdata/wire_cases.json @@ -0,0 +1,608 @@ +[ + { + "name": "bare query defaults to select_all", + "endpoint": "query", + "table": "clicks", + "operations": [], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "select_all": true, + "limit": 1000 + } + }, + { + "name": "select explicit columns", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page", "button"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page", "button"], + "limit": 1000 + } + }, + { + "name": "selectAll sends select_all flag", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "selectAll" } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "select_all": true, + "limit": 1000 + } + }, + { + "name": "where with eq operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "=", "/home"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "eq", "value": "/home" }], + "limit": 1000 + } + }, + { + "name": "where with neq operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "!=", "/home"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "neq", "value": "/home" }], + "limit": 1000 + } + }, + { + "name": "where with gt operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">", 10] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "gt", "value": 10 }], + "limit": 1000 + } + }, + { + "name": "where with gte operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">=", 10] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "gte", "value": 10 }], + "limit": 1000 + } + }, + { + "name": "where with lt operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", "<", 5] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "lt", "value": 5 }], + "limit": 1000 + } + }, + { + "name": "where with lte operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", "<=", 5] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "lte", "value": 5 }], + "limit": 1000 + } + }, + { + "name": "where with in operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "in", ["/home", "/about"]] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "in", "value": ["/home", "/about"] }], + "limit": 1000 + } + }, + { + "name": "where with like operator", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["page", "like", "/home%"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "page", "op": "like", "value": "/home%" }], + "limit": 1000 + } + }, + { + "name": "count aggregation", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "count", "args": ["*", "total"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], + "limit": 1000 + } + }, + { + "name": "sum aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "sum", "args": ["score", ""] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "aggregations": [{ "fn": "sum", "column": "score", "alias": "sum_score" }], + "limit": 1000 + } + }, + { + "name": "avg aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "avg", "args": ["score", ""] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "aggregations": [{ "fn": "avg", "column": "score", "alias": "avg_score" }], + "limit": 1000 + } + }, + { + "name": "min aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "min", "args": ["score", ""] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "aggregations": [{ "fn": "min", "column": "score", "alias": "min_score" }], + "limit": 1000 + } + }, + { + "name": "max aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "max", "args": ["score", ""] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "aggregations": [{ "fn": "max", "column": "score", "alias": "max_score" }], + "limit": 1000 + } + }, + { + "name": "countDistinct aggregation with default alias", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "countDistinct", "args": ["page", ""] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "aggregations": [{ "fn": "countDistinct", "column": "page", "alias": "count_distinct_page" }], + "limit": 1000 + } + }, + { + "name": "custom aggregate function", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "aggregate", "args": ["uniqExact", "user_id", "unique_users"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "aggregations": [{ "fn": "uniqExact", "column": "user_id", "alias": "unique_users" }], + "limit": 1000 + } + }, + { + "name": "groupBy", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "groupBy", "args": ["page"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "group_by": ["page"], + "limit": 1000 + } + }, + { + "name": "orderBy ascending (default)", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "orderBy", "args": ["page", "asc"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "order_by": [{ "column": "page", "dir": "asc" }], + "limit": 1000 + } + }, + { + "name": "orderBy descending", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "orderBy", "args": ["score", "desc"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "order_by": [{ "column": "score", "dir": "desc" }], + "limit": 1000 + } + }, + { + "name": "explicit limit", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "limit", "args": [50] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "limit": 50 + } + }, + { + "name": "timeRange with since only", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "time_range": { "column": "received_timestamp", "since": "1h" }, + "limit": 1000 + } + }, + { + "name": "timeRange with since and until", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "timeRange", "args": ["ts", "2026-01-01", "2026-02-01"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "time_range": { "column": "ts", "since": "2026-01-01", "until": "2026-02-01" }, + "limit": 1000 + } + }, + { + "name": "multiple where clauses (AND)", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">", 10] }, + { "method": "where", "args": ["page", "=", "/home"] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "filters": [ + { "column": "score", "op": "gt", "value": 10 }, + { "column": "page", "op": "eq", "value": "/home" } + ], + "limit": 1000 + } + }, + { + "name": "complex query combining everything", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "where", "args": ["score", ">", 10] }, + { "method": "count", "args": ["*", "total"] }, + { "method": "groupBy", "args": ["page"] }, + { "method": "orderBy", "args": ["total", "desc"] }, + { "method": "limit", "args": [50] }, + { "method": "timeRange", "args": ["received_timestamp", "1h", ""] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "filters": [{ "column": "score", "op": "gt", "value": 10 }], + "aggregations": [{ "fn": "count", "column": "*", "alias": "total" }], + "group_by": ["page"], + "order_by": [{ "column": "total", "dir": "desc" }], + "limit": 50, + "time_range": { "column": "received_timestamp", "since": "1h" } + } + }, + { + "name": "insert single row path", + "endpoint": "ingest", + "table": "clicks", + "operations": [ + { "method": "insert", "args": [{ "page": "/home", "button": "cta" }] } + ], + "expected_path": "/v1/ingest?table=clicks", + "expected_method": "POST", + "expected_content_type": "application/json", + "expected_body": { "page": "/home", "button": "cta" } + }, + { + "name": "insert batch as NDJSON", + "endpoint": "ingest_batch", + "table": "clicks", + "operations": [ + { + "method": "insert", + "args": [[{ "page": "/a" }, { "page": "/b" }]] + } + ], + "expected_path": "/v1/ingest?table=clicks", + "expected_method": "POST", + "expected_content_type": "application/x-ndjson", + "expected_raw_body": "{\"page\":\"/a\"}\n{\"page\":\"/b\"}" + }, + { + "name": "pipe execution", + "endpoint": "pipe", + "pipe_name": "top_pages", + "pipe_params": { "limit": 10 }, + "expected_path": "/v1/pipes/top_pages", + "expected_method": "POST", + "expected_body": { "limit": 10 } + }, + { + "name": "pipe execution with no params sends empty object", + "endpoint": "pipe", + "pipe_name": "simple", + "pipe_params": null, + "expected_path": "/v1/pipes/simple", + "expected_method": "POST", + "expected_body": {} + }, + { + "name": "raw SQL", + "endpoint": "sql", + "sql": "SELECT count() FROM clicks", + "expected_path": "/v1/ops/query", + "expected_method": "POST", + "expected_body": { "sql": "SELECT count() FROM clicks" } + }, + { + "name": "health check", + "endpoint": "health", + "expected_path": "/v1/health", + "expected_method": "GET" + }, + { + "name": "schema list", + "endpoint": "schema_list", + "expected_path": "/v1/ops/schema", + "expected_method": "GET" + }, + { + "name": "schema refresh", + "endpoint": "schema_refresh", + "expected_path": "/v1/ops/schema/refresh", + "expected_method": "POST" + }, + { + "name": "policy get", + "endpoint": "policy_get", + "expected_path": "/v1/ops/policy", + "expected_method": "GET" + }, + { + "name": "table with special characters URL-encodes correctly", + "endpoint": "query", + "table": "my table", + "operations": [ + { "method": "select", "args": ["page"] }, + { "method": "limit", "args": [10] } + ], + "expected_path": "/v1/query?table=my+table", + "expected_method": "POST", + "expected_body": { + "columns": ["page"], + "limit": 10 + } + }, + { + "name": "DLQ list", + "endpoint": "dlq_list", + "expected_path": "/v1/ops/dlq/stats", + "expected_method": "GET" + }, + { + "name": "DLQ table filter", + "endpoint": "dlq_table", + "table": "events", + "expected_path": "/v1/ops/dlq/stats?table=events", + "expected_method": "GET" + }, + { + "name": "policy set", + "endpoint": "policy_set", + "policy_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + }, + "expected_path": "/v1/ops/policy", + "expected_method": "PUT", + "expected_content_type": "application/json", + "expected_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + } + }, + { + "name": "policy validate", + "endpoint": "policy_validate", + "policy_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + }, + "expected_path": "/v1/ops/policy/validate", + "expected_method": "POST", + "expected_content_type": "application/json", + "expected_body": { + "default_role": "viewer", + "tables": { + "events": {} + } + } + }, + { + "name": "pipes list", + "endpoint": "pipes_list", + "expected_path": "/v1/ops/pipes", + "expected_method": "GET" + }, + { + "name": "pipes get", + "endpoint": "pipes_get", + "pipe_name": "my_pipe", + "expected_path": "/v1/ops/pipes/my_pipe", + "expected_method": "GET" + }, + { + "name": "pipes set", + "endpoint": "pipes_set", + "pipe_name": "my_pipe", + "pipe_def": { + "sql": "SELECT page, count() AS views FROM events GROUP BY page", + "parameters": [ + { "name": "limit", "type": "Int32", "default": 100 } + ], + "description": "Top pages by view count" + }, + "expected_path": "/v1/ops/pipes/my_pipe", + "expected_method": "PUT", + "expected_content_type": "application/json", + "expected_body": { + "sql": "SELECT page, count() AS views FROM events GROUP BY page", + "parameters": [ + { "name": "limit", "type": "Int32", "default": 100 } + ], + "description": "Top pages by view count" + } + }, + { + "name": "pipes delete", + "endpoint": "pipes_delete", + "pipe_name": "my_pipe", + "expected_path": "/v1/ops/pipes/my_pipe", + "expected_method": "DELETE" + }, + { + "name": "cacheTTL is client-side only and not sent on the wire", + "endpoint": "query", + "table": "clicks", + "operations": [ + { "method": "cacheTTL", "args": [60] }, + { "method": "limit", "args": [5] } + ], + "expected_path": "/v1/query?table=clicks", + "expected_method": "POST", + "expected_body": { + "select_all": true, + "limit": 5 + } + } +] diff --git a/clients/go/types.go b/clients/go/types.go new file mode 100644 index 00000000..1257e446 --- /dev/null +++ b/clients/go/types.go @@ -0,0 +1,246 @@ +package wavehouse + +import "context" + +// ── Structured query AST (matches backend wire format) ──────────────────── + +// StructuredQuery is the wire format for POST /v1/query. +type StructuredQuery struct { + // Columns to project. A literal "*" is a column named "*", not a wildcard. + // Omitting columns (with no aggregations and no select_all) selects nothing. + Columns []string `json:"columns,omitempty"` + // SelectAll requests every column the caller's role may read. + // Mutually exclusive with a non-empty Columns list. + SelectAll bool `json:"select_all,omitempty"` + // Aggregations (count, sum, avg, etc.). + Aggregations []Aggregation `json:"aggregations,omitempty"` + // Filters (WHERE conditions, ANDed). + Filters []QueryFilter `json:"filters,omitempty"` + // GroupBy columns. + GroupBy []string `json:"group_by,omitempty"` + // OrderBy clauses. + OrderBy []OrderClause `json:"order_by,omitempty"` + // Limit caps the result set. + Limit *int `json:"limit,omitempty"` + // TimeRange filters by a time window. + TimeRange *TimeRange `json:"time_range,omitempty"` +} + +// Aggregation describes a single aggregation (e.g. count, sum). +type Aggregation struct { + Fn string `json:"fn"` + Column string `json:"column"` + Alias string `json:"alias"` +} + +// QueryFilter describes a single WHERE condition. +type QueryFilter struct { + Column string `json:"column"` + Op string `json:"op"` + Value any `json:"value"` +} + +// OrderClause describes a single ORDER BY clause. +type OrderClause struct { + Column string `json:"column"` + Dir string `json:"dir"` // "asc" or "desc" +} + +// TimeRange filters by a time window on a column. +type TimeRange struct { + Column string `json:"column"` + Since string `json:"since"` + Until string `json:"until,omitempty"` +} + +// FilterOp is an SDK-facing filter operator. +type FilterOp string + +const ( + OpEq FilterOp = "=" + OpNeq FilterOp = "!=" + OpGt FilterOp = ">" + OpGte FilterOp = ">=" + OpLt FilterOp = "<" + OpLte FilterOp = "<=" + OpIn FilterOp = "in" + OpLike FilterOp = "like" + OpNotLike FilterOp = "not_like" +) + +// opMap translates SDK operators to backend wire tokens. +var opMap = map[FilterOp]string{ + OpEq: "eq", + OpNeq: "neq", + OpGt: "gt", + OpGte: "gte", + OpLt: "lt", + OpLte: "lte", + OpIn: "in", + OpLike: "like", + OpNotLike: "not_like", +} + +// ── Schema types ────────────────────────────────────────────────────────── + +// Column describes a single column in a table schema. +type Column struct { + Name string `json:"name"` + Type string `json:"type"` + IsNullable bool `json:"is_nullable"` + HasDefault bool `json:"has_default"` +} + +// TableSchema describes a table's schema. +type TableSchema struct { + Name string `json:"name"` + Columns []Column `json:"columns"` +} + +// Schemas maps table names to their schemas. +type Schemas map[string]TableSchema + +// ── Insert result ───────────────────────────────────────────────────────── + +// InsertRecordResult is a per-record outcome from a batch insert. +type InsertRecordResult struct { + Index int `json:"index"` + OK *bool `json:"ok,omitempty"` + Duplicate *bool `json:"duplicate,omitempty"` + Error string `json:"error,omitempty"` +} + +// InsertResult is the outcome of an insert operation. +type InsertResult struct { + OK bool `json:"ok"` + Duplicate *bool `json:"duplicate,omitempty"` + Total *int `json:"total,omitempty"` + Succeeded *int `json:"succeeded,omitempty"` + Failed *int `json:"failed,omitempty"` + Duplicates *int `json:"duplicates,omitempty"` + Results []InsertRecordResult `json:"results,omitempty"` +} + +// ── DLQ types ───────────────────────────────────────────────────────────── + +// DLQStats describes dead-letter-queue statistics. +type DLQStats struct { + Tables map[string]int `json:"tables"` + Total int `json:"total"` +} + +// ── Pipe types ──────────────────────────────────────────────────────────── + +// Pipe describes a named query pipe definition. +type Pipe struct { + Name string `json:"name"` + SQL string `json:"sql"` + Parameters []ParamDef `json:"parameters,omitempty"` + Description string `json:"description,omitempty"` + AllowedRoles []string `json:"allowed_roles,omitempty"` +} + +// ParamDef describes a pipe parameter. +type ParamDef struct { + Name string `json:"name"` + Type string `json:"type"` + Required bool `json:"required,omitempty"` + Default any `json:"default,omitempty"` +} + +// ── Policy types ────────────────────────────────────────────────────────── + +// Policy describes the server's access-control policy. +type Policy struct { + DefaultRole string `json:"default_role,omitempty"` + // AdminRole is the role granted full access and the allowlist bypass. + // Empty means the server's default ("admin") applies. + AdminRole string `json:"admin_role,omitempty"` + Tables map[string]TablePolicy `json:"tables"` +} + +// TablePolicy describes per-table access control. +type TablePolicy struct { + Select map[string]RolePermissions `json:"select,omitempty"` + Insert map[string]RolePermissions `json:"insert,omitempty"` +} + +// RolePermissions describes a role's access to a table. +type RolePermissions struct { + AllowColumns []string `json:"allow_columns,omitempty"` + DenyColumns []string `json:"deny_columns,omitempty"` + Filter map[string]PolicyFilter `json:"filter,omitempty"` + Check map[string]PolicyFilter `json:"check,omitempty"` + AllowedAggregations []string `json:"allowed_aggregations,omitempty"` + DeniedAggregations []string `json:"denied_aggregations,omitempty"` + MaxRows *int `json:"max_rows,omitempty"` + MaxExecutionTime any `json:"max_execution_time,omitempty"` + MaxRowsToRead *int64 `json:"max_rows_to_read,omitempty"` + MaxMemoryUsage any `json:"max_memory_usage,omitempty"` +} + +// PolicyFilter describes a policy filter predicate. Fields are pointers with +// omitempty so an intentional empty-string comparison (e.g. Eq pointing at "") +// is sent as "", while an unset operator is omitted entirely — never null — +// matching the server's absent-operator semantics. +type PolicyFilter struct { + Eq *string `json:"_eq,omitempty"` + Neq *string `json:"_neq,omitempty"` + Gt *string `json:"_gt,omitempty"` + Lt *string `json:"_lt,omitempty"` + In *string `json:"_in,omitempty"` +} + +// ValidationResult is the response from policy validation. +type ValidationResult struct { + Valid bool `json:"valid"` +} + +// ── Streaming types ─────────────────────────────────────────────────────── + +// StreamStatus represents the connection state of a stream. +type StreamStatus string + +const ( + StatusConnecting StreamStatus = "connecting" + StatusLive StreamStatus = "live" + StatusReconnecting StreamStatus = "reconnecting" + StatusClosed StreamStatus = "closed" +) + +// StreamEvent is a single event from an SSE stream. +type StreamEvent struct { + Table string `json:"table"` + Timestamp string `json:"timestamp"` + Data map[string]any `json:"data"` +} + +// StreamSubscriber receives events from a stream. +type StreamSubscriber struct { + // Initial is called once with historical backfill data (live queries only). + Initial func(rows []map[string]any, err error) + // Next is called for each live event. + Next func(event StreamEvent) + // Status is called when the connection status changes. + Status func(status StreamStatus) + // Error is called on stream errors. + Error func(err error) +} + +// StreamOptions configures a stream. +type StreamOptions struct { + // Since is an RFC3339 timestamp for gap-fill replay. + Since string +} + +// ── Fetch/page types ────────────────────────────────────────────────────── + +// Page wraps a result set with pagination metadata. +type Page[T any] struct { + // Data is the result rows. + Data []T + // HasMore is true if more rows may be available. + HasMore bool + // Next fetches the next page. Nil when no cursor is available. + Next func(ctx context.Context) (*Page[T], error) +} diff --git a/clients/go/wavehouse.go b/clients/go/wavehouse.go new file mode 100644 index 00000000..3698aedd --- /dev/null +++ b/clients/go/wavehouse.go @@ -0,0 +1,159 @@ +// Package wavehouse is the official Go SDK for WaveHouse — a schema-aware +// real-time API gateway for ClickHouse. Zero third-party runtime dependencies. +// +// Create a client with [NewClient], then use [Client.From] for table +// operations, [Client.Pipe] for named queries, or the admin namespaces +// ([Client.Schema], [Client.Policy], etc.) for management. +// +// client := wavehouse.NewClient(wavehouse.Config{ +// BaseURL: "http://localhost:8080", +// }) +// rows, err := client.From("clicks").SelectAll().FetchUntyped(ctx) +package wavehouse + +import ( + "context" + "fmt" + "net/http" + "strings" +) + +// Config configures a [Client]. +type Config struct { + // BaseURL of the WaveHouse server (e.g. "http://localhost:8080"). + BaseURL string + + // Auth provides a bearer token for authenticated requests. Called before + // each request; return "" to skip the Authorization header. Nil means + // unauthenticated access (the server falls back to default_role). + Auth func(ctx context.Context) (string, error) + + // Options tunes transport behavior. + Options *ClientOptions + + // HTTPClient overrides the default http.Client. Useful for custom TLS, + // proxies, or test transports. + HTTPClient *http.Client +} + +// ClientOptions tunes transport behavior. +type ClientOptions struct { + // MaxRetries is the maximum number of retry attempts for retryable errors. + // Total attempts = MaxRetries + 1. Default: 2. + MaxRetries int + + // Headers are sent on every request the client makes — REST calls and SSE + // streams alike. Use them for a gateway credential, a tenant selector, or + // tracing metadata that has no first-class option. + // + // The SDK's own headers win: Authorization, Accept, Content-Type, and the + // stream's Cache-Control are set after these and overwrite any entry that + // collides. Names are matched case-insensitively (canonicalized by + // net/http), and each entry replaces rather than appends. + Headers map[string]string +} + +// StaticToken returns an Auth function that always returns the same token. +// Convenience for cases where the token doesn't rotate. +func StaticToken(token string) func(context.Context) (string, error) { + return func(context.Context) (string, error) { return token, nil } +} + +// Client is the WaveHouse SDK entry point. +type Client struct { + ctx httpContext + + // Schema provides admin-only schema introspection. + Schema *SchemaNamespace + // Policy provides admin-only access-control policy management. + Policy *PolicyNamespace + // DLQ provides admin-only dead-letter-queue statistics. + DLQ *DLQNamespace + // Sys provides system health checks. + Sys *SysNamespace + // Pipes provides admin-only named-pipe management. + Pipes *PipesNamespace +} + +// NewClient creates a new WaveHouse client. +func NewClient(cfg Config) *Client { + maxRetries := 2 + if cfg.Options != nil && cfg.Options.MaxRetries >= 0 { + maxRetries = cfg.Options.MaxRetries + } + + // Copy so a later mutation of the caller's map can't reach into requests. + var headers map[string]string + if cfg.Options != nil && len(cfg.Options.Headers) > 0 { + headers = make(map[string]string, len(cfg.Options.Headers)) + for k, v := range cfg.Options.Headers { + headers[k] = v + } + } + + hc := cfg.HTTPClient + if hc == nil { + // Not http.DefaultClient: it's mutable global state another package + // could reconfigure (timeout, transport, redirects) after we're built. + hc = &http.Client{} + } + + c := &Client{ + ctx: httpContext{ + baseURL: strings.TrimRight(cfg.BaseURL, "/"), + auth: cfg.Auth, + maxRetries: maxRetries, + httpClient: hc, + headers: headers, + }, + } + + c.Schema = &SchemaNamespace{ctx: c.ctx} + c.Policy = &PolicyNamespace{ctx: c.ctx} + c.DLQ = &DLQNamespace{ctx: c.ctx, createStream: c.createStream} + c.Sys = &SysNamespace{ctx: c.ctx} + c.Pipes = &PipesNamespace{ctx: c.ctx} + + return c +} + +// From returns a reference to a table for queries, inserts, and streams. +func (c *Client) From(table string) *TableRef { + return &TableRef{ + ctx: c.ctx, + table: table, + createStream: c.createStream, + } +} + +// Pipe returns a reference to a named query pipe. Pass params for the pipe's +// template parameters. +func (c *Client) Pipe(name string, params map[string]any) *PipeRef { + return &PipeRef{ + ctx: c.ctx, + name: name, + params: params, + createStream: c.createStream, + } +} + +// SQL executes a raw SQL query against ClickHouse. Requires the admin role. +// The server proxies the SQL verbatim to ClickHouse's HTTP interface. Results +// are decoded into []T; use [map[string]any] for dynamic schemas. +func SQL[Row any](ctx context.Context, c *Client, query string) ([]Row, error) { + var rows []Row + err := doRequest(ctx, c.ctx, requestOptions{ + method: "POST", + path: "/v1/ops/query", + body: map[string]string{"sql": query}, + }, &rows) + if err != nil { + return nil, fmt.Errorf("sql query: %w", err) + } + return rows, nil +} + +// createStream opens an SSE stream for the given table. +func (c *Client) createStream(table string, opts *StreamOptions) *StreamController { + return newStreamController(c.ctx, table, opts) +} diff --git a/docs/src/config/sidebar.ts b/docs/src/config/sidebar.ts index b3a61a20..b4ec5da3 100644 --- a/docs/src/config/sidebar.ts +++ b/docs/src/config/sidebar.ts @@ -25,10 +25,9 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ items: [ { label: "API Reference", slug: "api" }, { - // Topic-first SDK pages: when a second SDK language lands, these - // shared usage pages grow code tabs and each - // language gets its own setup/caveats page — the topic URLs never - // churn (decision in PR #313). + // Separate trees per SDK — API shapes diverge enough that shared + // prose reads worse than dedicated pages. Revisit with tabs if a + // third language lands. label: "TypeScript SDK", items: [ { label: "Overview", slug: "sdk" }, @@ -39,6 +38,17 @@ export const sidebar: StarlightUserConfig["sidebar"] = [ { label: "Reference & CLI", slug: "sdk/reference" }, ], }, + { + label: "Go SDK", + items: [ + { label: "Overview", slug: "sdk/go" }, + { label: "Queries", slug: "sdk/go/queries" }, + { label: "Streaming & Live Queries", slug: "sdk/go/streaming" }, + { label: "Pipes", slug: "sdk/go/pipes" }, + { label: "Admin & System", slug: "sdk/go/admin" }, + { label: "Reference & CLI", slug: "sdk/go/reference" }, + ], + }, ], }, { diff --git a/docs/src/content/docs/404.md b/docs/src/content/docs/404.md index ed74af3a..0dd4e786 100644 --- a/docs/src/content/docs/404.md +++ b/docs/src/content/docs/404.md @@ -46,6 +46,7 @@ head: ArchitectureHow the pieces fit together API referenceEndpoints, payloads, and error semantics TypeScript SDKTyped client for browser and Node + Go SDKTyped client for Go services

Followed a link that should have worked? File an issue — broken links are bugs.

diff --git a/docs/src/content/docs/access-control.mdx b/docs/src/content/docs/access-control.mdx index 56564f80..7a42189c 100644 --- a/docs/src/content/docs/access-control.mdx +++ b/docs/src/content/docs/access-control.mdx @@ -184,7 +184,7 @@ The rules, in order: 2. **An empty (or `["*"]`) `allow_columns` means "all columns"** — every column not in `deny_columns` is permitted. Use this with `deny_columns` for a blocklist posture: see everything *except* a few sensitive columns. 3. **A non-empty `allow_columns` is an allowlist** — only the named columns (and never the denied ones) are permitted. -On a structured query (`POST /v1/query?table={table}`) the allowlist is a **hard cap on every column the query references — in any clause**: the projection, an aggregation argument, `filters`, `group_by`, `order_by`, and `time_range`. Naming a disallowed column anywhere is rejected with `403 column "x" not allowed`. A full-row read is requested explicitly with `"select_all": true`, which expands to exactly the columns the role may read — never a raw `SELECT *` that could include a denied column; if the role is allowed *no* columns, the read is rejected (`403`) rather than returning empty rows. **Omitting `columns` (or sending `[]` / `""`) returns nothing** — a request for no data — so a hidden column can't leak by being left out, grouped on, or filtered on to infer its values. (Note: in a query, `["*"]` is the *literal column named `*`*, not a wildcard — use `select_all` for all columns. In `allow_columns`, `["*"]` is still the all-columns wildcard.) On insert (`POST /v1/ingest?table={table}`) the body is `403 column "x" not allowed for insert`. On live streams, denied columns are silently **stripped** from each event rather than rejecting the connection. The structured-query and live-stream paths defer to the **same** per-column decision (`IsColumnAllowed`), so the two read surfaces enforce identical column visibility and can't drift apart. +On a structured query (`POST /v1/query?table={table}`) the allowlist is a **hard cap on every column the query references — in any clause**: the projection, an aggregation argument, `filters`, `group_by`, `order_by`, and `time_range`. Naming a disallowed column anywhere is rejected with `403 column "x" not allowed`. A full-row read is requested explicitly with `"select_all": true`; for a column-restricted role it expands to exactly the columns the role may read rather than a bare `SELECT *` that could include a denied column (an unrestricted/admin role does get `SELECT *`); if the role is allowed *no* columns, the read is rejected (`403`) rather than returning empty rows. **Omitting `columns` (or sending `[]` / `""`) returns nothing** — a request for no data — so a hidden column can't leak by being left out, grouped on, or filtered on to infer its values. (Note: in a query, `["*"]` is the *literal column named `*`*, not a wildcard — use `select_all` for all columns. In `allow_columns`, `["*"]` is still the all-columns wildcard.) On insert (`POST /v1/ingest?table={table}`) the body is `403 column "x" not allowed for insert`. On live streams, denied columns are silently **stripped** from each event rather than rejecting the connection. The structured-query and live-stream paths defer to the **same** per-column decision (`IsColumnAllowed`), so the two read surfaces enforce identical column visibility and can't drift apart. ## Row-level security @@ -451,7 +451,7 @@ curl -X PUT http://localhost:8080/v1/ops/policy \ -d @policy.json ``` -The full request and response shapes for these endpoints live in the [API Reference](/api); the [TypeScript SDK](/sdk) wraps them as `client.policy.get()`, `client.policy.set(policy)`, and `client.policy.validate(policy)`. +The full request and response shapes for these endpoints live in the [API Reference](/api); the [TypeScript SDK](/sdk) wraps them as `client.policy.get()`, `client.policy.set(policy)`, and `client.policy.validate(policy)`, and the [Go SDK](/sdk/go/admin) as `wh.Policy.Get(ctx)`, `wh.Policy.Set(ctx, policy)`, and `wh.Policy.Validate(ctx, policy)`. ## Bootstrapping and the policy lifecycle diff --git a/docs/src/content/docs/api.md b/docs/src/content/docs/api.md index c778a82f..1e44bf72 100644 --- a/docs/src/content/docs/api.md +++ b/docs/src/content/docs/api.md @@ -152,7 +152,7 @@ Status code: `503 Service Unavailable` Returns **`200 OK` with an empty body** once the gateway is past boot, or **`503 Service Unavailable`** (also empty) while boot-time schema discovery is still failing. No authentication required and no response body — the caller only branches on the status code, so there's nothing to JSON-encode or cache per request. -This is what the SDK's `wh.sys.health()` calls, and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDK relies on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. +This is what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`), and the endpoint to use when choosing among multiple servers in a distributed setup. It mirrors `/livez` under the hood but is intentionally a `/v1` API route rather than a Kubernetes probe path: an operator may filter the bare probe paths (`/livez`, `/readyz`, `/healthz`) out at the reverse proxy since they're internal probes, so the SDKs rely on `/v1/health`, which is documented public API surface meant to stay reachable. It does **not** ping ClickHouse — readiness-based load balancing is the proxy/LB's job (via `/readyz`), not the client's. --- @@ -305,7 +305,7 @@ Examples for a `DateTime64(3, 'America/New_York')` column: `"2026-06-21 00:00:00 #### Batch Ingest -A **JSON array** of objects (`[{…}, {…}]`) or an **NDJSON** body (`Content-Type: application/x-ndjson`, one JSON object per line) ingests a batch in a single request. Each record is validated, authorized, deduplicated, and published independently, so **one malformed or rejected record never blocks the rest of the batch**. (The SDK's `insert([...])` array helper uses the NDJSON form automatically; both forms return the same response.) +A **JSON array** of objects (`[{…}, {…}]`) or an **NDJSON** body (`Content-Type: application/x-ndjson`, one JSON object per line) ingests a batch in a single request. Each record is validated, authorized, deduplicated, and published independently, so **one malformed or rejected record never blocks the rest of the batch**. (Both SDKs' array/slice insert helpers use the NDJSON form automatically; both forms return the same response.) - **JSON array** — the most convenient form from most HTTP clients. A structural JSON syntax error fails the whole request (`400`), but a wrong-typed element (a non-object) is reported per-record like any other rejection. An explicit empty array (`[]`) is a valid, record-less batch (`200`, `total: 0`). - **NDJSON** — the streaming-friendly form for very large uploads. Blank lines are skipped, and a single malformed *line* is reported and skipped (the newline reframes the next record). @@ -367,7 +367,7 @@ A `200` is returned whenever the body was read and the records were processed | 503 | `{"error":"service unavailable"}` | NATS JetStream full (backpressure) mid-batch; includes `Retry-After: 30` | :::caution[At-least-once on retry] -A batch aborted partway (a `503`/`500`, or a JSON-array syntax error, after some leading records were already published) re-publishes those leading records when the whole batch is retried. Enable deduplication if duplicate suppression matters — this is the same at-least-once property the single-object path already has (the SDK retries both on `503`). +A batch aborted partway (a `503`/`500`, or a JSON-array syntax error, after some leading records were already published) re-publishes those leading records when the whole batch is retried. Enable deduplication if duplicate suppression matters — this is the same at-least-once property the single-object path already has (the SDKs retry both on `503`). ::: **curl example (JSON array):** @@ -467,7 +467,7 @@ curl -X POST http://localhost:8080/v1/ops/query \ Executes a type-safe structured query against a table. The query AST is validated against the schema and converted to parameterized SQL. Permissions from the access control policy are enforced (column filtering, row-level security, aggregation restrictions). :::note[The column allowlist is a hard cap on every clause] -Every column the query references — in `columns`, an aggregation argument, `filters`, `group_by`, `order_by`, or `time_range` — must be permitted by the role's `allow_columns`/`deny_columns`, or the request is rejected with `403 column "x" not allowed`. A full-row read is requested with `"select_all": true` (expanded to the columns the role may read — never a raw `SELECT *`); **omitting `columns` returns nothing**, so a hidden column never leaks by being left out, grouped on, or filtered on. See [Access control → Column permissions](/access-control#column-permissions). +Every column the query references — in `columns`, an aggregation argument, `filters`, `group_by`, `order_by`, or `time_range` — must be permitted by the role's `allow_columns`/`deny_columns`, or the request is rejected with `403 column "x" not allowed`. A full-row read is requested with `"select_all": true` (for a column-restricted role, expanded to exactly the columns the role may read rather than a bare `SELECT *`; unrestricted/admin roles do get `SELECT *`); **omitting `columns` returns nothing**, so a hidden column never leaks by being left out, grouped on, or filtered on. See [Access control → Column permissions](/access-control#column-permissions). ::: **Request:** @@ -495,7 +495,7 @@ Every column the query references — in `columns`, an aggregation argument, `fi | Field | Type | Required | Description | | ----- | ---- | -------- | ----------- | | `columns` | string \| string[] | No | Columns to SELECT — an array, or a single string for one column. A literal `"*"` is the column *named* `*`, **not** a wildcard. Omit (or send `[]` / `""`) to select nothing; use `select_all` for a full-row read. Mutually exclusive with `select_all`. | -| `select_all` | bool | No | Select every column the role may read (the all-columns wildcard, expanded server-side to the allow/deny set). Mutually exclusive with a non-empty `columns`, and with `aggregations`. | +| `select_all` | bool | No | Select every column the role may read (the all-columns wildcard; a column-restricted role's projection is expanded server-side to its allow/deny set, an unrestricted/admin role gets `SELECT *`). Mutually exclusive with a non-empty `columns`, and with `aggregations`. | | `aggregations` | object[] | No | Aggregation functions (`fn`, `column`, `alias`). | | `filters` | object[] | No | WHERE conditions (`column`, `op`, `value`). Ops: eq, neq, gt, gte, lt, lte, in, like. | | `group_by` | string[] | No | GROUP BY columns. | diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index b40663ac..3b1d3ac5 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -81,7 +81,7 @@ The API layer uses [Chi](https://github.com/go-chi/chi) for routing with Request - **stream.go** — Real-time streaming via SSE. Callers select a table with the `?table=` query parameter. Each connection registers one `Subscriber` (the `stream/` package) with both the event `Hub` (under its `(topic, role)`) and the shared keepalive wheel, then drains both from a single byte-pump — so idle streams keep emitting `:` keepalive comments (surviving reverse-proxy idle timeouts) while live events arrive already projected and serialized. Per-event projection/serialization happens **once per role** in the `Hub`, not once per subscriber ([#294](https://github.com/Wave-RF/WaveHouse/issues/294)); the handler also snapshots the connection's JWT claims onto the `Subscriber`, which the `Hub` evaluates per subscriber when the role carries a row-level `filter` ([#319](https://github.com/Wave-RF/WaveHouse/issues/319)). Gap-fill replay from NATS JetStream (`DeliverByStartTime`) stays per-connection (low-volume, one-time on connect). - **schema.go** — Schema discovery API: list all schemas, get one table, trigger refresh. - **dlq.go** — DLQ stats endpoint and `EnsureDLQStream` helper for creating the `WAVEHOUSE_DLQ` NATS stream. -- **health.go** — Liveness (`/livez`), readiness (`/readyz`), and a content-free `Online` ping (`/v1/health`, the SDK's public liveness check); `/healthz` is a permanent alias of `/livez`, and `/health`/`/ready` are deprecated aliases. All three consult an optional `BootState` so they can return 503 while boot-time schema discovery is still failing in the retry loop (see `cmd/wavehouse/main.go`); once `BootState.Set(nil)` fires, `/livez` returns 200 and stays there. `/readyz` additionally pings ClickHouse each call; `/v1/health` deliberately does not. +- **health.go** — Liveness (`/livez`), readiness (`/readyz`), and a content-free `Online` ping (`/v1/health`, the SDKs' public liveness check); `/healthz` is a permanent alias of `/livez`, and `/health`/`/ready` are deprecated aliases. All three consult an optional `BootState` so they can return 503 while boot-time schema discovery is still failing in the retry loop (see `cmd/wavehouse/main.go`); once `BootState.Set(nil)` fires, `/livez` returns 200 and stays there. `/readyz` additionally pings ClickHouse each call; `/v1/health` deliberately does not. ### `stream/` — SSE keepalive & fan-out @@ -155,7 +155,7 @@ The package's design invariants — stdout always 100%, WARN+ERROR always export ### `query/` — Structured Query Engine - **ast.go** — `StructuredQuery` AST types: columns, aggregations, filters, group by, order by, limit, time range. -- **builder.go** — `Build()` converts AST to parameterized SQL. It is the single chokepoint that validates every referenced identifier against the schema **and** authorizes every column reference — projection, aggregation args, filters, group_by, order_by, time_range — against the role's column allowlist (the [#223](https://github.com/Wave-RF/WaveHouse/issues/223) hard cap). A full-row read is requested with `select_all`, which expands to the role's allowed columns rather than emitting a raw `SELECT *`; an omitted projection selects nothing, and `*` in `columns` is a literal column name. Every identifier is backtick-quoted via `internal/chsql` (`QuoteIdent`) so any ClickHouse-legal name is accepted — a name containing `?` is refused fail-closed ([#279](https://github.com/Wave-RF/WaveHouse/issues/279)). The role's row-level-security predicate and `max_rows` cap are emitted by `Build()` itself, as part of the WHERE and LIMIT assembly — policy SQL is never spliced into rendered text ([#322](https://github.com/Wave-RF/WaveHouse/issues/322)). Timestamp bucketing for cache optimization. +- **builder.go** — `Build()` converts AST to parameterized SQL. It is the single chokepoint that validates every referenced identifier against the schema **and** authorizes every column reference — projection, aggregation args, filters, group_by, order_by, time_range — against the role's column allowlist (the [#223](https://github.com/Wave-RF/WaveHouse/issues/223) hard cap). A full-row read is requested with `select_all`, which for a column-restricted role expands to the role's allowed columns rather than emitting a raw `SELECT *` (unrestricted/admin roles do get `SELECT *`); an omitted projection selects nothing, and `*` in `columns` is a literal column name. Every identifier is backtick-quoted via `internal/chsql` (`QuoteIdent`) so any ClickHouse-legal name is accepted — a name containing `?` is refused fail-closed ([#279](https://github.com/Wave-RF/WaveHouse/issues/279)). The role's row-level-security predicate and `max_rows` cap are emitted by `Build()` itself, as part of the WHERE and LIMIT assembly — policy SQL is never spliced into rendered text ([#322](https://github.com/Wave-RF/WaveHouse/issues/322)). Timestamp bucketing for cache optimization. ### `chsql/` — ClickHouse SQL Helpers @@ -283,4 +283,5 @@ Client GET /v1/stream | Embedded KV | Pebble | Optional deduplication | | Config | cleanenv | YAML + env var config loading | | Release | GoReleaser | Cross-platform binary builds | +| Client SDKs | TypeScript, Go | Typed clients with the same feature set (ingest, query, pipes, streaming, admin) | | Containers | Docker (distroless) | Minimal production images | diff --git a/docs/src/content/docs/claude-code.md b/docs/src/content/docs/claude-code.md index 59abc2d0..7e233e2b 100644 --- a/docs/src/content/docs/claude-code.md +++ b/docs/src/content/docs/claude-code.md @@ -81,7 +81,7 @@ To add a command: drop a `.md` file in `.claude/commands/`. Filename becomes the | Subagent | When to use | | -------- | ----------- | | `pre-push-reviewer` | **Mandatory before pushing to a PR branch** (enforced by `.claude/hooks/agent-bash-gate.sh`), run in parallel with the other reviewers in `scripts/pre-push-reviewers.sh` — all must reach `ship_it`. Also used for auditing someone else's PR after `wt switch pr:`. Runs the canonical `.github/prompts/pr-review.md` workflow against the local branch in fresh context. Fetches PR comments + CI status + linked-issue acceptance criteria when on a PR branch. Returns `[MUST]`/`[SHOULD]`/`[MAY]` findings + a parseable `VERDICT: ship_it\|iterate\|block` line that drives the `tmp/pre-push-reviewer-passed-` marker. | -| `docs-reviewer` | **Mandatory before pushing to a PR branch** — runs in parallel with the other pre-push reviewers (all enforced by `.claude/hooks/agent-bash-gate.sh`). Reviews docs **prose** (accuracy-vs-code, runnable examples, clarity, completeness) **and code↔docs sync** (code that changed but whose docs didn't), using `.github/prompts/docs-review.md` over the `scripts/docs-prose.sh` denylist set (Starlight site + governance docs incl. the SDK readme). Default (branch) scope emits `VERDICT: ship_it\|iterate\|block` → writes `tmp/docs-reviewer-passed-`; a path/`all` is advisory (no marker). Posts no PR comments, never edits docs. Complements misspell / markdownlint / starlight-links-validator, never duplicates them. | +| `docs-reviewer` | **Mandatory before pushing to a PR branch** — runs in parallel with the other pre-push reviewers (all enforced by `.claude/hooks/agent-bash-gate.sh`). Reviews docs **prose** (accuracy-vs-code, runnable examples, clarity, completeness) **and code↔docs sync** (code that changed but whose docs didn't), using `.github/prompts/docs-review.md` over the `scripts/docs-prose.sh` denylist set (Starlight site + governance docs incl. the SDK readmes). Default (branch) scope emits `VERDICT: ship_it\|iterate\|block` → writes `tmp/docs-reviewer-passed-`; a path/`all` is advisory (no marker). Posts no PR comments, never edits docs. Complements misspell / markdownlint / starlight-links-validator, never duplicates them. | Invoke via the `Agent` tool with `subagent_type: pre-push-reviewer`, or via `/agents`. diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index fee146d3..76c7272c 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -278,7 +278,7 @@ API servers in standalone mode expose liveness and readiness endpoints under the Configure your load balancer or orchestrator to use these endpoints. -**Exposure.** Probes share the API server's port (`:8080`) — kubelet probes the container internally, so there's no separate-port convention for them (metrics are the signal that optionally gets its own `prometheus.port`). If you forward `:8080` to the public internet the probe paths become reachable. The **recommended** posture is to keep `/livez`/`/readyz`/`/healthz` to internal callers and expose only **`/v1/health`** publicly (the SDK's content-free liveness ping, which never touches ClickHouse). `/readyz` issues a ClickHouse `Ping` on every call, so a public `/readyz` lets an unauthenticated flood become per-request backend pings, and the bare probes leak boot/readiness state — keeping them internal is a [reverse-proxy/ingress concern](/reverse-proxy#health-probes), and your orchestrator reaches them the internal way (kubelet on the container, LB on the backend) regardless. +**Exposure.** Probes share the API server's port (`:8080`) — kubelet probes the container internally, so there's no separate-port convention for them (metrics are the signal that optionally gets its own `prometheus.port`). If you forward `:8080` to the public internet the probe paths become reachable. The **recommended** posture is to keep `/livez`/`/readyz`/`/healthz` to internal callers and expose only **`/v1/health`** publicly (the SDKs' content-free liveness ping — `wh.sys.health()` / `wh.Sys.Health(ctx)` — which never touches ClickHouse). `/readyz` issues a ClickHouse `Ping` on every call, so a public `/readyz` lets an unauthenticated flood become per-request backend pings, and the bare probes leak boot/readiness state — keeping them internal is a [reverse-proxy/ingress concern](/reverse-proxy#health-probes), and your orchestrator reaches them the internal way (kubelet on the container, LB on the backend) regardless. ### Boot-time degraded mode diff --git a/docs/src/content/docs/development.md b/docs/src/content/docs/development.md index e3f765db..cc07e18f 100644 --- a/docs/src/content/docs/development.md +++ b/docs/src/content/docs/development.md @@ -186,7 +186,7 @@ They block the terminal and stream logs; simply press `Ctrl+C` to instantly tear ### Using the SDK against `make dev` -There's no bundled playground — point the published `@wavehouse/sdk` client at your local server (`baseURL: "http://localhost:8080"`), with the dev policy seeded so requests are authorized: +There's no bundled playground — point the published `@wavehouse/sdk` client (or the Go SDK, `github.com/Wave-RF/WaveHouse/clients/go`) at your local server (`baseURL: "http://localhost:8080"`), with the dev policy seeded so requests are authorized: ```bash WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml make dev @@ -194,7 +194,7 @@ WH_POLICY_FILE_PATH=deployments/compose/dev-policy.yaml make dev See the [SDK guide](/sdk) for the client API and examples. -Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. +Frontend devs running their own dev server (Vite, Next.js, etc.) can `import { createClient } from '@wavehouse/sdk'` and point `baseURL: 'http://localhost:8080'`; CORS is permissive so cross-origin browser requests just work. Go services do the same with `wavehouse.NewClient(wavehouse.Config{BaseURL: "http://localhost:8080"})` — see the [Go SDK docs](/sdk/go). ### Validating tokens @@ -286,19 +286,21 @@ go build -o bin/wavehouse ./cmd/wavehouse ### How It Works -All **Go** test commands use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile invokes them as `go tool `, so no global installation is needed. +The Go suite targets (`test-unit`, `test-integration`) use [gotestsum](https://github.com/gotestyourself/gotestsum) for pytest-style colored output with pass/fail icons, durations, and a summary. Tool versions are pinned in `go.mod` via `tool` directives — the Makefile invokes them as `go tool `, so no global installation is needed. `test-e2e` runs the orchestrator + vitest and `test-ts` runs vitest directly, so neither uses gotestsum. The Go SDK and conformance targets (`test-go-sdk`, `test-go-sdk-e2e`, `test-conformance-ts`) run plain `go test` / `node` and ignore `ARGS` and `V=1`. -All tests run with Go's **race detector** (`-race`) enabled by default. WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. +Go tests run with the **race detector** (`-race`) enabled by default (including `test-go-sdk` — the SDK's streaming subsystem is highly concurrent; `test-go-sdk-e2e` skips it since it drives a live server). WaveHouse is highly concurrent (NATS consumers, singleflight caching, SSE hubs) — the race detector catches data races that would panic in production. ### Quick Reference ```bash -# Prefix any test target with V=1 for verbose output, e.g. `V=1 make test` +# V=1 gives verbose output on test-unit / test-integration / test-e2e, +# e.g. `V=1 make test-unit` -# Unit tests (compact output) — alias for `test-unit` +# Unit tests + Go SDK tests (compact output) — alias for `test-unit` + `test-go-sdk` make test -# Run specific test(s) +# Run specific root-module test(s) — ARGS reaches test-unit only; the +# test-go-sdk half of `make test` runs its full suite regardless make test ARGS="-run TestValidate" # Go integration tests (requires Docker) @@ -311,22 +313,22 @@ make test-ts # E2E SDK suite against bin/wavehouse-cov make test-e2e -# All four suites sequentially + merged coverage +# All suites sequentially + merged coverage make test-all -# Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts, -# then test-integration + test-e2e + cov +# Full CI: parallel verify + builds (Go + SDK + docs) + test + test-ts + +# test-conformance-ts, then test-integration + test-e2e + cov make ci # Merge available covdata + gate against total threshold make cov ``` -Each test target writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. +Each coverage-instrumented suite target (`test-unit`, `test-integration`, `test-e2e`, `test-go-sdk`, `test-ts`) writes `covdata` to `tmp/coverage//data/`, renders a textfmt + HTML report, and gates against the per-suite threshold in `.testcoverage.yml`. `make cov` merges whichever suites have run and gates against the total. `test-go-sdk` is gated but **not** merged: `clients/go` is a nested Go module, invisible to the root module's `-coverpkg=./...`, so its statements can never reach `tmp/coverage/total` — it carries its own `suites.go-sdk` floor instead, the same way the TS SDK carries `ts-*`. The remaining SDK/conformance targets (`test-go-sdk-e2e`, `test-conformance-ts`) run without coverage instrumentation or a per-suite gate. -**Verbose output**: Use `V=1` to switch from compact `testdox` format to full verbose output. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). +**Verbose output**: Use `V=1` to switch from the compact `pkgname-and-test-fails` format to `standard-verbose` on `test-unit` / `test-integration`, and to stream live output on `test-e2e`. This is a standard Makefile convention (`make test -v` can't work because `-v` is a `make` flag). `test-ts` and the Go SDK / conformance targets ignore it. -**Extra flags**: All test targets accept `ARGS="..."` for additional `go test` flags (e.g., `-run`, `-count`, `-timeout`). +**Extra flags**: `test-unit`, `test-integration`, and `test-ts` accept `ARGS="..."` for pass-through flags (e.g., `-run`, `-count`, `-timeout` for the Go targets; vitest flags for `test-ts`). `test-e2e` and the Go SDK / conformance targets ignore it. **Note on timing**: gotestsum's `DONE ... in X.XXXs` reports pure test execution time. The total wall time includes Go compiling all packages — the first run compiles everything (~15s), subsequent runs use the build cache (~1s). @@ -335,7 +337,10 @@ Each test target writes `covdata` to `tmp/coverage//data/`, renders a tex | Category | Location | Docker? | Command | | -------- | -------- | ------- | ------- | | Unit tests | `internal/*/_test.go` | No | `make test` | -| SDK unit tests | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | +| SDK unit tests (TS) | `clients/ts/src/**/*.test.ts` | No | `make test-ts` (always includes coverage + gate) | +| SDK unit tests (Go) | `clients/go/*_test.go` (nested Go module) | No | `make test-go-sdk` (runs with `-race`, always includes coverage + gate) | +| Wire-format conformance | `clients/go/conformance_test.go` + `tests/conformance/conformance_ts.mjs`, both replaying `clients/go/testdata/wire_cases.json` | No | Go half via `make test-go-sdk`; TS half via `make test-conformance-ts` | +| SDK E2E (Go, live server) | `clients/go/e2e_test.go` (`//go:build e2e`) | No | `make test-go-sdk-e2e` (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | | Integration tests (Go) | `tests/integration/*_test.go` | Yes | `make test-integration` | | E2E tests (SDK) | `tests/e2e/sdk/*.test.ts` | Yes | `make test-e2e` | @@ -349,6 +354,8 @@ Shared test utilities live in `internal/testutil/` (e.g., `testutil.NopLogger()` - **Unit test for `internal/foo/`** → create `internal/foo/foo_test.go` (same package). - **Integration test needing Docker** → add a subtest under `tests/integration/` (e.g. a new file with `//go:build integration`). - **E2E test via SDK** → add a `tests/e2e/sdk/*.test.ts` file. These tests exercise the full pipeline (ingest → ClickHouse → query) through the TypeScript SDK. Run with `make test-e2e`. +- **Go SDK unit test** → add to `clients/go/*_test.go` (nested module — outside `test-unit`'s scope). Run with `make test-go-sdk`. +- **Wire-format parity case** → when you add or change an endpoint, add an entry to `clients/go/testdata/wire_cases.json` plus its dispatch in both runners (`clients/go/conformance_test.go` and `tests/conformance/conformance_ts.mjs`). Required by the SDK sync rule in `AGENTS.md` / `CONTRIBUTING.md`. - **Test helpers** → add to `internal/testutil/` (Go) or `tests/e2e/sdk/helpers.ts` (E2E). ### E2E Tests via SDK @@ -460,8 +467,13 @@ WaveHouse/ │ ├── query/ # Structured query AST + SQL builder │ ├── stream/ # SSE fan-out: Hub, Subscriber queue, Bucket, keepalive wheel │ └── testutil/ # Shared test helpers and mocks +├── clients/ # Official SDKs +│ ├── ts/ # TypeScript SDK (@wavehouse/sdk) +│ └── go/ # Go SDK — a NESTED Go module (own go.mod, invisible +│ # to root `go list`; hence the *-go-sdk make targets) ├── tests/ # Integration & E2E tests │ ├── integration/ # Go integration tests (//go:build integration) +│ ├── conformance/ # TS half of the cross-SDK wire-format conformance suite │ └── e2e/ # E2E suite (orchestrator + ClickHouse testcontainer) │ ├── fixtures/ # ClickHouse DDL + config/policy fixtures │ └── sdk/ # E2E specs driven through the TypeScript SDK (Vitest) @@ -508,11 +520,11 @@ Run `make help` to see all targets. Key ones: | `make obs-grafana` | Grafana alternative to aspire, more advanced and complicated | | `make obs-front` | Custom graphs like grafana, but is simpler and easier to configure like aspire | | **Static checks** | | -| `make fmt` | Check formatting across Go (`gofumpt`) + TS (Biome). Run `make fix` to apply. | +| `make fmt` | Check formatting across root-module Go (`gofumpt`) + TS (Biome); the nested `clients/go` module's gofumpt check runs under `make verify` (`verify-go-sdk`). Run `make fix` to apply everywhere. | | `make tidy` | Verify `go.mod`/`go.sum` are tidy (run `make fix` to apply) | -| `make lint` | Run linters across Go (`golangci-lint`) + TS (Biome) + Markdown/MDX (markdownlint) + prose (misspell) | +| `make lint` | Run linters across Go (`golangci-lint`, root + `clients/go`) + TS (Biome) + Markdown/MDX (markdownlint) + prose (misspell) | | `make vulncheck` | Run `govulncheck` (V=1 for full call stacks) | -| `make verify` | Repo-wide static checks: Go (tidy + fmt + vulncheck + lint) + TS (Biome + `tsc` typecheck) + Markdown/MDX (markdownlint + rule fixtures) + prose (misspell) + shell (shellcheck) + workflows (actionlint) + path-classifier fixtures + release-channel fixtures + docs type-check (`astro check` — not a full build, so link validation stays CI's job) (parallel-safe: `make -j verify`) | +| `make verify` | Repo-wide static checks: root Go (tidy + fmt + vulncheck + lint), `clients/go` (fmt + vet + lint — no tidy/vulncheck: it's a nested module, invisible to the root-scoped `tidy`/`vulncheck` targets) + TS (Biome + `tsc` typecheck) + Markdown/MDX (markdownlint + rule fixtures) + prose (misspell) + shell (shellcheck) + workflows (actionlint) + path-classifier fixtures + release-channel fixtures + docs type-check (`astro check` — not a full build, so link validation stays CI's job) (parallel-safe: `make -j verify`) | | `make fix` | Auto-fixes across Go (`tidy` + `gofumpt` + `goimports` + `lint --fix`), TS (Biome `--write`), Markdown (markdownlint `--fix`), MDX (`fix-mdx-fences` only — the generic fixers never run over `.mdx`), and docs-prose spelling (misspell, both) | | **Build** | | | `make build` | Compile `wavehouse` → `bin/wavehouse` (debug symbols kept) | @@ -520,18 +532,21 @@ Run `make help` to see all targets. Key ones: | `make build-cover` | Coverage-instrumented build → `bin/wavehouse-cov` (used by E2E) | | `make build-ts` | Build TypeScript SDK → `clients/ts/dist/` | | **Test** | | -| `make test` | Alias for `test-unit` | +| `make test` | Alias for `test-unit` + `test-go-sdk` | | `make test-unit` | Go unit tests + render coverage + gate suite threshold | +| `make test-go-sdk` | Go SDK (`clients/go`, nested module) unit tests with `-race` + render coverage + gate `suites.go-sdk` (own gate; never merged into the Go total) | +| `make test-go-sdk-e2e` | Go SDK E2E against a live server (`WAVEHOUSE_URL`, `WAVEHOUSE_AUTH`) | +| `make test-conformance-ts` | TS SDK wire-format conformance against the shared `wire_cases.json` fixture (builds the TS SDK first) | | `make test-integration` | Go integration tests (requires Docker) + coverage gate | | `make test-ts` | SDK vitest unit tests + v8 coverage + gate against `suites.ts-unit` (matches Go's "always coverage" pattern) | | `make cov` | Merge Go + TS coverage and gate against thresholds. Auto-runs after `make test-all` and `make ci`; standalone `make cov` is "show me the merged numbers without re-running." Each side skips silently if its data is missing, but `make cov` fails if *both* are empty (you ran it before any test target). | | `make test-e2e` | E2E SDK suite against `bin/wavehouse-cov` + coverage gate | -| `make test-all` | All four suites sequentially + merged coverage gate | +| `make test-all` | All suites sequentially + merged coverage gate | | `make ci` | Full pipeline: parallel `verify` + builds + unit/SDK tests, then integration + E2E + cov | | **Release** (see [Cutting a release](#cutting-a-release)) | | | `make release-server VERSION=X.Y.Z` | Tag a server release — binaries + container image | | `make release-sdk-ts VERSION=X.Y.Z` | Tag a TypeScript SDK release — npm | -| `make release-sdk-go VERSION=X.Y.Z` | Tag a Go SDK release — `go get` (pending [#434](https://github.com/Wave-RF/WaveHouse/pull/434)) | +| `make release-sdk-go VERSION=X.Y.Z` | Tag a Go SDK release — `go get` | | **Analysis** (informational, not in CI) | | | `make size` | Binary size analysis → `tmp/analysis/` (text + SVG + interactive HTML) | | `make audit-cgo` | Audit dependency tree for C files (builds use `CGO_ENABLED=0`) | @@ -544,7 +559,7 @@ Run `make help` to see all targets. Key ones: | `make clean-tools` | Installed tools and pnpm deps (`.bin/`, `node_modules/`) | | `make clean-all` | Full reset: above + `data/` + Docker volumes | -All test targets accept `ARGS="..."` for pass-through `go test` flags. Build targets accept `TAGS="..."` for Go build tags. `V=1` switches to verbose `gotestsum` output. +`test-unit`, `test-integration`, and `test-ts` accept `ARGS="..."` for pass-through flags; `test-unit`, `test-integration`, and `test-e2e` accept `V=1` for verbose output. `test-go-sdk`, `test-go-sdk-e2e`, and `test-conformance-ts` ignore both. Build targets accept `TAGS="..."` for Go build tags. ## Dependency Management @@ -588,10 +603,9 @@ Cutting a release is **one tag** — no version bump in code, no release branch: ```bash make release-server VERSION=0.1.0 # tag v0.1.0 → binaries + container image make release-sdk-ts VERSION=0.1.0 # tag clients/ts/v0.1.0 → @wavehouse/sdk on npm +make release-sdk-go VERSION=0.1.0 # tag clients/go/v0.1.0 → the Go module proxy ``` -`make release-sdk-go` exists too, wired ahead of the Go SDK landing ([#434](https://github.com/Wave-RF/WaveHouse/pull/434)); it refuses to run until `clients/go/` is in the repo. - The one thing to do *before* tagging is promote the changelog: `AGENTS.md` requires every PR to add its entry under `## Unreleased`, so open a PR renaming that heading to `## [X.Y.Z] - YYYY-MM-DD` and adding the matching link reference at the foot of the file. Nothing in the release pipeline reads `CHANGELOG.md` — this is for the file's own readers. Each runs [`scripts/release.sh`](https://github.com/Wave-RF/WaveHouse/blob/main/scripts/release.sh), which preflights (on `main`, clean tree, in sync with `origin/main`, the tag free both locally and on the remote, the required `CI` check green on *this exact commit*), prints exactly what will be published, and asks before pushing. `DRY_RUN=1 make release-…` stops after the plan. Tag creation is admin-only via the `release tag protection` ruleset. @@ -625,9 +639,10 @@ Tag globs are anchored at the start of the ref name, so `v*` never matches a `cl ### What a release publishes - **Server —** a **GitHub Release** with the cross-compiled archives (linux/darwin/windows/freebsd × amd64/arm64; `.zip` on Windows, `.tar.gz` elsewhere) and `checksums.txt`. A tag carrying a prerelease suffix (`v0.1.0-alpha.1`) is marked as a GitHub pre-release, so it never takes the "Latest release" badge from a shipped stable version. -- **Both —** **release notes generated by GitHub** from the PRs merged since the previous tag *in the same family* — one line per PR, since `main` is squash-merged, grouped into the categories defined in [`.github/release.yml`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/release.yml). Grouping is by **PR label**: `github_actions` / `documentation` are applied automatically by `actions/labeler`, but `breaking-change`, `security`, `bug`, and `enhancement` are applied by hand — an unlabelled PR lands in "Other changes". Dependabot is split out by **author** rather than by label, because the labels `actions/labeler` applies by path — `github_actions`, `documentation` — mark our own PRs too; our CI work gets its own "CI & build" section — ordered above Documentation, since a CI PR here nearly always updates docs too — and Dependencies is pure Dependabot residue. **Any category keyed on a label a Dependabot PR can carry needs that author exclude** — labeler's path labels *and* the ecosystem labels Dependabot applies itself (`dependencies`, `javascript`, `go`, `github_actions`; `javascript` is in neither `labeler.yml` nor our categories) — or that category intercepts bumps before they reach the `📦 Dependencies` catch-all. `CHANGELOG.md` is *not* the source of the release body; it is the longer-form record of why each change was made. +- **Server + TypeScript SDK —** **release notes generated by GitHub** from the PRs merged since the previous tag *in the same family* — one line per PR, since `main` is squash-merged, grouped into the categories defined in [`.github/release.yml`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/release.yml). Grouping is by **PR label**: `github_actions` / `documentation` are applied automatically by `actions/labeler`, but `breaking-change`, `security`, `bug`, and `enhancement` are applied by hand — an unlabelled PR lands in "Other changes". Dependabot is split out by **author** rather than by label, because the labels `actions/labeler` applies by path — `github_actions`, `documentation` — mark our own PRs too; our CI work gets its own "CI & build" section — ordered above Documentation, since a CI PR here nearly always updates docs too — and Dependencies is pure Dependabot residue. **Any category keyed on a label a Dependabot PR can carry needs that author exclude** — labeler's path labels *and* the ecosystem labels Dependabot applies itself (`dependencies`, `javascript`, `go`, `github_actions`; `javascript` is in neither `labeler.yml` nor our categories) — or that category intercepts bumps before they reach the `📦 Dependencies` catch-all. `CHANGELOG.md` is *not* the source of the release body; it is the longer-form record of why each change was made. - **Server —** a **GHCR image** at `ghcr.io/wave-rf/wavehouse`, with two tags: the immutable `:vX.Y.Z`, and one moving *channel* pointer. A stable release moves `:latest`; a prerelease moves `:alpha` / `:beta` / `:rc` / `:next` instead, matching the npm dist-tag it would get. The channel comes from the **first** prerelease identifier, matched **exactly**: `v0.2.0-rc.1` → `:rc`, while `-alpha1`, `-preview.1`, or any other form → `:next`. `scripts/ci/release-channel.sh` is the single rule every publisher uses, so `ghcr.io/wave-rf/wavehouse:rc` and `@wavehouse/sdk@rc` can't drift apart. **A prerelease-only project therefore has no `:latest` tag** — that is deliberate; `:latest` starts existing when the first stable release ships. - **TypeScript SDK —** an **npm publish** of `@wavehouse/sdk` under `latest` (stable) or `alpha`/`beta`/`rc`/`next` (prerelease), plus its own GitHub Release. +- **Go SDK —** nothing to upload. A `clients/go/vX.Y.Z` tag *is* the release: the [Go module proxy](https://proxy.golang.org) serves it on the first `go get github.com/Wave-RF/WaveHouse/clients/go@vX.Y.Z`, and `sum.golang.org` records the module hash. No workflow fires — the release tag globs (`v*`, `clients/ts/v*`) are anchored and never match it — so there is no GitHub Release or provenance attestation for the Go SDK today. - **Server —** **build-provenance attestations** (Sigstore, free for public repos) over every archive and over the image's multi-arch manifest digest; the image attestation is stored alongside the image in GHCR. The release job verifies its own attestations before finishing, so a release that publishes unverifiable provenance goes red. - **TypeScript SDK —** an **npm provenance attestation** via `npm publish --provenance`, surfaced as the provenance badge on the package page and checkable with `npm audit signatures`. The npm job does not re-verify it the way the server job does. @@ -688,7 +703,7 @@ If the title doesn't match, a sticky comment posts on the PR explaining the form The `main branch protection` ruleset requires one status check to pass before any PR can merge: -- `CI` — the aggregator job of `.github/workflows/ci.yml`. The workflow is a job DAG over the same Makefile targets local `make ci` runs: `lint` (`make verify`), `unit` (`make test-unit test-ts`), `integration` (`make test-integration`), `e2e` (`make -j test-e2e` — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), `coverage` (`make cov` over every suite's uploaded coverage fragment + threshold gates, like local `make ci`'s final step), `docs-build` (`make build-docs` when docs-affecting files changed, uploading the docs dist artifact), `PR title` (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run's Summary page gets a per-job wall-clock table from the non-gating `Timing summary` job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md). +- `CI` — the aggregator job of `.github/workflows/ci.yml`. The workflow is a job DAG over the same Makefile targets local `make ci` runs: `lint` (`make verify`), `unit` (`make test-unit test-ts test-go-sdk test-conformance-ts`), `integration` (`make test-integration`), `e2e` (`make -j test-e2e` — builds its own SDK dist + cover binary on a warm cache, runs the suite exactly like a local run), `coverage` (`make cov` over every suite's uploaded coverage fragment + threshold gates, like local `make ci`'s final step), `docs-build` (`make build-docs` when docs-affecting files changed, uploading the docs dist artifact), `PR title` (Conventional Commits), and the docs preview/deploy jobs. The aggregator fails if any job failed or was canceled and treats skipped jobs as passing — docs-only PRs skip the Go test suites by design, and fork PRs run everything except the (secret-bearing) docs deploys. Every run's Summary page gets a per-job wall-clock table from the non-gating `Timing summary` job. The full architecture — DAG diagram, design invariants, cache policy, how to add a job — lives in [`.github/workflows/README.md`](https://github.com/Wave-RF/WaveHouse/blob/main/.github/workflows/README.md). The `PR housekeeping` workflow still runs on every PR (labels + the title explainer comment) but is no longer a required check. diff --git a/docs/src/content/docs/getting-started.md b/docs/src/content/docs/getting-started.md index 8b368d8f..69e0a097 100644 --- a/docs/src/content/docs/getting-started.md +++ b/docs/src/content/docs/getting-started.md @@ -76,7 +76,7 @@ curl -s -X POST "http://localhost:8080/v1/query?table=clicks" \ `POST /v1/query?table={table}` and `GET/POST /v1/pipes/{name}` are cached in-process (L1 Ristretto) with singleflight coalescing — duplicate concurrent queries hit ClickHouse once. For raw SQL there's `POST /v1/ops/query` (an admin escape hatch that never caches, emitting `Cache-Control: no-store`), but it's **admin-only** — the trial `public` role can't reach it. To use it, swap the public default for real auth: configure a JWT secret and present a token whose role is the policy [`admin_role`](/access-control#admin_role--the-privileged-role). :::tip[Prefer a type-safe client?] -The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). +The [TypeScript SDK](/sdk) wraps this endpoint in a chainable query builder with autocomplete on your table names and row types — plus live queries and streaming; the [Go SDK](/sdk/go) offers the same builder with generics for typed rows. The raw shapes are in the [structured query reference](/api#post-v1querytabletable--structured-query). ::: ## 5. Subscribe to real-time updates @@ -106,6 +106,7 @@ The handful of things that most often trip up a first session — each is expect - **[Architecture](/architecture)** — how ingest, query, cache, and streaming fit together. - **[API Reference](/api)** — every endpoint, request/response shape, and error code. - **[TypeScript SDK](/sdk)** — client with query builder, live queries, and codegen; one runtime dependency. +- **[Go SDK](/sdk/go)** — the same surface for Go: context-first, generics for typed rows, codegen CLI. - **[Configuration](/configuration)** — full YAML + environment variable reference. - **[Deployment](/deployment)** — Docker images, releases, health checks. - **[Development](/development)** — building from source, running tests, hot-reload workflow. diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index 0fe6c504..da92388a 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -102,8 +102,8 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click Per-table, per-role column and row-level policies with JWT claim templating. Stored in NATS KV with file-based bootstrap and cluster sync. - - `@wavehouse/sdk` — a type-safe query builder, live queries, real-time streaming, and codegen from your schemas, with one runtime dependency of ~1.4 KB gzipped. + + `@wavehouse/sdk` and `github.com/Wave-RF/WaveHouse/clients/go` — type-safe query builders, live queries, real-time streaming, and codegen from your schemas, speaking one shared wire format. One runtime dependency of ~1.4 KB gzipped in TypeScript; none in Go. @@ -111,7 +111,7 @@ If you're building user-facing analytics, **WaveHouse is like Supabase for Click ## Query it like a database. Subscribe to it like a socket -The [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming: +The [TypeScript SDK](/sdk) wraps the whole surface — typed inserts, a chainable query builder, and live queries that backfill history before streaming (examples below are TypeScript). Writing Go? The official [Go SDK](/sdk/go) mirrors the same feature set — see the [Go quick start](/sdk/go#quick-start). @@ -216,6 +216,11 @@ Self-hosting WaveHouse is deliberately boring — one binary, one dependency. Bu description="Query builder, live queries, streaming, and schema codegen." href="/sdk" /> +
diff --git a/docs/src/content/docs/pipes.mdx b/docs/src/content/docs/pipes.mdx index 78c753cf..dc3ac13d 100644 --- a/docs/src/content/docs/pipes.mdx +++ b/docs/src/content/docs/pipes.mdx @@ -148,7 +148,7 @@ curl -X PUT http://localhost:8080/v1/ops/pipes/top_pages \ }' ``` -A `PUT` is a full replace of that named pipe; `name` is taken from the URL. Definitions are stored in NATS KV and synced across nodes, so a create/update/delete applies cluster-wide without a restart. The [TypeScript SDK](/sdk) exposes the same operations as `client.pipes.list()`, `client.pipes.get(name)`, `client.pipes.set(name, def)`, and `client.pipes.delete(name)`. +A `PUT` is a full replace of that named pipe; `name` is taken from the URL. Definitions are stored in NATS KV and synced across nodes, so a create/update/delete applies cluster-wide without a restart. The [TypeScript SDK](/sdk) exposes the same operations as `client.pipes.list()`, `client.pipes.get(name)`, `client.pipes.set(name, def)`, and `client.pipes.delete(name)`; the [Go SDK](/sdk/go/pipes) as `wh.Pipes.List(ctx)`, `wh.Pipes.Get(ctx, name)`, `wh.Pipes.Set(ctx, name, def)`, and `wh.Pipes.Delete(ctx, name)`. ## Executing a pipe diff --git a/docs/src/content/docs/reverse-proxy.mdx b/docs/src/content/docs/reverse-proxy.mdx index 63e2ea95..f68dac7e 100644 --- a/docs/src/content/docs/reverse-proxy.mdx +++ b/docs/src/content/docs/reverse-proxy.mdx @@ -82,7 +82,7 @@ handle_path /api/wavehouse/* { The ingress controller forwards the full path unless you ask it to rewrite. Pair a capture-group path (`/api/wavehouse(/|$)(.*)` with `pathType: ImplementationSpecific`) with the `nginx.ingress.kubernetes.io/rewrite-target: /$2` annotation, or the prefix arrives at WaveHouse unstripped. ::: -Point the SDK at the prefixed URL and it does the rest — `createClient({ baseURL: 'https://app.example.com/api/wavehouse' })` sends both REST calls and SSE streams under the prefix ([SDK → Serving under a path prefix](/sdk#serving-under-a-path-prefix)). +Point either SDK at the prefixed URL and it does the rest — `createClient({ baseURL: 'https://app.example.com/api/wavehouse' })` in TypeScript, `wavehouse.NewClient(wavehouse.Config{BaseURL: "https://app.example.com/api/wavehouse"})` in Go — both sending REST calls and SSE streams under the prefix ([SDK → Serving under a path prefix](/sdk#serving-under-a-path-prefix), [Go SDK → Creating a client](/sdk/go#creating-a-client)). ## Request-body size limits @@ -209,10 +209,10 @@ WaveHouse serves Kubernetes-convention probes on `:8080` (full behavior in [Depl - **`/livez`** — liveness; sticky-200 after first successful boot. Does not touch ClickHouse. - **`/readyz`** — readiness; issues a ClickHouse `Ping` on **every** call. Point your load balancer's (internal) health check here so it routes around an instance whose ClickHouse is unreachable. - **`/healthz`** — permanent alias of `/livez`. -- **`/v1/health`** — the SDK's content-free liveness ping; mirrors `/livez` (200 once booted) and never touches ClickHouse. +- **`/v1/health`** — the SDKs' content-free liveness ping; mirrors `/livez` (200 once booted) and never touches ClickHouse. :::caution[Recommended: keep the bare probe paths off the public vhost] -Expose only **`/v1/health`** (and your API) to the internet — route `/livez`, `/readyz`, and `/healthz` on a private/internal listener, not the public proxy. Your orchestrator still reaches them the internal way regardless: kubelet probes the container directly on `:8080`, and a load balancer health-checks the backend — neither goes through the public proxy. Two reasons to keep them internal: `/readyz` pings ClickHouse on every call, so a public `/readyz` lets an unauthenticated flood turn into a per-request backend ping; and the probes leak boot/readiness state. `/v1/health` is the safe public liveness endpoint because it answers the same "is this server up" question without touching ClickHouse — it's what the SDK's `wh.sys.health()` calls. +Expose only **`/v1/health`** (and your API) to the internet — route `/livez`, `/readyz`, and `/healthz` on a private/internal listener, not the public proxy. Your orchestrator still reaches them the internal way regardless: kubelet probes the container directly on `:8080`, and a load balancer health-checks the backend — neither goes through the public proxy. Two reasons to keep them internal: `/readyz` pings ClickHouse on every call, so a public `/readyz` lets an unauthenticated flood turn into a per-request backend ping; and the probes leak boot/readiness state. `/v1/health` is the safe public liveness endpoint because it answers the same "is this server up" question without touching ClickHouse — it's what the SDKs' liveness ping calls (`wh.sys.health()` / `wh.Sys.Health(ctx)`). ::: ## Timeouts and slow links diff --git a/docs/src/content/docs/sdk/admin.md b/docs/src/content/docs/sdk/admin.md index c1523fb9..852ea02b 100644 --- a/docs/src/content/docs/sdk/admin.md +++ b/docs/src/content/docs/sdk/admin.md @@ -1,5 +1,5 @@ --- -title: "SDK Admin & System" +title: "TypeScript SDK Admin & System" description: "Schema introspection, access-control policy, DLQ stats, and health checks in @wavehouse/sdk." --- diff --git a/docs/src/content/docs/sdk/go/admin.md b/docs/src/content/docs/sdk/go/admin.md new file mode 100644 index 00000000..7e06eeed --- /dev/null +++ b/docs/src/content/docs/sdk/go/admin.md @@ -0,0 +1,100 @@ +--- +title: "Go SDK Admin & System" +description: "Schema introspection, access-control policy, DLQ stats, and health checks in the WaveHouse Go SDK." +--- + +Operational surfaces of `github.com/Wave-RF/WaveHouse/clients/go`. All except `client.Sys.Health` require the admin role (`policy.admin_role`)—see [Access Control](/access-control) and the TypeScript SDK's [Admin & System](/sdk/admin) page. + +Every namespace on this page is admin-gated: the server mounts them under `/v1/ops/*` behind one gate, which a caller clears either with a JWT resolving to the policy admin role (`admin_role`, `"admin"` by default) or with the server's non-JWT [operator key](/api#authentication) sent as `X-Operator-Key` via [`ClientOptions.Headers`](/sdk/go#clientoptions). + +## Schema — `client.Schema` + +Introspect ClickHouse table schemas. `Schema.List`, `Schema.Refresh`, and `From(t).Schema` hit the **admin-gated** `/v1/ops/schema*`; against any non-dev policy (anything but `default_role: admin`) build the client with an admin-role token or they return a `*wavehouse.Error` with `Status: 403`. + +```go +// List all table schemas. +schemas, err := wh.Schema.List(ctx) +// schemas is wavehouse.Schemas — map[string]TableSchema, keyed by table name + +// Force refresh from ClickHouse. +err = wh.Schema.Refresh(ctx) +``` + +Individual table schema: `wh.From("clicks").Schema(ctx)`. + +--- + +## Policy — `client.Policy` + +Manage Hasura-style access control policies (admin role required). + +```go +// Get current policy. +policy, err := wh.Policy.Get(ctx) + +// Update policy. +tenantFilter := "{{ jwt.app_metadata.tenant_id }}" +policyDraft := &wavehouse.Policy{ + DefaultRole: "viewer", + Tables: map[string]wavehouse.TablePolicy{ + "clicks": { + Select: map[string]wavehouse.RolePermissions{ + "viewer": { + AllowColumns: []string{"page", "button", "received_timestamp"}, + Filter: map[string]wavehouse.PolicyFilter{ + "tenant_id": {Eq: &tenantFilter}, + }, + }, + "admin": {AllowColumns: []string{"*"}}, + }, + }, + }, +} +err = wh.Policy.Set(ctx, policyDraft) + +// Validate without applying (dry run). +result, err := wh.Policy.Validate(ctx, policyDraft) +// result.Valid == true, or err wraps the validation failure details +``` + +`PolicyFilter` fields (`Eq`, `Neq`, `Gt`, `Lt`, `In`) are `*string` to distinguish empty strings from absent operators. Use a helper: + +```go +func strPtr(s string) *string { return &s } +``` + +--- + +## DLQ — `client.DLQ` + +Dead Letter Queue operations (admin role required). + +```go +// Get DLQ statistics. +stats, err := wh.DLQ.List(ctx) +// stats.Tables: map[string]int{"clicks": 3, "users": 0} +// stats.Total: 3 + +// Stats for a specific table. +stats, err = wh.DLQ.Table(ctx, "clicks") +``` + +`wh.DLQ.Stream(opts)` is **not yet functional**: no server-side DLQ stream exists (the SSE bridge carries only `ingest.>` subjects), so it connects and receives nothing. Tracked in [#197](https://github.com/Wave-RF/WaveHouse/issues/197). + +--- + +## System — `client.Sys` + +Server-online check. + +```go +// Health hits the public, content-free /v1/health route — 200 → nil error, +// any other status (including 503) → a non-nil *wavehouse.Error. +// Use it to check a server is reachable before sending data. +if err := wh.Sys.Health(ctx); err != nil { + // server is unreachable or not yet past boot + log.Println(err) +} +``` + +> Readiness (`/readyz`) is intentionally **not** exposed through the SDK — it runs a ClickHouse query per call and is a load-balancer / reverse-proxy concern. Probe it directly from your orchestrator. diff --git a/docs/src/content/docs/sdk/go/index.md b/docs/src/content/docs/sdk/go/index.md new file mode 100644 index 00000000..65afc19f --- /dev/null +++ b/docs/src/content/docs/sdk/go/index.md @@ -0,0 +1,198 @@ +--- +title: "Go SDK" +description: "Zero-dependency Go client SDK — query builder, real-time streaming, codegen." +--- + +`github.com/Wave-RF/WaveHouse/clients/go` — zero third-party runtime dependency Go client for WaveHouse (stdlib only). + +:::tip[Looking for the TypeScript SDK?] +This page and the rest of `/sdk/go/*` cover the Go client. The JavaScript/TypeScript client (`@wavehouse/sdk`) has its own docs starting at [SDK Overview](/sdk) — the two SDKs speak the same wire format, so anything you learn about WaveHouse's query builder, streaming, or admin endpoints on either page mostly carries over. +::: + +## Installation + +```bash +go get github.com/Wave-RF/WaveHouse/clients/go +``` + +Requires Go 1.24+ (the `go.mod` floor, matching supported releases rather than server's patch-pinned toolchain). + +## Import + +```go +import wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +``` + +Aliasing `wavehouse` is optional but keeps call sites short; all examples here assume it. + +## Quick Start + +```go +package main + +import ( + "context" + "fmt" + "log" + + wavehouse "github.com/Wave-RF/WaveHouse/clients/go" +) + +func main() { + wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), + }) + + page, err := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(context.Background()) + if err != nil { + log.Fatal(err) + } + for _, row := range page.Data { + fmt.Println(row["page"], row["button"]) + } +} +``` + +Find more examples in the [README](https://github.com/Wave-RF/WaveHouse/blob/main/clients/go/README.md). + +## Creating a Client + +```go +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "https://wavehouse.example.com", + Auth: func(ctx context.Context) (string, error) { + return myAuthProvider.GetToken(ctx) + }, + Options: &wavehouse.ClientOptions{ + MaxRetries: 2, + }, +}) +``` + +### `Config` + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `BaseURL` | `string` | — | Required WaveHouse server URL, optionally with a path prefix. A trailing `/` is trimmed and every request path is appended on both transports, so a server under `https://app.example.com/wavehouse` works as-is. | +| `Auth` | `func(context.Context) (string, error)` | `nil` | Token provider called before each request. `nil` means unauthenticated access. | +| `Options` | `*ClientOptions` | `nil` | Transport tuning (see below). | +| `HTTPClient` | `*http.Client` | fresh `&http.Client{}` | Override for custom TLS, proxies, or test transports. | + +:::caution[Timeouts: use contexts, not `http.Client.Timeout`] +The default client has no `Timeout`; use a `context.Context` deadline to prevent hangs. If supplying your own `HTTPClient`, leave `Timeout` unset, as it would kill long-lived SSE streams and force reconnect loops. Use `Transport`-level dial/TLS/response-header timeouts instead. +::: + +### `ClientOptions` + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `MaxRetries` | `int` | `2` | Retry attempts for retryable errors (5xx, 429, network failures). | +| `Headers` | `map[string]string` | `nil` | Sent on every request the client makes — REST calls and SSE streams alike. | + +`*Client` is safe for concurrent use; state is immutable after `NewClient` and builder chains copy. Ensure your `Auth` function is concurrency-safe. + +:::caution[`Options` opts you out of the default, not just in] +The 2-retry default only applies if `Config.Options` is `nil`. If `Options` is provided, an unset `MaxRetries` field defaults to Go's int zero value (`0`), which explicitly disables retries. Passing `&wavehouse.ClientOptions{}` removes the default retry behavior. +::: + +`Headers` is the Go analog of the TypeScript SDK's [`options.headers`](/sdk#custom-headers) — a gateway credential, a tenant selector, or tracing metadata that has no first-class option. It is also how an operator sends the server's non-JWT [operator key](/api#authentication): + +```go +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Options: &wavehouse.ClientOptions{ + MaxRetries: 2, // Options opts out of the default — set it explicitly. + Headers: map[string]string{"X-Operator-Key": os.Getenv("WH_OPERATOR_KEY")}, + }, +}) +``` + +The SDK's own headers win: `Authorization`, `Accept`, `Content-Type`, and the stream's `Cache-Control` are set after yours and overwrite any entry that collides. Names are matched case-insensitively (`net/http` canonicalizes them), and each entry replaces rather than appends. The map is copied at `NewClient`, so mutating it afterwards changes nothing. + +For the two remaining TypeScript knobs there is no Go field, because `Config.HTTPClient` already covers them: `options.fetch` maps to supplying your own `*http.Client`, and `options.fetchOptions` maps to a custom `http.RoundTripper` on that client's `Transport`. + +For static tokens, use `wavehouse.StaticToken(token)`: + +```go +wh := wavehouse.NewClient(wavehouse.Config{ + BaseURL: "http://localhost:8080", + Auth: wavehouse.StaticToken("your-jwt"), +}) +``` + +:::note[How the token is transmitted] +The Go SDK sends `Authorization: Bearer ` on every request, including SSE streams, and never uses a `?token=` query fallback. Both SDKs work this way: the TypeScript SDK streams over `fetch` rather than `EventSource` for exactly this reason, so header auth is now the shared behavior rather than a Go-only property (see its [equivalent note](/sdk#creating-a-client)). The token is re-read from `Auth` on every reconnect attempt, so a rotating token keeps a long-lived stream alive. +::: + +:::caution[A credentialed stream will not follow a redirect] +When the stream request carries a credential — an `Auth` token or a `ClientOptions.Headers` entry — the SDK refuses any 3xx and fails the stream with a terminal `SSE_REDIRECT`. Following it would either strip `Authorization` on a cross-host hop and silently downgrade the stream to `default_role`, or forward your configured headers to wherever the redirect points. Uncredentialed streams follow redirects normally. +::: + +:::caution[Use HTTPS for authenticated non-local servers] +While the SDK allows `http://` for local development or private networks, bearer tokens over plaintext HTTP are insecure. Use `https://` for endpoints outside trusted networks. +::: + +## Typed Rows (Generics) + +Pass a row type parameter to decode results into your struct instead of `map[string]any`: + +```go +type ClickRow struct { + Page string `json:"page"` + Button string `json:"button"` + DurationMS int `json:"duration_ms"` +} + +page, err := wavehouse.FetchTyped[ClickRow](ctx, + wh.From("clicks").Select("page", "button", "duration_ms").Limit(100), +) +// page.Data is []ClickRow +``` + +Use the [codegen CLI](/sdk/go/reference#codegen-cli) to generate row structs from a running server. + +`FetchTyped`, `Fetch[Row]` (pipes), and `SQL[Row]` (raw SQL) are package-level generic functions because Go lacks generic methods. Untyped equivalents (`.FetchUntyped(ctx)`) are ordinary methods. + +## Error Handling + +Request-response operations (queries, ingest, pipes, admin) return `(T, error)` or just `error` if no body exists (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`). HTTP exchange errors are `*wavehouse.Error`; unwrap via `errors.As`. Client-side failures (e.g., `Auth` provider, marshal errors) are plain wrapped errors; handle the `errors.As == false` case. Streaming methods (`Stream`, `Subscribe`, `Close`, `Connected`) use callbacks or plain errors; see [Streaming](/sdk/go/streaming). + +```go +page, err := wh.From("clicks").Fetch(ctx) +if err != nil { + var whErr *wavehouse.Error + if errors.As(err, &whErr) { + fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } else { + fmt.Println("client-side failure:", err) // auth provider, marshal, ... + } + return err +} +``` + +See [Reference → Error Handling](/sdk/go/reference#error-handling) for retry behavior and error codes. + +## Differences from the TypeScript SDK + +Both SDKs share a wire format and feature set, verified by a shared `wire_cases.json` fixture in CI to ensure equivalent HTTP requests for builder calls. However, API shapes differ: + +- **No `Result` union.** Go returns `(T, error)`. A non-nil `error` is the only failure signal; no `{ok, data, error}` objects or `error: null` sentinels are used. +- **`context.Context` instead of `AbortSignal`.** Non-streaming calls take `ctx context.Context` as the first argument. Use timeout or `cancel()` instead of `AbortController`. See [Reference → Context Cancellation](/sdk/go/reference#context-cancellation). +- **Streams closed explicitly.** `TableRef.Stream` and `QueryBuilder.Stream` omit `context.Context`. The returned `*StreamController` manages its own goroutine and connection, torn down by `.Close()` (deferred `stream.Close()` is usual). See [Streaming](/sdk/go/streaming). +- **Generics on package functions.** Go lacks type parameters on methods; use `FetchTyped[Row]`, `Fetch[Row]`, or `SQL[Row]`. +- **No implicit "await."** Call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly; `QueryBuilder` is not `PromiseLike`. +- **No third-party dependencies.** The Go SDK is stdlib-only, including its SSE frame parser. The TypeScript SDK carries exactly one runtime dependency (`eventsource-parser`, ~1.4 KB gzipped). +- **Any slice batches.** Reflection allows `[]ClickRow{...}` to use the same NDJSON batch path as `[]map[string]any`. See [Queries → Insert](/sdk/go/queries#insertctx-data). + +## Explore the Go SDK + +- [Queries](/sdk/go/queries) — Tables, chainable query builder, pagination, and raw SQL. +- [Streaming & Live Queries](/sdk/go/streaming) — SSE streams, client-side filtering, and backfill-then-live queries. +- [Pipes](/sdk/go/pipes) — Manage named query pipes. +- [Admin & System](/sdk/go/admin) — Schema introspection, access-control policy, DLQ stats, and health checks. +- [Reference & CLI](/sdk/go/reference) — Error codes, context cancellation, API tree, and codegen CLI. diff --git a/docs/src/content/docs/sdk/go/pipes.md b/docs/src/content/docs/sdk/go/pipes.md new file mode 100644 index 00000000..7ec96f91 --- /dev/null +++ b/docs/src/content/docs/sdk/go/pipes.md @@ -0,0 +1,85 @@ +--- +title: "Go SDK Pipes" +description: "Execute and manage named query pipes with the WaveHouse Go SDK." +--- + +Named pipes are server-defined, parameterized queries ([Named Pipes guide](/pipes)). The SDK executes them for allowed roles and manages definitions under the admin role. Compare with the TypeScript SDK's [Pipes](/sdk/pipes) page. + +## Named Pipes — `client.Pipe(name, params)` + +Execute a pre-defined named query pipe. Returns a `*PipeRef`. Unlike the TypeScript SDK's `PromiseLike` `PipeRef`, you must explicitly call `.FetchUntyped(ctx)` or the package-level `wavehouse.Fetch[Row]`. + +```go +rows, err := wavehouse.Fetch[map[string]any](ctx, + wh.Pipe("top_pages", map[string]any{"start_date": "2026-01-01", "limit": 50}), +) +``` + +### `wavehouse.Fetch[Row](ctx, pipeRef)` + +Execute and decode results into `[]Row`. Package-level generic function (Go has no generic methods) — same pattern as `FetchTyped` for queries and `SQL` for raw SQL. + +```go +type TopPage struct { + Page string `json:"page"` + Views int `json:"views"` +} + +rows, err := wavehouse.Fetch[TopPage](ctx, wh.Pipe("top_pages", map[string]any{"limit": 50})) +``` + +### `.FetchUntyped(ctx)` + +Execute and decode results into `[]map[string]any`. The non-generic method form of `Fetch`. + +```go +rows, err := wh.Pipe("top_pages", nil).FetchUntyped(ctx) +``` + +Pass `nil` for `params` if the pipe takes none or only requires server-side defaults. + +### `.Stream(opts)` + +Open a live stream from the pipe's underlying query; see [Streaming](/sdk/go/streaming). Streams by table name using the pipe's own name, so it works only when that name is a valid table name — the same limitation as the TypeScript SDK's `PipeRef.stream()`. + +```go +stream := wh.Pipe("top_pages", nil).Stream(nil) +``` + +--- + +## Pipes Admin — `client.Pipes` + +Manage named query pipes. These sit behind the admin gate on `/v1/ops/*`, which a caller clears one of two ways: a JWT resolving to the policy admin role (`policy.admin_role`), or the server's non-JWT [operator key](/api#authentication) sent as `X-Operator-Key` via [`ClientOptions.Headers`](/sdk/go#clientoptions). + +```go +// List all pipes. +pipes, err := wh.Pipes.List(ctx) + +// Get a single pipe definition. +pipe, err := wh.Pipes.Get(ctx, "top_pages") + +// Create or update. +err = wh.Pipes.Set(ctx, "top_pages", wavehouse.PipeDef{ + SQL: "SELECT page, count() as views FROM clicks GROUP BY page LIMIT {{limit}}", + Parameters: []wavehouse.ParamDef{ + {Name: "limit", Type: "number", Required: false, Default: 100}, + }, + Description: "Top pages by view count", + AllowedRoles: []string{"viewer", "admin"}, +}) + +// Delete. +err = wh.Pipes.Delete(ctx, "old_pipe") +``` + +`PipeDef` is `Pipe` minus `Name` — the name is the method's path argument: + +```go +type PipeDef struct { + SQL string + Parameters []ParamDef + Description string + AllowedRoles []string +} +``` diff --git a/docs/src/content/docs/sdk/go/queries.md b/docs/src/content/docs/sdk/go/queries.md new file mode 100644 index 00000000..c1fc8e2a --- /dev/null +++ b/docs/src/content/docs/sdk/go/queries.md @@ -0,0 +1,325 @@ +--- +title: "Go SDK Queries" +description: "Tables, the chainable query builder, pagination, and raw SQL in the WaveHouse Go SDK." +--- + +Reading and writing data with `github.com/Wave-RF/WaveHouse/clients/go`: table references, the chainable query builder, cursor pagination, and the admin-only raw-SQL escape hatch. Every request-response operation takes a `context.Context` as its first argument and returns `(T, error)`; the chainable builder methods and `.Stream(opts)` are the exceptions — see [Error Handling](/sdk/go#error-handling). Compare with the TypeScript SDK's [Queries](/sdk/queries) page, which covers the same surface with a `Result`-returning, `PromiseLike` builder. + +## Tables — `client.From(table)` + +`From` returns a `*TableRef`. It performs no request, making it safe to store or pass around. + +```go +clicks := wh.From("clicks") +``` + +### `.Fetch(ctx)` + +Shortcut for "select every column" with a default limit of 1000 (`wavehouse.DefaultLimit`). Internally it is `t.SelectAll().Limit(DefaultLimit).FetchUntyped(ctx)`. Unlike the TypeScript SDK's `.fetch(opts?)`, there is no options struct to override the limit or attach anything per-call; chain `.SelectAll().Limit(n)` yourself ([Query Builder](#query-builder)). + +Access-control policies restrict returned columns; `.Fetch()` cannot bypass `deny_columns`/`allow_columns` (see [Access control](/access-control#column-permissions)). + +```go +page, err := clicks.Fetch(ctx) +if err != nil { + log.Fatal(err) +} +for _, row := range page.Data { + fmt.Println(row["page"]) +} +``` + +For pagination, use the query builder with `.OrderBy()` (see [Pagination](#pagination)). + +### `.Insert(ctx, data)` + +Inserts one or many rows based on the input type: + +- **Map or struct** (excluding slices and `[]byte`): Sent as JSON via `POST /v1/ingest?table={table}`. For raw NDJSON, use `.InsertNDJSON`. +- **Any slice** (`[]map[string]any`, `[]ClickRow`, etc.): Serialized to NDJSON via reflection and sent as one `application/x-ndjson` request. Per-record outcomes are returned in the result. + +```go +// Single row → InsertResult{OK: true} (or Duplicate: &true when dedup skips it) +res, err := clicks.Insert(ctx, map[string]any{"page": "/home", "button": "cta"}) + +// Many rows (map slice) → one NDJSON request, per-record summary +res, err = clicks.Insert(ctx, []map[string]any{ + {"page": "/home", "button": "cta"}, + {"page": "/about", "button": "nav"}, +}) +// res.OK, res.Total, res.Succeeded, res.Failed, res.Duplicates, res.Results + +// Many rows (typed slice) — same NDJSON path, via reflection +type ClickRow struct { + Page string `json:"page"` + Button string `json:"button"` +} +res, err = clicks.Insert(ctx, []ClickRow{ + {Page: "/home", Button: "cta"}, + {Page: "/about", Button: "nav"}, +}) +``` + +For batches, `res.OK` is `true` only if all records succeeded (`*res.Failed == 0`). Check `res.Failed` and `res.Results` (each `InsertRecordResult{Index, OK, Duplicate, Error}`, 1-based `Index`) for partial failures. The returned `error` indicates whole-request failures (network, `404`, `403`, `503`). Empty slices are no-ops. + +> The server is format-agnostic: `POST /v1/ingest` also accepts a raw JSON array or a single object (`Content-Type` is only a hint). See [API reference](/api#post-v1ingesttabletable--ingest-data). + +### `.InsertNDJSON(ctx, ndjson)` + +Inserts pre-formatted NDJSON as a `string` without parsing it into Go values. Returns the same summary as slice `Insert`. + +```go +// From a literal string. +res, err := clicks.InsertNDJSON(ctx, `{"page":"/a"}`+"\n"+`{"page":"/b"}`) + +// From a file on disk. +raw, err := os.ReadFile("events.ndjson") +if err != nil { + log.Fatal(err) +} +res, err = clicks.InsertNDJSON(ctx, string(raw)) +``` + +### `.Schema(ctx)` + +Fetch table column definitions from ClickHouse. Admin-only. + +```go +schema, err := clicks.Schema(ctx) +// schema.Name == "clicks" +// schema.Columns: []Column{{Name: "page", Type: "String", IsNullable: false, HasDefault: false}, ...} +``` + +### `.Select(...columns)` + +Start a query builder chain. See [Query Builder](#query-builder). + +```go +page, err := clicks.Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Limit(10). + FetchUntyped(ctx) +``` + +### `.SelectAll()` + +Selects every column your role is allowed to read. This is the explicit version of `.Fetch()`. It is mutually exclusive with `.Select(...)` and aggregations (`.Count()`, `.Sum()`). For restricted roles, the server expands this to allowed columns rather than a bare `SELECT *`; it never bypasses `deny_columns`/`allow_columns` (see [Access control → Column permissions](/access-control#column-permissions)). + +```go +page, err := clicks.SelectAll().Where("country", wavehouse.OpEq, "US").Limit(10).FetchUntyped(ctx) +``` + +### `.Stream(opts)` + +Open a real-time event subscription. See [Streaming](/sdk/go/streaming). + +```go +stream := clicks.Stream(&wavehouse.StreamOptions{Since: "2026-01-01T00:00:00Z"}) +``` + +## Query Builder + +Returned by `tableRef.Select()`. Immutable—every chain method returns a new `*QueryBuilder`. Unlike the TypeScript SDK, Go builders do not auto-execute; call `.FetchUntyped(ctx)` or `wavehouse.FetchTyped[Row](ctx, builder)` explicitly: + +```go +page, err := clicks.Select("page").Limit(10).FetchUntyped(ctx) +``` + +### Chain Methods + +All methods return a new `*QueryBuilder`; the original remains unchanged. + +#### `.Select(...columns)` + +Append columns to the SELECT clause. A literal `"*"` is treated as a column named `*`—use `.SelectAll()` for all columns. + +```go +q := clicks.Select("page").Select("button") // SELECT page, button +``` + +#### `.SelectAll()` + +Selects every readable column (expanded server-side based on role). Mutually exclusive with `.Select(...)` and aggregations (`.Count()`, `.Sum()`, etc.). + +```go +q := clicks.Select().SelectAll().Where("country", wavehouse.OpEq, "US") +``` + +#### `.Where(column, op, value)` + +Add a filter using `FilterOp` constants: + +```go +clicks.Select("page"). + Where("score", wavehouse.OpGt, 10). + Where("page", wavehouse.OpLike, "/home%") +``` + +| `FilterOp` constant | Backend wire token | Description | +|----------------------|---------------------|--------------| +| `wavehouse.OpEq` | `eq` | Equal | +| `wavehouse.OpNeq` | `neq` | Not equal | +| `wavehouse.OpGt` | `gt` | Greater than | +| `wavehouse.OpGte` | `gte` | Greater than or equal | +| `wavehouse.OpLt` | `lt` | Less than | +| `wavehouse.OpLte` | `lte` | Less than or equal | +| `wavehouse.OpIn` | `in` | Value in array (accepts any Go slice) | +| `wavehouse.OpLike` | `like` | SQL LIKE pattern | +| `wavehouse.OpNotLike` | `not_like` | SQL NOT LIKE — **client-side only**; `/v1/query` rejects this token | + +#### Aggregations + +```go +clicks.Select("page"). + Count("*", "total"). // COUNT(*) + Sum("score", "total_score"). // SUM(score) + Avg("score", "avg_score"). // AVG(score) + Min("score", "min_score"). // MIN(score) + Max("score", "max_score"). // MAX(score) + CountDistinct("page", "unique_pages"). + Aggregate("uniqExact", "user_id", "unique_users") // allowlisted fn +``` + +Custom functions via `.Aggregate(fn, column, alias)` are validated server-side (case-insensitive). Allowlist: `count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, `any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, `stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Others return `400 unsupported aggregation function`. + +`Count`/`Sum`/`Avg`/`Min`/`Max`/`CountDistinct` take `(column, alias)`; `Aggregate` takes `(fn, column, alias)`. Empty-alias defaults: `Count` → `count` (and `column=""` becomes `*`); `Sum`/`Avg`/`Min`/`Max` → `sum_`/`avg_`/`min_`/`max_`; `CountDistinct` uses `count_distinct_`. `Aggregate` has no default; pass one or it is sent as `""`. + +#### `.GroupBy(...columns)` + +```go +clicks.Select("page").Count("", "").GroupBy("page") +``` + +#### `.OrderBy(column, dir)` + +```go +clicks.Select("page").Count("", "total").OrderBy("total", "desc") +``` + +`dir` defaults to `"asc"` if `""`. + +#### `.Limit(n)` + +```go +clicks.Select().Limit(100) +``` + +If unspecified, `wavehouse.DefaultLimit` (1000) is applied. The server also enforces a maximum (`query.default_max_rows`, default 10,000). + +#### `.TimeRange(column, since, until)` + +Filter by time window. `since`/`until` accept RFC3339 timestamps or relative durations (`"1h"`, `"30m"`, `"7d"`, `"2w"`; day/week suffixes expand to hours, so `"7d"` is `"168h"`). Pass `""` for `until` for open-ended ranges. + +```go +clicks.Select("page").TimeRange("received_timestamp", "1h", "") +clicks.Select("page").TimeRange( + "received_timestamp", "2026-01-01T00:00:00Z", "2026-02-01T00:00:00Z", +) +``` + +#### `.CacheTTL(seconds)` + +Sets a desired result-cache TTL. **Currently client-side only**; the server derives TTL adaptively from execution time. See [#280](https://github.com/Wave-RF/WaveHouse/issues/280). + +```go +clicks.Select("page").Count("", "").CacheTTL(300) // not yet honored server-side — see #280 +``` + +### `wavehouse.FetchTyped[Row](ctx, q)` + +Executes the query and decodes rows into `[]Row`. + +```go +type PageCount struct { + Page string `json:"page"` + Count int `json:"total"` +} + +page, err := wavehouse.FetchTyped[PageCount](ctx, + clicks.Select("page").Count("*", "total").GroupBy("page"), +) +// page.Data is []PageCount +``` + +### `.FetchUntyped(ctx)` + +Executes the query and decodes rows into `[]map[string]any`. + +```go +page, err := clicks.Select("page").OrderBy("page", "asc").Limit(50).FetchUntyped(ctx) +if err != nil { + return err +} + +if page.HasMore && page.Next != nil { + page, err = page.Next(ctx) // cursor-based pagination — needs OrderBy + if err != nil { + return err + } +} +``` + +### `.Stream(opts)` + +Opens a live stream from the builder's table with client-side filtering and projection. See [Streaming](/sdk/go/streaming). + +### Pagination + +`Page[T]`: + +```go +type Page[T any] struct { + Data []T + HasMore bool + Next func(ctx context.Context) (*Page[T], error) // nil when no cursor is available +} +``` + +If `Limit` is set and results meet that limit, `HasMore` is `true`. `Next` walks the **first** `.OrderBy()` column using a filter on the last row's value; thus, `Next` requires an explicit `.OrderBy()`. Without one, `Next` is `nil`. If the order column is omitted from `.Select(...)`, `Next` returns an empty page. + +The cursor filter is strict (`gt`/`lt` on the first `.OrderBy()` column, no tie-breaker), so rows sharing a boundary value with the last row are skipped. Paginate on a per-row-unique column, or accept dropped ties; the TypeScript SDK's `next()` has the same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). + +On the untyped path (`FetchUntyped` / `TableRef.Fetch`), JSON numbers decode as `float64`, so integer cursors lose exactness past 2^53 and pagination can repeat or skip a row. `FetchTyped` with an `int64` field, or codegen structs, keep it exact. + +```go +page, err := clicks.Select(). + OrderBy("received_timestamp", "desc"). + Limit(100). + FetchUntyped(ctx) +if err != nil { + log.Fatal(err) +} + +allRows := append([]map[string]any(nil), page.Data...) +for page.HasMore && page.Next != nil { + page, err = page.Next(ctx) + if err != nil { + log.Fatal(err) + } + allRows = append(allRows, page.Data...) +} +``` + +## Raw SQL — `wavehouse.SQL[Row](ctx, client, query)` + +Execute a raw SQL query via `/v1/ops/query`. This endpoint is admin-only: JWT tokens must resolve to the admin role (`admin_role`, default `"admin"`). Requests without valid tokens fall back to `default_role` and are rejected unless `default_role` is set to admin (dev-only). Alternatively, an operator key (`Authorization: Operator ` or `X-Operator-Key`) authorizes `/v1/ops/*`. Since `Config.Auth` uses `Bearer `, provide a `Config.HTTPClient` with a `Transport` that sets the `X-Operator-Key` header to use an operator key. Use `map[string]any` for dynamic schemas. + +```go +rows, err := wavehouse.SQL[map[string]any](ctx, wh, + "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") + +// Or decode into a struct that matches the projected columns/aliases. +// NOTE: this path forwards ClickHouse's own JSON, which QUOTES 64-bit +// integers (count() is UInt64) — decode them with the `,string` tag, or +// use map[string]any. See Reference → Codegen CLI for the full story. +type PageTotal struct { + Page string `json:"page"` + Total uint64 `json:"total,string"` +} +typed, err := wavehouse.SQL[PageTotal](ctx, wh, + "SELECT page, count() AS total FROM clicks GROUP BY page LIMIT 10") +``` + +:::note[No parameter binding through the SDK] +Positional `?` substitution is unsupported. The SDK cannot forward ClickHouse named params (`WHERE id = {id:UInt32}` + `param_id=42`) because the proxy blocks arbitrary query-string params and `SQL[Row]` lacks a hook to add them. Use inline literals or the structured query builder (`wh.From(table)...`) for safe binding of user input. +::: diff --git a/docs/src/content/docs/sdk/go/reference.md b/docs/src/content/docs/sdk/go/reference.md new file mode 100644 index 00000000..b7123fce --- /dev/null +++ b/docs/src/content/docs/sdk/go/reference.md @@ -0,0 +1,209 @@ +--- +title: "Go SDK Reference & CLI" +description: "Error codes, context cancellation, the full API tree, and the codegen CLI for the WaveHouse Go SDK." +--- + +Cross-cutting reference for `github.com/Wave-RF/WaveHouse/clients/go`: cancellation, the error model behind every request-response call's `(T, error)` return, the complete API tree at a glance, and the `wavehouse-codegen` tool that ships with the module. Compare with the TypeScript SDK's [Reference & CLI](/sdk/reference) page. + +## Context Cancellation + +Non-streaming operations take a `context.Context` as their first argument (similar to TypeScript's `AbortSignal`). Cancel it using a timeout or explicit `cancel()`: + +```go +ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) +defer cancel() + +page, err := wh.From("clicks").Fetch(ctx) +var whErr *wavehouse.Error +if errors.As(err, &whErr) && whErr.Code == "ABORTED" { + fmt.Println("Request timed out") +} +``` + +Cancellation returns immediately (no retry) with `&wavehouse.Error{Status: 0, Code: "ABORTED", Retryable: false}`. + +`.Stream(opts)` ignores `context.Context`; the returned `*StreamController` manages its own context and goroutine, closed via `.Close()`. See [Streaming](/sdk/go/streaming#streamoptions). + +## Error Handling + +The SDK never panics on API or network failures. Request-response operations (queries, ingest, pipes, admin) return `(T, error)`, while result-less operations (`Pipes.Set`/`Delete`, `Policy.Set`, `Schema.Refresh`, `Sys.Health`) return a bare `error`. + +HTTP exchange errors are `*wavehouse.Error` (unwrap via `errors.As`). Client-side failures (e.g., `Auth` provider, marshal failures) are plain wrapped errors; handle the `errors.As == false` case. Streaming methods (`Stream`, `Subscribe`, `Close`) do not return `(T, error)`; stream errors use the subscriber's `Error` callback. `Connected(ctx)` returns plain errors. This mirrors the TypeScript SDK's "never throws" guarantee. + +| Status | Code | Retryable | Description | +|--------|------|-----------|--------------| +| 400 | `HTTP_400` | No | Bad request (validation, missing fields) | +| 401 | `HTTP_401` | No | Invalid or expired JWT (missing tokens use `default_role`, resulting in success or 403) | +| 403 | `HTTP_403` | No | Insufficient permissions | +| 404 | `HTTP_404` | No | Table or pipe not found | +| 429 | `HTTP_429` | Yes | Rate limited (auto-retries, honoring `Retry-After`, capped at 30s) | +| 500 | `HTTP_500` | Yes | Server error (retried per `ClientOptions.MaxRetries`) | +| 503 | `HTTP_503` | Yes | Service unavailable (auto-retries, honoring `Retry-After`, capped at 30s) | +| 0 | `NETWORK_ERROR` | Yes | Network failure (retried with exponential backoff) | +| 0 | `ABORTED` | No | Request canceled via `context.Context` | +| 0 | `SSE_AUTH_ERROR` | Yes | The `Auth` provider returned an error for this attempt; the stream retries, so a token endpoint having a bad minute doesn't tear down a healthy stream | +| 0 | `SSE_NETWORK_ERROR` | Yes | Transport failure opening or holding the stream connection | +| 0 | `SSE_CONNECT_ERROR` | No | `BaseURL` is unparseable, or its scheme is not `http`/`https` — retrying cannot fix it | +| *3xx* | `SSE_REDIRECT` | No | The stream endpoint redirected while the request carried a credential, and the SDK refused to follow it | +| 200 | `SSE_BAD_CONTENT_TYPE` | No | A `200` that wasn't `text/event-stream` — something between you and WaveHouse answered (a captive portal, an auth gateway's login page) | +| 0 | `SSE_PARSE_ERROR` | Yes | A frame's JSON didn't decode; the frame is dropped and the stream continues | +| 0 | `SSE_READ_ERROR` | Yes | The connection failed mid-read; the stream reconnects from the last event ID | +| 0 | `SSE_ERROR` | Yes | Stream failure the SDK could not classify further | + +```go +page, err := wh.From("clicks").Fetch(ctx) +if err != nil { + var whErr *wavehouse.Error + if errors.As(err, &whErr) { + fmt.Println(whErr.Status, whErr.Code, whErr.Message, whErr.Retryable) + } else { + fmt.Println("client-side failure:", err) // auth provider, marshal, ... + } + return err +} +``` + +`wavehouse.IsRetryable(err)` shortcuts the `errors.As` + `.Retryable` check. + +Retries apply to all HTTP methods, matching TypeScript's `http.ts`. For `/v1/ingest`, at-least-once delivery on retry is a documented contract (see API docs ["At-least-once on retry"](/api#post-v1ingesttabletable--ingest-data)); use server-side dedup for duplicate suppression. `/v1/ops/query` (raw SQL) requires `admin_role`, so repeated execution on retry is an accepted risk. + +## Full API Tree + +```text +NewClient(Config) → *Client +├── .From(table) → *TableRef +│ ├── .Fetch(ctx) → (*Page[map[string]any], error) +│ ├── .Select(...cols) → *QueryBuilder +│ │ ├── .Select() .SelectAll() .Where() .Count() .Sum() .Avg() .Min() .Max() +│ │ │ .CountDistinct() .Aggregate() .GroupBy() .OrderBy() +│ │ │ .Limit() .TimeRange() .CacheTTL() +│ │ ├── FetchTyped[Row](ctx, q) → (*Page[Row], error) // package-level generic func +│ │ ├── .FetchUntyped(ctx) → (*Page[map[string]any], error) +│ │ ├── .Stream(opts) → *StreamController +│ │ └── .LiveQuery(sub, opts) → *LiveQueryHandle +│ ├── .SelectAll() → *QueryBuilder +│ ├── .Insert(ctx, data) → (*InsertResult, error) +│ ├── .InsertNDJSON(ctx, ndjson) → (*InsertResult, error) +│ ├── .Schema(ctx) → (*TableSchema, error) +│ └── .Stream(opts) → *StreamController +├── .Pipe(name, params) → *PipeRef +│ ├── Fetch[Row](ctx, p) → ([]Row, error) // package-level generic func +│ ├── .FetchUntyped(ctx) → ([]map[string]any, error) +│ └── .Stream(opts) → *StreamController +├── .Pipes (admin) → *PipesNamespace +│ ├── .List(ctx) → ([]Pipe, error) +│ ├── .Get(ctx, name) → (*Pipe, error) +│ ├── .Set(ctx, name, PipeDef) → error +│ └── .Delete(ctx, name) → error +├── SQL[Row](ctx, client, query) → ([]Row, error) // package-level generic func, admin-only +├── .Schema (admin) → *SchemaNamespace +│ ├── .List(ctx) → (Schemas, error) +│ └── .Refresh(ctx) → error +├── .Policy (admin) → *PolicyNamespace +│ ├── .Get(ctx) → (*Policy, error) +│ ├── .Set(ctx, *Policy) → error +│ └── .Validate(ctx, *Policy) → (*ValidationResult, error) +├── .DLQ (admin) → *DLQNamespace +│ ├── .List(ctx) → (*DLQStats, error) +│ ├── .Table(ctx, name) → (*DLQStats, error) +│ └── .Stream(opts) → *StreamController // not yet functional server-side — #197 +└── .Sys → *SysNamespace + └── .Health(ctx) → error + +*StreamController +├── .Subscribe(*StreamSubscriber) → func() // unsubscribe +├── .Events() → <-chan StreamEvent // idiomatic Go alternative to an async iterator +├── .Close() +├── .Status() → StreamStatus +└── .Connected(ctx) → error // Go-only addition, blocks until live +``` + +## Codegen CLI + +Generate Go structs from a running WaveHouse instance using the `wavehouse-codegen` command in `cmd/`: + +```bash +export WAVEHOUSE_AUTH='' # avoids leaking the token via argv +go run github.com/Wave-RF/WaveHouse/clients/go/cmd/wavehouse-codegen@latest \ + --url http://localhost:8080 \ + --out ./db_types.go \ + --package myapp +``` + +Or, inside `clients/go/`: + +```bash +go run ./cmd/wavehouse-codegen --url http://localhost:8080 --out ./db_types.go +``` + +Codegen reads the admin-only `/v1/ops/schema` endpoint; non-dev servers require an admin token or return `403`. Use `WAVEHOUSE_AUTH` instead of `--auth ` to keep tokens out of shell history and process listings. + +**Options:** + +| Flag | Description | Default | +|------|-------------|---------| +| `--url`, `-u` | WaveHouse base URL | `http://localhost:8080` | +| `--out`, `-o` | Output `.go` file path | `./wavehouse_types.go` | +| `--auth`, `-a` | Bearer token; prefer `WAVEHOUSE_AUTH` env var | `$WAVEHOUSE_AUTH` | +| `--package`, `-p` | Go package name for the generated file | `main` | +| `--help`, `-h` | Show usage and exit | — | + +Output is processed via `go/format`. If a table or column name produces invalid Go source, codegen fails loudly. + +**Example output:** + +```go +// Code generated by wavehouse-codegen. DO NOT EDIT. + +package myapp + +// ClicksRow represents a row in the "clicks" table. +type ClicksRow struct { + Page string `json:"page"` + Button string `json:"button"` + Score float64 `json:"score"` + ReceivedTimestamp *string `json:"received_timestamp,omitempty"` +} +``` + +(Example for the [development quick-start](/development#quick-start) `clicks` table; `received_timestamp` is `*string` + `,omitempty` due to its `DEFAULT` clause.) + +The generator does not special-case initialisms: `event_id` becomes `EventId`, not the Go-idiomatic `EventID` — each `_`-separated part just gets its first letter upper-cased. Table and column names are converted to `PascalCase`; leading digits get an `X` prefix (e.g., `2fa_events` $\rightarrow$ `X2faEventsRow`). Columns with `has_default: true` become pointer fields with `,omitempty`: `nil` uses the server default, a pointed-at value is sent — including an explicit `0`/`false`/`""`. + +**ClickHouse → Go type mapping:** + +| ClickHouse Type | Go Type | +|------------------|---------| +| `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Time`/`Time64`, `Enum8`/`Enum16`, `IPv4`/`IPv6` | `string` | +| `Bool` / `Boolean` | `bool` | +| `UInt8` / `UInt16` / `UInt32` / `UInt64` | `uint8` / `uint16` / `uint32` / `uint64` | +| `Int8` / `Int16` / `Int32` / `Int64` | `int8` / `int16` / `int32` / `int64` | +| `Float32`, `BFloat16` | `float32` | +| `Float64` | `float64` | +| `UInt128`/`UInt256`, `Int128`/`Int256` | `json.Number` | +| `Decimal*` | `string` | +| `Nullable(T)` | `*T` | +| `LowCardinality(T)` | same as `T` | +| `Array(T)` | `[]T` (except `Array(UInt8)` $\rightarrow$ `json.RawMessage` per [#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | +| `Map(K, V)` | `map[K]V` (fallback: `map[string]any`) | +| `SimpleAggregateFunction(fn, T)` | same as `T` (rollup tables from `AggregatingMergeTree`/`SummingMergeTree` generate usable structs) | +| anything unrecognized | `any` | + +Unlike the TypeScript SDK, Go codegen preserves ClickHouse integer **widths** (`UInt64` → `uint64`, not a generic `number`), so 64-bit columns decode exactly where TS hits the 2^53 ceiling. Generated structs target `/v1/query` and `/v1/pipes/*`. For the raw-SQL path (`/v1/ops/query`), which quotes 64-bit+ integers, use `map[string]any` with `SQL[Row]`. + +## Testing + +Unit tests are colocated in `clients/go/` (module `clients/go/go.mod`), separate from the root `WaveHouse` module. The cross-language wire-format **conformance suite** uses `clients/go/conformance_test.go` to replay the shared fixture (`clients/go/testdata/wire_cases.json`), asserting correct HTTP methods, paths, content types, and bodies. The TypeScript half—`tests/conformance/conformance_ts.mjs`, run via `make test-conformance-ts` (builds TS SDK first)—uses the same fixture; CI runs both to ensure wire format consistency. + +```bash +cd clients/go +go test ./... +``` + +E2E tests (build tag `e2e`) run against a live WaveHouse instance via a dedicated Make target: + +```bash +WAVEHOUSE_URL=http://localhost:8080 WAVEHOUSE_AUTH='' make test-go-sdk-e2e +``` + +`WAVEHOUSE_URL` defaults to `http://localhost:8080`; optional `WAVEHOUSE_AUTH` is for admin cases. The suite skips if the server is unreachable. Unlike the TypeScript SDK, Go isn't yet in the repo's `make test-e2e` harness (see [E2E Testing](/sdk/reference#e2e-testing)). diff --git a/docs/src/content/docs/sdk/go/streaming.md b/docs/src/content/docs/sdk/go/streaming.md new file mode 100644 index 00000000..e136c814 --- /dev/null +++ b/docs/src/content/docs/sdk/go/streaming.md @@ -0,0 +1,249 @@ +--- +title: "Go SDK Streaming & Live Queries" +description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in the WaveHouse Go SDK." +--- + +Real-time consumption with `github.com/Wave-RF/WaveHouse/clients/go`: SSE event streams from tables, builders, and pipes, plus live queries that backfill history before going live. Builders and table refs come from [Queries](/sdk/go/queries). Compare with the TypeScript SDK's [Streaming & Live Queries](/sdk/streaming) page — the two implement the same protocol and mostly the same client-side filtering, but connection lifecycle differs: Go streams are goroutine-backed and closed explicitly, not tied to a `context.Context` or a browser's `EventSource`. + +## Streaming + +Streams use SSE (Server-Sent Events) parsed via `net/http` with zero runtime dependencies. + +### `*StreamController` + +Returned by `.Stream(opts)` on `*TableRef`, `*QueryBuilder`, `*PipeRef`, and `*DLQNamespace` (DLQ is not yet functional server-side — [#197](https://github.com/Wave-RF/WaveHouse/issues/197)). Calling `.Stream` returns immediately; the connection opens in a background goroutine. + +```go +stream := wh.From("clicks").Stream(&wavehouse.StreamOptions{ + Since: "2026-01-01T00:00:00Z", +}) +defer stream.Close() +``` + +### `.Subscribe(sub) → func()` + +Callback-based consumption. Returns an unsubscribe function. The `Status` callback fires immediately with the current status. + +```go +unsub := stream.Subscribe(&wavehouse.StreamSubscriber{ + Next: func(e wavehouse.StreamEvent) { + // e: {Table: "clicks", Timestamp: "2026-...", Data: map[string]any{"page": "/", ...}} + fmt.Println("New event:", e.Data) + }, + Status: func(s wavehouse.StreamStatus) { + // s: StatusConnecting | StatusLive | StatusReconnecting | StatusClosed + updateIndicator(s) + }, + Error: func(err error) { + fmt.Println("Stream error:", err) + }, +}) + +// Cleanup — removes this subscriber; the connection stays open for any +// others and must still be closed with stream.Close() when you're done +// with the stream itself. +defer unsub() +``` + +Cleanup via `unsub()` removes the subscriber; the connection remains open for others and must be closed with `stream.Close()`. + +### Channel-based consumption — `.Events()` + +A read-only channel, closed automatically when the stream shuts down. + +```go +stream := wh.From("clicks").Stream(nil) +defer stream.Close() + +for event := range stream.Events() { + fmt.Println(event.Table, event.Data) + if shouldStop { + break + } +} +``` + +:::caution[`break` does not close the stream] +Unlike the TypeScript SDK's async iterator, where breaking a `for await` loop closes the connection, breaking a Go `for range stream.Events()` loop only stops consumption — the background goroutine and HTTP connection persist. Always `defer stream.Close()`. +::: + +The channel is buffered (256 events). A slow consumer makes the SDK **drop** new events for that channel rather than block the read loop. The first drop logs via `log`; later drops are silent (`.Subscribe` callbacks fire regardless). + +### `.Close()` + +Explicitly closes the stream and releases resources. Non-blocking and safe to call from inside a subscriber callback. + +```go +stream.Close() +``` + +### `.Status()` + +Returns the current `StreamStatus`. + +```go +status := stream.Status() +``` + +### `.Connected(ctx)` + +Blocks until the stream reaches `StatusLive` or `ctx` is canceled; returns an error if the stream closes before connecting. Useful for ensuring a stream is live (e.g., in tests). + +```go +ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) +defer cancel() +if err := stream.Connected(ctx); err != nil { + log.Fatal(err) +} +``` + +### `StreamOptions` + +| Field | Type | Description | +| ----- | ---- | ----------- | +| `Since` | `string` | RFC3339 timestamp for gap-fill replay | + +There's no `Signal`/context field: a stream isn't canceled by passing a `context.Context` into `.Stream()` — call `.Close()` instead. + +### `StreamEvent` + +```go +type StreamEvent struct { + Table string // table name (e.g. "clicks") + Timestamp string // received_timestamp (RFC3339Nano) + Data map[string]any // row data +} +``` + +Top-level `DateTime`/`DateTime64` values inside `Data` arrive in canonical RFC 3339 UTC, byte-identical to what `/v1/query` renders for the same stored value — the ingest handler rewrites them before publishing, so a live frame and a later query can't disagree on the spelling of an instant. Two consequences worth knowing: a value you sent as `2026-06-21T06:00:00.123+02:00` comes back as `2026-06-21T04:00:00.123Z` (same instant, different spelling), and the canonicalization is deliberately fail-open — a value the server can't parse, or one whose zone it can't resolve, is published verbatim. See [Timestamp canonicalization](/api#timestamp-canonicalization). + +:::note[`Events()` carries events only] +`Error` and `Status` are delivered exclusively via `.Subscribe(...)`. The channel closes on terminal errors (401/403/404). Pair `Events()` with a subscriber to determine why a stream ended. +::: + +:::note[The channel buffers from stream construction] +Events buffer (up to 256) starting at `.Stream()`; events arriving before the first `Events()` call are not lost. +::: + +### Transport Behavior + +| Transport | Reconnect | Protocol | +| --------- | --------- | -------- | +| SSE | Automatic, exponential backoff (max 30s), gap-fill replay via last event ID | HTTP/2 recommended | + +Reconnect covers transport failures and retryable responses (5xx/429, plus `SSE_AUTH_ERROR`, `SSE_PARSE_ERROR`, and `SSE_READ_ERROR`). Terminal failures fire the `Error` callback, set status `StatusClosed`, and stop: non-retryable HTTP statuses, `SSE_CONNECT_ERROR` (bad `BaseURL`), `SSE_REDIRECT` (a credentialed request was redirected), and `SSE_BAD_CONTENT_TYPE` (a `200` that wasn't an event stream). Every error reaches the callback as a `*wavehouse.Error`, so `errors.As` and `wavehouse.IsRetryable` work on all of them — see the [error-code table](/sdk/go/reference#error-handling). + +Note that `/v1/stream` is not admin-gated, so WaveHouse itself never answers a stream with `401`. A `401` on a stream came from something in front of it. `Auth` provider errors during (re)connect are retryable (`SSE_ERROR`) and reconnects continue — `ClientOptions.MaxRetries` bounds request retries only, not stream reconnects — so call `.Close()` if the provider fails permanently. + +Auth goes as an `Authorization: Bearer` header on every connection, re-read from `Auth` per attempt ([note in Getting Started](/sdk/go#creating-a-client)). The TypeScript SDK streams over `fetch` and authenticates the same way, so this is shared behavior rather than a Go-only property — what Go avoids is the browser's per-domain connection ceiling, not a different auth mechanism. + +Delivery across a reconnect is **at-least-once**: the server replays from the last event ID *inclusively*, so the first frame after a gap-fill is usually one you already saw. Replay reaches back only as far as the server's `mq.gap_window_minutes` (15 minutes by default); a longer outage resumes live with a hole. + +### Server-Side Policy Filtering + +Before anything reaches the client, the server applies the caller's policy to the stream: a table the role can't `select` never opens, denied columns are stripped from every frame, and a role carrying a row `filter` has non-matching rows withheld per subscriber — on live frames and on `Since` gap-fill replay alike. The claims are captured from the JWT at connect time. + +Two things follow. **Event-id gaps are normal on a filtered stream** — a gap means a row was withheld, not that a frame was dropped. And **the row filter fails closed**: a comparison the server can't prove — an unresolvable claim, a type it can't compare — withholds the row rather than passing it. See [Access control](/access-control#row-level-security). + +### Client-Side Stream Filtering + +When a `*QueryBuilder` with `.Where()` or `.Select()` calls `.Stream()`, filters are applied client-side: + +```go +stream := wh.From("clicks"). + Select("page", "button"). + Where("page", wavehouse.OpEq, "/home"). + Stream(nil) + +// Only events where page == "/home" are emitted, with only page + button fields +``` + +Supported operators: `OpEq`, `OpNeq`, `OpGt`, `OpGte`, `OpLt`, `OpLte`, `OpIn`, `OpLike`, `OpNotLike` — the `FilterOp` set `.Where()` takes everywhere (mapped to wire tokens `eq`/`neq`). `OpLike`/`OpNotLike` use SQL LIKE semantics (`%`, `_`), case-insensitively. `OpIn` accepts any Go slice type (e.g., `[]string`, `[]int`). + +#### How values are compared + +The client-side evaluator mirrors the server's row-filter comparison rules rather than comparing everything as text: + +- **Timestamps compare chronologically.** Since the server canonicalizes every top-level `DateTime`/`DateTime64` value to RFC 3339 UTC before publishing, a payload reads `2026-06-21T04:00:00Z` while your filter constant may name the same instant as `2026-06-21T06:00:00+02:00`. Comparing those as text is wrong in both directions — lexically the payload sorts *below* the constant, so `OpGte` would miss a row that is chronologically equal. Both sides are parsed as instants instead. +- **Only unambiguous spellings count as instants.** RFC 3339 with an explicit offset or `Z`. A zone-less spelling like `2026-06-21 04:00:00` names an instant only relative to the column's declared timezone, which the server reads from the schema and a stream subscriber does not have — guessing UTC would move the instant. Such a constant is not treated as a timestamp. +- **Ordering an instant against a non-instant fails closed.** If one side parses as a timestamp and the other does not, `OpGt`/`OpGte`/`OpLt`/`OpLte` withhold the row rather than falling back to text comparison, which could otherwise admit rows the query path excludes. The usual cause is a zone-less filter constant — give it an offset. +- **A missing column equals only `nil`.** A column absent from the payload does not match the string `""`. +- **Numbers compare numerically**, so `9 < 100` as you would expect rather than as text. + +:::caution[Integer precision above 2^53] +Event data decodes through `encoding/json` into `map[string]any`, so JSON numbers arrive as `float64`. An integer column beyond `Number.MAX_SAFE_INTEGER` (2^53) has already lost exactness before any filter runs — the server compares such columns in their exact storage domain, so a client-side filter on a very large `UInt64` can disagree with the server's verdict. Filter on a string or timestamp column instead when exactness at that magnitude matters. +::: + +## Live Queries + +Live queries combine a historical backfill (`.FetchUntyped`) with a real-time stream for seamless initial loads and updates. They are available only on `*QueryBuilder` (no `TableRef.LiveQuery` shortcut), matching the TypeScript SDK. + +```go +lq := wh.From("clicks"). + SelectAll(). + Where("page", wavehouse.OpEq, "/home"). + OrderBy("received_timestamp", "desc"). + Limit(100). + LiveQuery(&wavehouse.StreamSubscriber{ + Initial: func(rows []map[string]any, err error) { + // Called once with the historical backfill. + setRows(rows) + }, + Next: func(e wavehouse.StreamEvent) { + // Called for each live event after backfill. + addRow(e.Data) + }, + Error: func(err error) { + log.Println(err) + }, + }, nil) + +// Cleanup +defer lq.Close() +``` + +### `StreamSubscriber` + +```go +type StreamSubscriber struct { + // Initial is called once with historical backfill data (live queries only). + Initial func(rows []map[string]any, err error) + // Next is called for each live event. + Next func(event StreamEvent) + // Status is called when the connection status changes. + Status func(status StreamStatus) + // Error is called on stream errors. + Error func(err error) +} +``` + +:::note[`Initial` is always untyped] +Unlike the TypeScript SDK's `initial: (result: Result) => void`, Go's `LiveQuery` takes no type parameter: `Initial` always receives `[]map[string]any` plus a plain `error`, even if you'd use `wavehouse.FetchTyped[Row]` for the same query outside a live query. Decode inside the callback if needed. +::: + +### How it works + +1. Subscribes to the stream immediately and buffers events. +2. Runs `.FetchUntyped(ctx)` for historical data, then calls `sub.Initial(rows, err)`. +3. Deduplicates buffered events against the maximum `received_timestamp` in the backfill (not necessarily the last row). +4. Flushes remaining buffered events and switches to live mode. + +This "stream-first" approach prevents event loss between fetch and stream start. + +:::caution[Dedup needs `received_timestamp` in the projection] +Dedup relies on `received_timestamp`. `.SelectAll()` (or no projection) includes it; a `.Select(...)` omitting it disables dedup, causing events in the overlap window to be delivered twice (via `Initial` and `Next`). +::: + +:::caution[`OpLike` matching differs between backfill and live] +Client-side `OpLike` is case-insensitive, but server-side backfills use ClickHouse `LIKE`, which is case-sensitive. Consequently, a live query filtering on `OpLike` may exclude rows in the backfill that it includes in the live stream ([#451](https://github.com/Wave-RF/WaveHouse/issues/451)). + +`OpNotLike` is rejected by `/v1/query` with a `400`, causing `Initial` callbacks to fail. See [Queries](/sdk/go/queries#wherecolumn-op-value). +::: + +### `.Close()` + +Shuts down the live query and its underlying stream. Safe to call more than once (idempotent via `sync.Once`). + +```go +lq.Close() +``` diff --git a/docs/src/content/docs/sdk/index.mdx b/docs/src/content/docs/sdk/index.mdx index f5bc73bc..652f9ba5 100644 --- a/docs/src/content/docs/sdk/index.mdx +++ b/docs/src/content/docs/sdk/index.mdx @@ -7,6 +7,10 @@ import { Tabs, TabItem, LinkCard, CardGrid } from "@astrojs/starlight/components `@wavehouse/sdk` — TypeScript client for WaveHouse. One runtime dependency: `eventsource-parser` (~1.4 KB gzipped, itself dependency-free), which frames the SSE stream. +:::tip[Writing Go instead?] +WaveHouse also ships an official Go SDK (`github.com/Wave-RF/WaveHouse/clients/go`) — zero third-party dependencies, `context.Context`-first, generics for typed rows. See the [Go SDK docs](/sdk/go). The two clients speak the same wire format, so everything below about tables, the query builder, streaming, and admin endpoints carries over conceptually, but API and lifecycle details differ — Go uses context-first calls and package-level generics, and streams must be closed explicitly. +::: + ## Installation @@ -556,3 +560,15 @@ The full error-code table lives in [Error Handling](/sdk/reference#error-handlin href="/sdk/reference" /> + +## Go SDK + +Prefer Go? The same server, the same wire format, an idiomatic Go client: + + + + diff --git a/docs/src/content/docs/sdk/pipes.md b/docs/src/content/docs/sdk/pipes.md index 8b41f8af..cd1fad9e 100644 --- a/docs/src/content/docs/sdk/pipes.md +++ b/docs/src/content/docs/sdk/pipes.md @@ -1,5 +1,5 @@ --- -title: "SDK Pipes" +title: "TypeScript SDK Pipes" description: "Execute and manage named query pipes with @wavehouse/sdk." --- diff --git a/docs/src/content/docs/sdk/queries.md b/docs/src/content/docs/sdk/queries.md index e4e497cb..3b7f177b 100644 --- a/docs/src/content/docs/sdk/queries.md +++ b/docs/src/content/docs/sdk/queries.md @@ -1,5 +1,5 @@ --- -title: "SDK Queries" +title: "TypeScript SDK Queries" description: "Tables, the chainable query builder, pagination, and raw SQL in @wavehouse/sdk." --- @@ -81,7 +81,7 @@ const { data } = await clicks.select('page', 'button').where('page', '=', '/home ### `.selectAll()` -Selects **every column your role is allowed to read** — the explicit form of what a bare `.fetch()` does. Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.); the server expands it to your allowed columns (never a raw `SELECT *`) and never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). +Selects **every column your role is allowed to read** — the explicit form of what a bare `.fetch()` does. Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.); for a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`) and never bypasses `deny_columns`/`allow_columns`. See [Access control → Column permissions](/access-control#column-permissions). ```ts const { data } = await clicks.selectAll().where('country', '=', 'US').limit(10); @@ -121,7 +121,7 @@ const q = clicks.select('page').select('button'); // SELECT page, button #### `.selectAll()` -Select every column your role may read (the all-columns wildcard, expanded server-side to your allowed columns). Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.). +Select every column your role may read. For a column-restricted role the server expands it to exactly that role's allowed columns rather than a bare `SELECT *` (unrestricted/admin roles do get `SELECT *`). Mutually exclusive with `.select(...)` and with aggregations (`.count()`, `.sum()`, etc.). ```ts const q = clicks.selectAll().where('country', '=', 'US'); @@ -145,7 +145,7 @@ clicks.select('page').where('score', '>', 10).where('page', 'like', '/home%') | `'<='` | `lte` | Less than or equal | | `'in'` | `in` | Value in array | | `'like'` | `like` | SQL LIKE pattern. Case-**sensitive** on `/v1/query`; the client-side filter used by `.stream()` / `.liveQuery()` matches case-**insensitively** | -| `'not_like'` | — | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | +| `'not_like'` | `not_like` | SQL NOT LIKE — **client-side only** (live-query / stream filtering); the `/v1/query` backend rejects it | #### Aggregations @@ -157,9 +157,11 @@ clicks.select('page') .min('score', 'min_score') // MIN(score) .max('score', 'max_score') // MAX(score) .countDistinct('page', 'unique_pages') - .aggregate('uniqExact', 'user_id', 'unique_users') // custom fn + .aggregate('uniqExact', 'user_id', 'unique_users') // allowlisted fn ``` +Custom function names pass through `.aggregate(fn, column, alias)` but are validated server-side against a fixed allowlist (matched case-insensitively): `count`, `sum`, `avg`, `min`, `max`, `countDistinct`, `uniq`, `uniqExact`, `any`, `anyLast`, `argMin`, `argMax`, `groupArray`, `median`, `quantile`, `stddevPop`, `stddevSamp`, `varPop`, `varSamp`. Anything else is rejected with `400 unsupported aggregation function`. + Each aggregation method signature: `(column: string, alias?: string)`. `count()` defaults to `column='*'`, `alias='count'`. #### `.groupBy(...columns)` @@ -242,6 +244,10 @@ while (result.hasMore && result.next) { } ``` +The cursor filter is strict (`gt`/`lt` against the last row's value) and uses only that one order column, with no tie-breaker — so rows sharing the boundary value with the last row of a page are skipped. Paginate on a column that is unique per row (or made unique by a monotonic timestamp), or accept that ties at a page edge can be dropped. The Go SDK's `Next` has the same limitation ([#452](https://github.com/Wave-RF/WaveHouse/issues/452)). + +Rows decode with JSON numbers as JS `number`s, so an integer cursor column past `Number.MAX_SAFE_INTEGER` (2^53) loses exactness and pagination can repeat or skip a row at that scale. The Go SDK's `FetchTyped` with an `int64` field avoids this; there is no JS equivalent short of a string or `bigint` column. + --- ## Raw SQL — `wh.sql(query, opts?)` diff --git a/docs/src/content/docs/sdk/reference.md b/docs/src/content/docs/sdk/reference.md index bd511700..44832116 100644 --- a/docs/src/content/docs/sdk/reference.md +++ b/docs/src/content/docs/sdk/reference.md @@ -1,5 +1,5 @@ --- -title: "SDK Reference & CLI" +title: "TypeScript SDK Reference & CLI" description: "Error codes, AbortController, the full API tree, the codegen CLI, and E2E testing with @wavehouse/sdk." --- @@ -166,10 +166,11 @@ export interface ClicksRow { | ClickHouse Type | TypeScript Type | |----------------|-----------------| | `String`, `FixedString`, `UUID`, `DateTime*`, `Date*`, `Enum*`, `IPv4/6` | `string` | -| `UInt*`, `Int*`, `Float*`, `Decimal*` | `number` | +| `UInt*`, `Int*`, `Float*` | `number` | +| `Decimal*` | `number` *(generated)* — but `/v1/query` returns Decimals as **quoted strings**, so treat the field as `string` until codegen is fixed ([#453](https://github.com/Wave-RF/WaveHouse/issues/453)) | | `Bool` | `boolean` | | `Nullable(T)` | `T \| null` | -| `Array(T)` | `T[]` | +| `Array(T)` | `T[]` — except `Array(UInt8)`, which `/v1/query` base64-encodes, so the generated `number[]` is a `string` at runtime ([#436](https://github.com/Wave-RF/WaveHouse/issues/436)) | | `Map(K, V)` | `Record` | | `LowCardinality(T)` | same as `T` | diff --git a/docs/src/content/docs/sdk/streaming.md b/docs/src/content/docs/sdk/streaming.md index a7f8e3ed..c4b7dc69 100644 --- a/docs/src/content/docs/sdk/streaming.md +++ b/docs/src/content/docs/sdk/streaming.md @@ -1,5 +1,5 @@ --- -title: "SDK Streaming & Live Queries" +title: "TypeScript SDK Streaming & Live Queries" description: "Real-time SSE streams, client-side filtering, and backfill-then-live queries in @wavehouse/sdk." --- diff --git a/docs/src/content/docs/why-wavehouse.md b/docs/src/content/docs/why-wavehouse.md index 6798ded0..8aaf5798 100644 --- a/docs/src/content/docs/why-wavehouse.md +++ b/docs/src/content/docs/why-wavehouse.md @@ -157,7 +157,7 @@ flowchart TB | Schema validation | Custom code in ingest API | Built in (discovers `system.columns`) | | Row/column access control | Custom middleware or a dedicated service | Built in (Hasura-style, JWT-driven) | | Dead letter queue | Custom retry + dead topic on Kafka | Built in (`WAVEHOUSE_DLQ`) | -| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript, one dependency, codegen) | +| Client SDK | Each team writes one | `@wavehouse/sdk` (TypeScript, one dependency, codegen) + `github.com/Wave-RF/WaveHouse/clients/go` (Go, zero dependencies, codegen) | The DIY path works — big teams run it — but the ops cost is not small. You're paying for a Kafka cluster (or Confluent bill), a second service you wrote from scratch, and all the debugging hours when the batching consumer stalls at 3 a.m. @@ -197,7 +197,7 @@ Tinybird wins on "zero ops to start." WaveHouse wins on "own your data plane and | Thundering-herd coalescing | ✗ | Custom | ✓ | ✓ Ristretto + singleflight | | Row/column policies with JWT claims | ✗ | Custom | Tokens only | ✓ Hasura-style | | Named parameterized pipes | ✗ | Custom | ✓ | ✓ stored in NATS KV | -| Type-safe client SDK with codegen | ✗ | Per team | Partial | ✓ `@wavehouse/sdk` | +| Type-safe client SDK with codegen | ✗ | Per team | Partial | ✓ TypeScript + Go SDKs | | Cost model | Infra only | Infra + eng time | Per-vCPU SaaS | Infra only | ## Part IV — End-to-end data journey diff --git a/scripts/cov/main.go b/scripts/cov/main.go index f4250bfa..2a3948bb 100644 --- a/scripts/cov/main.go +++ b/scripts/cov/main.go @@ -59,6 +59,30 @@ const ( // see the ts-total path below. var goSuites = []string{"unit", "integration", "e2e"} +// Go suites that produce covdata in the same layout as goSuites and are +// rendered + gated identically, but are deliberately NOT merged into the +// Go total: they come from a NESTED module (clients/go has its own +// go.mod). The root module cannot see a nested one — `go list ./...` at +// the repo root never yields clients/go — so the goSuites' -coverpkg=./... +// can't reach these files in the first place; they can only ever be in +// tmp/coverage/total if we put them there, which we don't. Folding a +// shipped client library into the server's project-wide number (and into +// the README badge `cov badge` derives from it) would move that number for +// reasons that have nothing to do with the server, so the SDK gets its own +// floor instead — the same separation the TS SDK gets via ts-*. +var standaloneGoSuites = []string{"go-sdk"} + +// suiteModuleDir maps a suite to the module directory its covdata was +// produced in, for suites that aren't the root module. `go tool cover +// -html` reads the source of every package named in the profile and +// resolves it through the module in the process's working directory, so a +// nested module's profile has to be rendered from inside that module — +// from the repo root the tool fails with "no required module provides +// package github.com/Wave-RF/WaveHouse/clients/go/...". `go tool covdata +// textfmt` has no such constraint (it only reads the covdata files), so +// only the HTML step needs the chdir. +var suiteModuleDir = map[string]string{"go-sdk": "clients/go"} + // TypeScript SDK suites (vitest). ts-unit comes from clients/ts; ts-e2e // from tests/e2e/sdk run with --coverage. Both produce Istanbul-format // coverage-final.json that `cov ts-merge` combines into ts-total. @@ -212,7 +236,7 @@ func goSuiteCoverage(c *config, suite string) (rows []pkgRow, total, covered int if err = sh("go", "tool", "covdata", "textfmt", "-i="+dataDir, "-o", profile); err != nil { return nil, 0, 0, "", err } - if err = sh("go", "tool", "cover", "-html="+profile, "-o", htmlOut); err != nil { + if err = renderHTML(suite, profile, htmlOut); err != nil { return nil, 0, 0, "", err } rows, total, covered, err = parseCoverage(profile, c, c.excludesFor(suite)) @@ -222,6 +246,27 @@ func goSuiteCoverage(c *config, suite string) (rows []pkgRow, total, covered int return rows, total, covered, htmlOut, nil } +// renderHTML turns a textfmt profile into the clickable HTML report. For a +// suite whose covdata came from a nested module (see suiteModuleDir) the +// tool runs with that module as its working directory — otherwise it can't +// resolve the profile's package paths to source and bails — so the profile +// and output paths are made absolute first. +func renderHTML(suite, profile, htmlOut string) error { + dir, nested := suiteModuleDir[suite] + if !nested { + return sh("go", "tool", "cover", "-html="+profile, "-o", htmlOut) + } + absProfile, err := filepath.Abs(profile) + if err != nil { + return err + } + absHTML, err := filepath.Abs(htmlOut) + if err != nil { + return err + } + return shIn(dir, "go", "tool", "cover", "-html="+absProfile, "-o", absHTML) +} + func renderSuite(c *config, suite string) error { rows, total, covered, htmlOut, err := goSuiteCoverage(c, suite) if err != nil { @@ -247,7 +292,7 @@ func renderSuite(c *config, suite string) error { // "one side legitimately absent" (skip, fine) from "nothing ran at all" // (fail, because the caller expected a gate). func hasAnyCoverage() bool { - for _, s := range goSuites { + for _, s := range slices.Concat(goSuites, standaloneGoSuites) { if hasCovdata(filepath.Join(root, s, "data")) { return true } @@ -298,6 +343,13 @@ func merge(c *config) error { for _, s := range goSuites { fmt.Printf(" %s%-13s%s %s\n", cyan, s+":", reset, suitePct(c, s)) } + // Nested-module Go suites: gated on their own, never merged above. + for _, s := range standaloneGoSuites { + if pct := suitePct(c, s); pct != "n/a" { + fmt.Printf(" %s%-13s%s %s %s(separate gate; not in merge above)%s\n", + cyan, s+":", reset, pct, yellow, reset) + } + } // Surface TS SDK coverage alongside the Go total — informational only, // not part of the Go merged number above. `make cov` is the gate. for _, s := range append(tsSuites, "ts-total") { @@ -551,6 +603,25 @@ func report(c *config) error { }) } + // --- Nested-module Go suites (own gate, below the Go total) --- + // Rendered and gated exactly like the suites above, but listed after + // the total they are deliberately not part of — see standaloneGoSuites. + for i, s := range standaloneGoSuites { + if !hasCovdata(filepath.Join(root, s, "data")) { + rows = append(rows, reportRow{name: s, pct: "n/a", rule: i == 0}) + continue + } + _, total, covered, html, err := goSuiteCoverage(c, s) + if err != nil { + return err + } + th := thresholdFor(c, s) + rows = append(rows, reportRow{ + name: s, pct: formatPctBare(covered, total), gated: true, thresh: th, + pass: meetsThreshold(covered, total, th), html: html, rule: i == 0, + }) + } + // --- TS suites + merged ts-total --- merged, err := mergeTSArtifacts("html", "json-summary") if err != nil { @@ -917,9 +988,15 @@ func meetsThreshold(covered, total, threshold int) bool { // sh runs an external command with stdio wired through. Every call site // passes "go" as the program and a fixed series of "tool", "", // flag, … args; the only variable bits are paths we computed ourselves. +func sh(name string, args ...string) error { return shIn("", name, args...) } + +// shIn is sh with an explicit working directory ("" = inherit ours) — for +// the one tool that cares which module it runs in, `go tool cover -html` +// on a nested module's profile. See renderHTML. // #nosec G204,G702 — name and args are not user input. -func sh(name string, args ...string) error { +func shIn(dir, name string, args ...string) error { cmd := exec.CommandContext(context.Background(), name, args...) + cmd.Dir = dir cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr return cmd.Run() diff --git a/tests/conformance/conformance_ts.mjs b/tests/conformance/conformance_ts.mjs new file mode 100644 index 00000000..b145e994 --- /dev/null +++ b/tests/conformance/conformance_ts.mjs @@ -0,0 +1,316 @@ +#!/usr/bin/env node +/** + * Cross-language wire-format conformance test for the TypeScript SDK. + * + * Reads wire_cases.json (owned by the Go module, at clients/go/testdata/) + * and verifies the TS SDK produces identical HTTP requests (method, path, + * content-type, body) to the shared fixture. + * + * Run: node tests/conformance/conformance_ts.mjs + * Exit 0 = all pass, exit 1 = failures. + */ + +import { readFileSync } from "node:fs"; +import { createServer } from "node:http"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); + +// Import the built SDK. +let createClient; +try { + ({ createClient } = await import(join(__dirname, "../../clients/ts/dist/index.js"))); +} catch (err) { + console.error( + "Cannot load the TypeScript SDK build. Run the SDK build first (e.g. `pnpm --dir clients/ts build`).", + ); + console.error(err.message); + process.exit(1); +} + +const cases = JSON.parse( + readFileSync(join(__dirname, "../../clients/go/testdata/wire_cases.json"), "utf-8"), +); + +let lastCapture = { method: "", path: "", contentType: "", body: "" }; + +function resetCapture() { + lastCapture = { method: "", path: "", contentType: "", body: "" }; +} + +// Start echo server. +const server = createServer((req, res) => { + const chunks = []; + req.on("data", (c) => chunks.push(c)); + req.on("end", () => { + lastCapture = { + method: req.method ?? "", + path: req.url ?? "", + contentType: req.headers["content-type"] ?? "", + body: Buffer.concat(chunks).toString("utf-8"), + }; + res.setHeader("Content-Type", "application/json"); + if (req.url?.startsWith("/v1/ops/dlq")) { + res.end(JSON.stringify({ tables: {}, total: 0 })); + } else if (req.url?.startsWith("/v1/ops/schema") && req.method === "GET") { + res.end(JSON.stringify([])); + } else if (req.url === "/v1/ops/policy/validate" && req.method === "POST") { + res.end(JSON.stringify({ valid: true })); + } else if (req.url?.startsWith("/v1/ops/policy") && req.method === "GET") { + res.end(JSON.stringify({ tables: {} })); + } else if (req.url?.startsWith("/v1/ops/pipes/") && req.method === "GET") { + res.end(JSON.stringify({ name: "test", sql: "SELECT 1" })); + } else if (req.url === "/v1/ops/pipes" && req.method === "GET") { + res.end(JSON.stringify([])); + } else if (req.url?.startsWith("/v1/ingest")) { + // Same shapes the real server returns (internal/api/ingest.go). + if (lastCapture.contentType === "application/x-ndjson") { + res.end(JSON.stringify({ total: 0, succeeded: 0, failed: 0, duplicates: 0 })); + } else { + res.end(JSON.stringify({ ok: true })); + } + } else if (req.url === "/v1/health") { + // Real server shape (internal/api/health.go). + res.end(JSON.stringify({ status: "ok" })); + } else { + res.end(JSON.stringify([])); + } + }); +}); + +await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); +const { port } = server.address(); +const baseURL = `http://127.0.0.1:${port}`; + +function applyQueryOps(wh, table, operations) { + let q = wh.from(table).select(); + for (const op of operations) { + switch (op.method) { + case "select": + q = wh.from(table).select(...op.args); + break; + case "selectAll": + q = q.selectAll(); + break; + case "where": + q = q.where(op.args[0], op.args[1], op.args[2]); + break; + case "count": + q = q.count(op.args[0] || "*", op.args[1] || "count"); + break; + case "sum": + q = q.sum(op.args[0], op.args[1] || undefined); + break; + case "avg": + q = q.avg(op.args[0], op.args[1] || undefined); + break; + case "min": + q = q.min(op.args[0], op.args[1] || undefined); + break; + case "max": + q = q.max(op.args[0], op.args[1] || undefined); + break; + case "countDistinct": + q = q.countDistinct(op.args[0], op.args[1] || undefined); + break; + case "aggregate": + q = q.aggregate(op.args[0], op.args[1], op.args[2]); + break; + case "groupBy": + q = q.groupBy(...op.args); + break; + case "orderBy": + q = q.orderBy(op.args[0], op.args[1] || "asc"); + break; + case "limit": + q = q.limit(op.args[0]); + break; + case "timeRange": + q = q.timeRange(op.args[0], op.args[1], op.args[2] || undefined); + break; + case "cacheTTL": + q = q.cacheTTL(op.args[0]); + break; + } + } + return q; +} + +// Compare request URIs by meaning: same path, same decoded query values, +// regardless of + vs %20 spelling or parameter order (mirrors the Go harness). +function normalizePath(p) { + let u; + try { + u = new URL(p, "http://conformance.invalid"); + } catch { + return p; + } + u.searchParams.sort(); + return `${u.pathname}?${u.searchParams.toString()}`; +} + +function deepEqual(a, b) { + return JSON.stringify(sortKeys(a)) === JSON.stringify(sortKeys(b)); +} + +function sortKeys(v) { + if (v === null || v === undefined) return v; + if (Array.isArray(v)) return v.map(sortKeys); + if (typeof v === "object") { + const sorted = {}; + for (const k of Object.keys(v).sort()) { + sorted[k] = sortKeys(v[k]); + } + return sorted; + } + return v; +} + +let passed = 0; +let failed = 0; +let skipped = 0; +const skippedNames = []; +const failures = []; + +for (const tc of cases) { + resetCapture(); + const wh = createClient({ baseURL, options: { maxRetries: 0 } }); + + try { + switch (tc.endpoint) { + case "query": { + const q = applyQueryOps(wh, tc.table, tc.operations ?? []); + await q.fetch(); + break; + } + case "ingest": + case "ingest_batch": + if (tc.operations?.[0]?.method !== "insert") { + // Hard failure, matching the Go harness. + throw new Error(`${tc.name}: ingest case has no insert operation`); + } + await wh.from(tc.table).insert(tc.operations[0].args[0]); + break; + case "pipe": + await wh.pipe(tc.pipe_name, tc.pipe_params ?? undefined).fetch(); + break; + case "sql": + await wh.sql(tc.sql); + break; + case "health": + await wh.sys.health(); + break; + case "schema_list": + await wh.schema.list(); + break; + case "schema_refresh": + await wh.schema.refresh(); + break; + case "policy_get": + await wh.policy.get(); + break; + case "policy_set": + await wh.policy.set(tc.policy_body); + break; + case "policy_validate": + await wh.policy.validate(tc.policy_body); + break; + case "dlq_list": + await wh.dlq.list(); + break; + case "dlq_table": + await wh.dlq.table(tc.table); + break; + case "pipes_list": + await wh.pipes.list(); + break; + case "pipes_get": + await wh.pipes.get(tc.pipe_name); + break; + case "pipes_set": + await wh.pipes.set(tc.pipe_name, tc.pipe_def); + break; + case "pipes_delete": + await wh.pipes.delete(tc.pipe_name); + break; + default: + // Not a pass — the Go harness hard-fails on these; we count and exit + // non-zero below. Fixture cases with a new endpoint value must be + // wired up here before they count. + skipped++; + skippedNames.push(`${tc.name} (endpoint: ${tc.endpoint})`); + continue; + } + + const errs = []; + + if (tc.expected_method && lastCapture.method !== tc.expected_method) { + errs.push(`method: want ${tc.expected_method}, got ${lastCapture.method}`); + } + + if (tc.expected_path && normalizePath(lastCapture.path) !== normalizePath(tc.expected_path)) { + errs.push(`path: want ${tc.expected_path}, got ${lastCapture.path}`); + } + + if (tc.expected_content_type && lastCapture.contentType !== tc.expected_content_type) { + errs.push(`content-type: want ${tc.expected_content_type}, got ${lastCapture.contentType}`); + } + + if (tc.expected_raw_body !== undefined) { + if (lastCapture.body !== tc.expected_raw_body) { + errs.push(`raw body:\n want: ${tc.expected_raw_body}\n got: ${lastCapture.body}`); + } + } else if (tc.expected_body !== undefined && tc.expected_body !== null) { + let captured; + try { + captured = JSON.parse(lastCapture.body); + } catch { + errs.push(`body not valid JSON: ${lastCapture.body}`); + } + if (captured !== undefined && !deepEqual(captured, tc.expected_body)) { + errs.push( + `body mismatch:\n want: ${JSON.stringify(tc.expected_body)}\n got: ${JSON.stringify(captured)}`, + ); + } + } + + if (errs.length > 0) { + failed++; + failures.push({ name: tc.name, errors: errs }); + } else { + passed++; + } + } catch (err) { + failed++; + failures.push({ name: tc.name, errors: [`exception: ${err.message}`] }); + } +} + +server.closeAllConnections?.(); +server.close(); + +console.log( + `\nWire-format conformance (TS SDK): ${passed} passed, ${failed} failed, ${skipped} skipped, ${cases.length} total\n`, +); + +for (const name of skippedNames) { + console.log(` - skipped: ${name}`); +} + +for (const f of failures) { + console.log(` ✗ ${f.name}`); + for (const e of f.errors) { + console.log(` ${e}`); + } +} + +if (failed > 0 || skipped > 0 || passed === 0) { + if (passed === 0) console.log(" ✗ nothing ran — every case skipped or the fixture is empty\n"); + if (skipped > 0) + console.log(" ✗ skipped cases break cross-SDK parity — wire up the endpoint above\n"); + process.exit(1); +} else { + console.log(" ✓ All cases passed\n"); + process.exit(0); +}