diff --git a/.claude/skills/release/SKILL.md b/.claude/skills/release/SKILL.md new file mode 100644 index 000000000..0e85b4e48 --- /dev/null +++ b/.claude/skills/release/SKILL.md @@ -0,0 +1,306 @@ +--- +name: release +description: > + Drive a package release of `ClickHouse/clickhouse-js` end to end: bump the + version, sync the protected `release` branch from `main`, watch the npm + publish (which is gated by a manual approval on the `npm-publish` environment), + and create the GitHub Release from the CHANGELOG. Use this skill whenever the + task is to "release", "cut a release", "publish a new version", or "ship" one + of the packages: `@clickhouse/client` (Node.js), `@clickhouse/client-web` + (Web), `@clickhouse/client-common` (deprecated, rarely released), the + standalone `@clickhouse/datatype-parser` (the type parser, under + `packages/datatype-parser`), or `@clickhouse/rowbinary` (the RowBinary codec + skill/package, under `skills/clickhouse-js-node-rowbinary`). The agent + drives the GitHub Actions workflows (`gh workflow run`), watches CI, pauses at + the human-judgment points (PR review, the approval gate, GitHub Release text), + and hands the deployment-approval link back to the human. Do NOT use this for + fixing a failing release PR — that is the `fix-release-pr` skill. +--- + +# Releasing a `clickhouse-js` package + +This skill is the source of truth for the release process. It supersedes the old +`RELEASING.md` (which now just points here). + +## Before you start — read this + +1. **Releases are per package.** There is no "release everything" button. Ask the + user **which package(s)** they want to release and confirm — the packages are + versioned independently and ship on independent cadences: + - `@clickhouse/client` — Node.js client (`packages/client-node`) + - `@clickhouse/client-web` — Web client (`packages/client-web`) + - `@clickhouse/client-common` — **deprecated**, effectively frozen; only cut a + final standalone release if explicitly asked. + - `@clickhouse/datatype-parser` — standalone type parser (`packages/datatype-parser`) + - `@clickhouse/rowbinary` — standalone RowBinary codec skill/package + (`skills/clickhouse-js-node-rowbinary`) + + The flow forks into two families — **workspace client packages** (client / + client-web / client-common) and **standalone packages** (datatype-parser / + rowbinary) — which differ in version bumping and publish workflow. Pick the + right section below. + +2. **`release` is a long-lived, protected branch.** It is _not_ a per-release + branch. The `npm-publish` GitHub Actions environment only permits the + `release` branch to deploy, and it additionally **requires a manual approval** + before any publish job runs. You will hand the human an approval link and wait. + +3. **You drive, the human judges.** Run the workflows with `gh`, watch CI, and + pause at: (a) reviewing the release PR, (b) the `npm-publish` approval gate, + (c) the GitHub Release text. Never push to `release` directly — if the release + PR needs fixes, route them through `main` (see the `fix-release-pr` skill). + +> **One workflow per package.** Each package has its own publish workflow: +> +> - `@clickhouse/client` → `publish-client.yml` +> - `@clickhouse/client-web` → `publish-client-web.yml` +> - `@clickhouse/client-common` → `publish-client-common.yml` +> - `@clickhouse/datatype-parser` → `publish-datatype-parser.yml` +> - `@clickhouse/rowbinary` → `publish-skill-rowbinary.yml` +> +> Each client workflow has two triggers: an automatic `head` publish on push to +> `release` and a manual `latest` publish via `workflow_dispatch`. The standalone +> packages have the manual trigger only. +> +> The client `head` triggers are path-scoped: a change touching only one client's +> own sources no longer republishes the others. Both `@clickhouse/client` and +> `@clickhouse/client-web` bundle the shared common sources via the `src/common` +> symlink (`packages/*/src/common` → `packages/client-common/src`), so +> `packages/client-common/**` is also an input to both client workflows — a +> change to the common sources publishes a new `head` for every client that +> bundles them. + +--- + +## Part A — Workspace client packages (`@clickhouse/client`, `-web`, `-common`) + +### Step 1 — Verify the package CHANGELOG on `main` + +Each package now keeps its **own** `CHANGELOG.md` (the repo-wide root +`CHANGELOG.md` is frozen). The package you're releasing maps to: + +- `@clickhouse/client` → `packages/client-node/CHANGELOG.md` +- `@clickhouse/client-web` → `packages/client-web/CHANGELOG.md` +- `@clickhouse/client-common` → `packages/client-common/CHANGELOG.md` + +These are normally updated **inside feature PRs** as they merge to `main`, under +a top `# ` header. A common mistake: the in-progress entries sit under +the wrong header — e.g. still under the **previously released** version, or under +a version that doesn't match the one you're about to cut. (Note: shared/common +code is bundled into both clients, so such changes should appear in both the +client-node and client-web changelogs.) + +- Compute the version this release will produce: current version + the + `bump_type` you'll pick. Current version: + ```bash + node -p "require('./packages/client-node/package.json').version" # or client-web / client-common + ``` +- Confirm that package's `CHANGELOG.md` has a top `# ` section that + actually contains this release's notes. +- If it's wrong, **fix it in a small separate PR to `main` and merge it quickly** + before bumping — do not bundle the changelog fix into the bump PR. + +### Step 2 — Bump the version (opens a PR to `main`) + +Dispatch the `bump-version` workflow. It bumps the selected package's +`package.json` + `src/version.ts` and opens a `release--` PR **against +`main`**: + +```bash +gh workflow run bump-version.yml --ref main \ + -f package='@clickhouse/client' \ + -f bump_type=patch # patch | minor | major +``` + +Find and watch the resulting PR, get it reviewed, and **merge it into `main`**: + +```bash +gh run list --workflow=bump-version.yml --limit 1 +gh pr list --search 'head:release-client-' --state open # locate the bump PR +``` + +### Step 3 — Sync `release` from `main` (the "release PR") + +Open a **real PR with base `release`, head `main`**. This PR is the snapshot +that gets reviewed once more _as a whole_ (code-review tooling runs on the full +diff): + +```bash +gh pr create --base release --head main \ + --title " release" \ + --body "Sync release from main to cut ." +``` + +- If review surfaces issues, **do not push to `release`**. Fix on `main` via a + separate PR (use the `fix-release-pr` skill), then the release PR picks the fix + up automatically. Keep `release` and `main` from drifting. +- When it's clean, **merge the release PR**. + +### Step 4 — Watch the automatic `head` publish + e2e + +Merging into `release` pushes to `release`, which triggers the **path-scoped** +`head` publish for each package whose sources changed (`publish-client.yml`, +`publish-client-web.yml`, `publish-client-common.yml`). It publishes a `head` +pre-release build (an **internal beta — not tracked** as a GitHub Release); the +client workflows then run an automatic `e2e` job against the published version +and a live ClickHouse — the Node.js client installs it across Node 20/22/24/26 +and runs integration tests, and the Web client runs a real-browser smoke +(chromium + firefox) via Playwright. + +- These publishes run under the `npm-publish` environment → **they pause for + manual approval**. Find the run(s) — one per package that changed — give the + human the approval link, and wait: + ```bash + gh run list --workflow=publish-client.yml --branch release --limit 3 + gh run view --json url -q .url # hand this URL to the human to approve + gh run watch # waits for completion + ``` + (Use `publish-client-web.yml` / `publish-client-common.yml` for the other + packages.) +- The `e2e` job in `publish-client.yml` / `publish-client-web.yml` is sufficient + verification — **no manual `@head` testing is needed**. Just watch it. +- **If e2e fails:** the broken build is already on npm under the `head` tag. + Suggest moving that tag out of the way so nobody installs it, e.g.: + ```bash + npm dist-tag add @clickhouse/client@ debugging + ``` + Then fix forward on `main` and re-sync `release`. + +### Step 5 — Publish to `latest` (manual dispatch, per package) + +Once the `head` e2e is green, dispatch the package's publish workflow **manually** +from the `release` branch (no inputs). This builds, publishes to the `latest` tag +(npm OIDC + provenance), and pushes a git tag (`client-`, `client-web-`, +or `client-common-`): + +```bash +gh workflow run publish-client.yml --ref release # @clickhouse/client +# gh workflow run publish-client-web.yml --ref release # @clickhouse/client-web +# gh workflow run publish-client-common.yml --ref release # @clickhouse/client-common (deprecated) +``` + +This also runs under `npm-publish` → **approval gate again**. Same drill: hand +over the approval URL, then watch: + +```bash +gh run list --workflow=publish-client.yml --branch release --event workflow_dispatch --limit 1 +gh run view --json url -q .url +gh run watch +``` + +Repeat for each package being released, dispatching its own workflow. + +For the deprecated `@clickhouse/client-common`, if you ever cut one, also confirm +its npm deprecation notice is in place (it should already be). + +### Step 6 — Create the GitHub Release + +The auto-published `head` betas are **not** tracked as Releases. The `latest` +release **is**. Create a GitHub Release from the matching section of **that +package's** `CHANGELOG.md` (e.g. `packages/client-node/CHANGELOG.md` for +`@clickhouse/client`). + +- The tag was pushed by the publish workflow: `client-` / `client-web-` + / `client-common-` (note: client tags have **no** `v` prefix). +- Use the CHANGELOG section that documents this release as the body. Extract it + (the lines under `# ` up to the next `# ` header): + ```bash + awk -v v="1.23.0" '$0=="# "v{f=1;next} /^# /{f=0} f' \ + packages/client-node/CHANGELOG.md > /tmp/notes.md + ``` +- **Do not pass `--title`** — GitHub renders the title from the tag itself. These + are not pre-releases, so do not pass `--prerelease`: + ```bash + gh release create client-1.23.0 --notes-file /tmp/notes.md + ``` +- **Backfill check:** sometimes earlier releases are missing their GitHub Release. + Compare pushed tags against existing releases and create any that are missing, + pulling each one's notes from the matching CHANGELOG section: + ```bash + gh release list --limit 50 + git ls-remote --tags origin | grep -oE 'client(-web|-common)?-[0-9.]+' + ``` + +--- + +## Part B — Standalone packages (`@clickhouse/datatype-parser`, `@clickhouse/rowbinary`) + +These ship independently. **There is no `bump-version` workflow and no `head` +beta** for them. + +### Step 1 — Bump the version manually on `main` + +Edit the version in the package's `package.json` (these have **no** `src/version.ts`): + +- `@clickhouse/datatype-parser` → `packages/datatype-parser/package.json` +- `@clickhouse/rowbinary` → `skills/clickhouse-js-node-rowbinary/package.json` + +Do this in a normal PR to `main`, together with the relevant entry in **that +package's own** `CHANGELOG.md` (verify the changelog as in Part A, Step 1): + +- `@clickhouse/datatype-parser` → `packages/datatype-parser/CHANGELOG.md` +- `@clickhouse/rowbinary` → `skills/clickhouse-js-node-rowbinary/CHANGELOG.md` + +Merge it. + +### Step 2 — Sync `release` from `main` + +Same as Part A, Step 3: open a `base release ← head main` PR, get it reviewed as +a whole, merge it. (Pushing to `release` does **not** auto-publish these +standalone packages — only the client packages have an auto `head` publish.) + +### Step 3 — Publish to `latest` (manual dispatch, no inputs) + +Dispatch the package's own publish workflow from the `release` branch. These take +**no inputs**; the `release`-branch ref is required by the workflow's `if` guard: + +```bash +# type parser: +gh workflow run publish-datatype-parser.yml --ref release +# rowbinary codec skill/package: +gh workflow run publish-skill-rowbinary.yml --ref release +``` + +Each workflow builds, packs + smoke-tests the tarball, publishes it to `latest` +(OIDC + provenance), pushes a git tag (`datatype-parser-v` / +`rowbinary-v` — note the **`v`** prefix here), then runs the `e2e` job +across Node 20/22/24/26. + +Same `npm-publish` **approval gate** applies — hand over the approval URL and +watch: + +```bash +gh run list --workflow=publish-datatype-parser.yml --branch release --limit 1 +gh run view --json url -q .url +gh run watch +``` + +### Step 4 — Create the GitHub Release + +Same as Part A, Step 6, using the standalone tag (`datatype-parser-v` / +`rowbinary-v`) and the notes from that package's own `CHANGELOG.md` +(`packages/datatype-parser/CHANGELOG.md` or +`skills/clickhouse-js-node-rowbinary/CHANGELOG.md`). No `--title`, no +`--prerelease`. Backfill any missing releases. + +--- + +## Quick reference + +**Client packages:** + +1. Verify/fix the top section of the package's own `CHANGELOG.md` (e.g. `packages/client-node/CHANGELOG.md`) on `main` (quick PR if wrong). +2. `gh workflow run bump-version.yml --ref main -f package=… -f bump_type=…` → review & merge the bump PR to `main`. +3. `gh pr create --base release --head main` → review the whole release PR → merge. +4. Auto `head` publish (per-package `publish-client*.yml`, path-scoped) + e2e on push to `release` → **approve** at `npm-publish`, watch e2e. (If e2e fails: retag the bad build to `debugging`, fix forward.) +5. `gh workflow run publish-client.yml --ref release` (or `-web` / `-common`) per package → **approve**, watch. +6. `gh release create --notes-file ` (no `--title`, not prerelease); backfill missing releases. + +**Standalone packages (datatype-parser / rowbinary):** + +1. Bump version in the package's `package.json` + its own `CHANGELOG.md` via a PR to `main` → merge. +2. `gh pr create --base release --head main` → review → merge. +3. `gh workflow run publish-datatype-parser.yml --ref release` (or `publish-skill-rowbinary.yml`) → **approve**, watch. +4. `gh release create <…-v…-tag> --notes-file `; backfill. + +**Always:** per package · each package has its **own** `CHANGELOG.md` (root is frozen) · `release` is protected (fix via `main`) · every publish needs `npm-publish` approval · `head` betas are untracked · GitHub Release notes come from the package's `CHANGELOG.md` with no explicit `--title`. diff --git a/.claude/skills/setup/SKILL.md b/.claude/skills/setup/SKILL.md index 8063ae094..c0aaf064f 100644 --- a/.claude/skills/setup/SKILL.md +++ b/.claude/skills/setup/SKILL.md @@ -20,7 +20,7 @@ Use this skill before running any of the `npm run test:*`, `npm run lint`, `npm ## Prerequisites -- **Node.js 22 recommended** (matches `.nvmrc`). The root `package.json` declares `"engines": { "node": ">=20.19.0" }`, and CI tests Node 20, 22, and 24. +- **Node.js 22 recommended** (matches `.nvmrc`). The root `package.json` declares `"engines": { "node": ">=20.19.0" }`, and CI tests Node 20, 22, 24, and 26. - **Docker** with the Compose plugin (`docker compose ...`). Required only for integration tests and any example that talks to a real server. ## 1. Install dependencies diff --git a/.github/instructions/review.instructions.md b/.github/instructions/review.instructions.md index 84aa789bf..4001ee673 100644 --- a/.github/instructions/review.instructions.md +++ b/.github/instructions/review.instructions.md @@ -19,7 +19,7 @@ When reviewing PRs to the `release` branch focus on flagging breaking changes an ## CHANGELOG -Every PR that adds a feature, fixes a bug, or changes observable behavior or the public API **must update `CHANGELOG.md` in the same PR**. If such a change does not touch `CHANGELOG.md`, flag it and ask the author to add an entry (a note in the PR description alone is not sufficient). Verify the entry follows the conventions in [`AGENTS.md`](../../AGENTS.md) ("API quality and stability"): placed under the top-most version heading (a new `# x.y.z` heading matching the unreleased `package.json` version when the latest heading is already released), grouped under a lowercase section heading (`## New features` / `## Improvements` / `## Bug fixes`), and ending with a `([#])` PR reference link. Pure refactors, test-only changes, docs, and CI/tooling changes do not require a CHANGELOG entry. +Every PR that adds a feature, fixes a bug, or changes observable behavior or the public API **must update the changelog of every affected package in the same PR**. Each package keeps its own `CHANGELOG.md` (the repository-wide root `CHANGELOG.md` is **frozen** — new entries must not go there); see the package → changelog mapping in [`AGENTS.md`](../../AGENTS.md) ("API quality and stability"). A change to shared/common code bundled into both clients affects `@clickhouse/client` and `@clickhouse/client-web`, so expect both of their changelogs to be updated. If such a change does not touch the relevant package changelog(s), flag it and ask the author to add an entry (a note in the PR description alone is not sufficient). Verify the entry follows the `AGENTS.md` conventions: placed under the top-most version heading (a new `# x.y.z` heading matching that package's unreleased `package.json` version when the latest heading is already released), grouped under a lowercase section heading (`## New features` / `## Improvements` / `## Bug fixes`), and ending with a `([#])` PR reference link. Pure refactors, test-only changes, docs, and CI/tooling changes do not require a changelog entry. ## Breaking changes diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index f5b7d0efd..18b59e672 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -7,5 +7,5 @@ A short description of the changes with a link to an open issue. Delete items not relevant to your PR: - [ ] Unit and integration tests covering the common scenarios were added -- [ ] A human-readable description of the changes was provided to include in CHANGELOG +- [ ] A human-readable changelog entry was added to every affected package's `CHANGELOG.md` (e.g. `packages/client-node/CHANGELOG.md`; a shared/common change updates both client packages — `client-node` and `client-web`; the root `CHANGELOG.md` is frozen) - [ ] For significant changes, documentation in https://github.com/ClickHouse/clickhouse-docs was updated with further explanations or tutorials diff --git a/.github/workflows/e2e-skills.yml b/.github/workflows/e2e-skills.yml index ea0a22579..5725ec643 100644 --- a/.github/workflows/e2e-skills.yml +++ b/.github/workflows/e2e-skills.yml @@ -18,6 +18,14 @@ on: branches: - release +env: + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" + jobs: skills-packaging: runs-on: ubuntu-latest diff --git a/.github/workflows/examples.yml b/.github/workflows/examples.yml index a1d7a09c5..22757f7b1 100644 --- a/.github/workflows/examples.yml +++ b/.github/workflows/examples.yml @@ -50,6 +50,12 @@ env: OTEL_RESOURCE_ATTRIBUTES: "service.namespace=clickhouse-js,deployment.environment=ci" VITEST_OTEL_ENABLED: "true" VITEST_COVERAGE: "true" + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" jobs: code-quality: diff --git a/.github/workflows/publish-client-common.yml b/.github/workflows/publish-client-common.yml new file mode 100644 index 000000000..10e23a61b --- /dev/null +++ b/.github/workflows/publish-client-common.yml @@ -0,0 +1,135 @@ +name: "publish: client-common" + +# Publish + release for @clickhouse/client-common (DEPRECATED). +# +# This package is deprecated and effectively frozen: @clickhouse/client and +# @clickhouse/client-web no longer depend on the published package — they build +# the common sources straight from packages/client-common/src via their +# src/common symlinks. It can still be cut a final standalone release. +# +# Split out of the former all-in-one publish.yml so each client package +# publishes on its own. Two triggers, like the standalone-package workflows: +# - push to `release` -> publishes the "head" pre-release tag (internal beta). +# - manual workflow_dispatch -> publishes the "latest" tag and pushes the +# release git tag. The `main` branch never publishes. +# Both use npm OIDC authentication with provenance, and both deploy through the +# protected `npm-publish` environment (release branch only + manual approval). +# +# The push trigger is path-scoped to this package's own sources plus the +# repo-root files its published tarball depends on. + +permissions: + contents: read + id-token: write # Required for npm OIDC authentication and provenance + +concurrency: + # event_name separates the head (push) and latest (workflow_dispatch) runs so + # a manual publish never cancels an in-progress head publish (or vice versa). + group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }} + cancel-in-progress: true + +on: + # for the latest workflow + workflow_dispatch: + # for the head workflow + push: + branches: + - release + paths: + - "packages/client-common/**" + - "package.json" + - "package-lock.json" + - "tsconfig.base.json" + - "README.md" + - "LICENSE" + - ".github/workflows/publish-client-common.yml" + +env: + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" + +jobs: + head: + name: "Publish @clickhouse/client-common (head)" + if: github.ref == 'refs/heads/release' && github.event_name == 'push' + runs-on: ubuntu-latest + environment: npm-publish + steps: + - name: Checkout repository + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + + - name: Setup Node.js + uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 24 + registry-url: "https://registry.npmjs.org" + + - name: Install dependencies + run: npm ci + + - name: Set head pre-release version + run: | + BASE_VERSION=$(node -p "require('./packages/client-common/package.json').version") + HEAD_VERSION="${BASE_VERSION}-head.${GITHUB_SHA::7}.${GITHUB_RUN_ATTEMPT}" + echo "Setting version to: $HEAD_VERSION" + npm --workspace @clickhouse/client-common version --no-git-tag-version "$HEAD_VERSION" + echo "export default \"$HEAD_VERSION\";" > packages/client-common/src/version.ts + + - name: Build the package + run: npm --workspace @clickhouse/client-common run build + + - name: Publish @clickhouse/client-common with head tag + run: | + npm --workspace @clickhouse/client-common publish \ + --access public \ + --provenance \ + --tag head + + publish: + name: "Publish @clickhouse/client-common (deprecated)" + if: github.ref == 'refs/heads/release' && github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + environment: npm-publish + permissions: + contents: write # Required to push the release git tag + id-token: write # Required for npm OIDC authentication and provenance + outputs: + version: ${{ steps.version.outputs.version }} + packages: "@clickhouse/client-common" + steps: + - name: Checkout repository + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + + - name: Setup Node.js + uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 24 + registry-url: "https://registry.npmjs.org" + + - name: Install dependencies + run: npm ci + + - name: Get the release version + id: version + run: | + BASE_VERSION=$(node -p "require('./packages/client-common/package.json').version") + echo "Using version: $BASE_VERSION" + echo "version=$BASE_VERSION" >> "$GITHUB_OUTPUT" + + - name: Build the package + run: npm --workspace @clickhouse/client-common run build + + - name: Publish @clickhouse/client-common to the latest tag (implicit) + run: | + npm --workspace @clickhouse/client-common publish \ + --access public \ + --provenance + + - name: Create and push release git tag + env: + RELEASE_VERSION: ${{ steps.version.outputs.version }} + run: .scripts/push_release_tag.sh "client-common-${RELEASE_VERSION}" diff --git a/.github/workflows/publish-client-web.yml b/.github/workflows/publish-client-web.yml new file mode 100644 index 000000000..f3a017383 --- /dev/null +++ b/.github/workflows/publish-client-web.yml @@ -0,0 +1,225 @@ +name: "publish: client-web" + +# Publish + release for @clickhouse/client-web (the Web client). +# +# Split out of the former all-in-one publish.yml so each client package +# publishes on its own. Two triggers, like the standalone-package workflows: +# - push to `release` -> publishes the "head" pre-release tag (internal beta). +# - manual workflow_dispatch -> publishes the "latest" tag and pushes the +# release git tag. The `main` branch never publishes. +# Both use npm OIDC authentication with provenance, and both deploy through the +# protected `npm-publish` environment (release branch only + manual approval). +# +# The push trigger is path-scoped to this package's own sources, the shared +# common sources it bundles via the `src/common` symlink +# (packages/client-common/src), and the repo-root files its published tarball +# depends on. So a change to only one client no longer republishes the others, +# but a change to the shared common sources publishes a new head build for +# every client that bundles them. + +permissions: + contents: read + id-token: write # Required for npm OIDC authentication and provenance + +concurrency: + # event_name separates the head (push) and latest (workflow_dispatch) runs so + # a manual publish never cancels an in-progress head publish (or vice versa). + group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }} + cancel-in-progress: true + +on: + # for the latest workflow + workflow_dispatch: + # for the head workflow + push: + branches: + - release + paths: + - "packages/client-web/**" + # The Web client bundles the common sources via the src/common symlink. + - "packages/client-common/**" + - "package.json" + - "package-lock.json" + - "tsconfig.base.json" + - "README.md" + - "LICENSE" + - ".github/workflows/publish-client-web.yml" + +env: + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" + +jobs: + head: + name: "Publish @clickhouse/client-web (head)" + if: github.ref == 'refs/heads/release' && github.event_name == 'push' + runs-on: ubuntu-latest + environment: npm-publish + outputs: + version: ${{ steps.version.outputs.version }} + packages: "@clickhouse/client-web" + steps: + - name: Checkout repository + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + + - name: Setup Node.js + uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 24 + registry-url: "https://registry.npmjs.org" + + - name: Install dependencies + run: npm ci + + - name: Set head pre-release version + id: version + run: | + BASE_VERSION=$(node -p "require('./packages/client-web/package.json').version") + HEAD_VERSION="${BASE_VERSION}-head.${GITHUB_SHA::7}.${GITHUB_RUN_ATTEMPT}" + echo "Setting version to: $HEAD_VERSION" + npm --workspace @clickhouse/client-web version --no-git-tag-version "$HEAD_VERSION" + echo "export default \"$HEAD_VERSION\";" > packages/client-web/src/version.ts + echo "version=$HEAD_VERSION" >> "$GITHUB_OUTPUT" + + - name: Build the package + run: npm --workspace @clickhouse/client-web run build + + - name: Publish @clickhouse/client-web with head tag + run: | + npm --workspace @clickhouse/client-web publish \ + --access public \ + --provenance \ + --tag head + + publish: + name: "Publish @clickhouse/client-web" + if: github.ref == 'refs/heads/release' && github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + environment: npm-publish + permissions: + contents: write # Required to push the release git tag + id-token: write # Required for npm OIDC authentication and provenance + outputs: + version: ${{ steps.version.outputs.version }} + packages: "@clickhouse/client-web" + steps: + - name: Checkout repository + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + + - name: Setup Node.js + uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 24 + registry-url: "https://registry.npmjs.org" + + - name: Install dependencies + run: npm ci + + - name: Get the release version + id: version + run: | + BASE_VERSION=$(node -p "require('./packages/client-web/package.json').version") + echo "Using version: $BASE_VERSION" + echo "version=$BASE_VERSION" >> "$GITHUB_OUTPUT" + + - name: Build the package + run: npm --workspace @clickhouse/client-web run build + + - name: Publish @clickhouse/client-web to the latest tag (implicit) + run: | + npm --workspace @clickhouse/client-web publish \ + --access public \ + --provenance + + - name: Create and push release git tag + env: + RELEASE_VERSION: ${{ steps.version.outputs.version }} + run: .scripts/push_release_tag.sh "client-web-${RELEASE_VERSION}" + + e2e: + name: e2e (browser ${{ matrix.browser }}) + needs: [head, publish] + # Runs whether the published version came from the head job or the manual + # latest publish. + if: | + always() && + (needs.head.result == 'success' || needs.publish.result == 'success') + runs-on: ubuntu-latest + strategy: + fail-fast: true + matrix: + # Matches the browsers exercised by the main web suite (WebKit is not + # tested at the moment). + browser: [chromium, firefox] + defaults: + run: + working-directory: tests/e2e/web-browser + env: + PUBLISHED_VERSION: ${{ needs.head.outputs.version || needs.publish.outputs.version }} + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + + # The single-node service sends CORS headers, so the browser can reach it + # cross-origin from the vitest page (see its config.xml). + - name: Start ClickHouse (stable) in Docker + uses: hoverkraft-tech/compose-action@11beaa1c2dae4e8ed7b1665aa074723b6cecb0e4 # v3.0.0 + env: + CLICKHOUSE_VERSION: latest + with: + compose-file: "docker-compose.yml" + down-flags: "--volumes" + + - name: Setup NodeJS + uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 24 + registry-url: "https://registry.npmjs.org" + + - name: Wait for ${{ env.PUBLISHED_VERSION }} to be available on npm + run: | + set -euo pipefail + if [ -z "${PUBLISHED_VERSION}" ]; then + echo "PUBLISHED_VERSION is empty; cannot wait for npm publication." >&2 + exit 1 + fi + pkg="@clickhouse/client-web" + # Poll the registry for up to ~5 minutes. New versions usually surface + # in seconds, but the registry CDN can lag. + max_attempts=60 + sleep_seconds=5 + attempt=1 + echo "Waiting for ${pkg}@${PUBLISHED_VERSION} to be available on npm..." + while true; do + if npm view "${pkg}@${PUBLISHED_VERSION}" version >/dev/null 2>&1; then + echo " ${pkg}@${PUBLISHED_VERSION} is available." + break + fi + if [ "$attempt" -ge "$max_attempts" ]; then + echo "Timed out waiting for ${pkg}@${PUBLISHED_VERSION} on npm" >&2 + exit 1 + fi + echo " attempt ${attempt}/${max_attempts}: not available yet, sleeping ${sleep_seconds}s..." + attempt=$((attempt + 1)) + sleep "$sleep_seconds" + done + + - name: Install dependencies + run: npm install + + - name: Install the Playwright browser + run: npx playwright install ${{ matrix.browser }} + + - name: Install @clickhouse/client-web at the published version + run: npm install "@clickhouse/client-web@${PUBLISHED_VERSION}" + + # Real-browser e2e: vitest's Vite bundler serves the installed package to a + # Playwright-driven browser and runs it against the live ClickHouse started + # above (import, ping, query, streaming, ClickHouseError on a bad query). + - name: Run the Web browser e2e against the published package + env: + BROWSER: ${{ matrix.browser }} + run: npm test diff --git a/.github/workflows/publish-client.yml b/.github/workflows/publish-client.yml new file mode 100644 index 000000000..9e8b88cd2 --- /dev/null +++ b/.github/workflows/publish-client.yml @@ -0,0 +1,252 @@ +name: "publish: client" + +# Publish + release for @clickhouse/client (the Node.js client). +# +# Split out of the former all-in-one publish.yml so each client package +# publishes on its own. Two triggers, like the standalone-package workflows: +# - push to `release` -> publishes the "head" pre-release tag (internal beta). +# - manual workflow_dispatch -> publishes the "latest" tag and pushes the +# release git tag. The `main` branch never publishes. +# Both use npm OIDC authentication with provenance, and both deploy through the +# protected `npm-publish` environment (release branch only + manual approval). +# +# The push trigger is path-scoped to this package's own sources, the shared +# common sources it bundles via the `src/common` symlink +# (packages/client-common/src), and the repo-root files its published tarball +# depends on (README/LICENSE, the bundled skills/, and the shared +# lockfile/tsconfig). So a change to only one client no longer republishes the +# others, but a change to the shared common sources publishes a new head build +# for every client that bundles them. +# +# Before publishing, a smoke test packs the freshly built package into a +# tarball, installs it into a throwaway app, and runs ESM + CJS checks against +# it (.scripts/smoke_test_pack.sh). After a successful publish, the `e2e` job +# waits for the version to surface on the npm registry, installs that exact +# version into a tiny downstream project, and verifies it is usable. + +permissions: + contents: read + id-token: write # Required for npm OIDC authentication and provenance + +concurrency: + # event_name separates the head (push) and latest (workflow_dispatch) runs so + # a manual publish never cancels an in-progress head publish (or vice versa). + group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }} + cancel-in-progress: true + +on: + # for the latest workflow + workflow_dispatch: + # for the head workflow + push: + branches: + - release + paths: + - "packages/client-node/**" + # The Node client bundles the common sources via the src/common symlink. + - "packages/client-common/**" + - "skills/**" + - "package.json" + - "package-lock.json" + - "tsconfig.base.json" + - "README.md" + - "LICENSE" + - ".github/workflows/publish-client.yml" + +env: + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" + +jobs: + head: + name: "Publish @clickhouse/client (head)" + if: github.ref == 'refs/heads/release' && github.event_name == 'push' + runs-on: ubuntu-latest + environment: npm-publish + outputs: + version: ${{ steps.version.outputs.version }} + packages: "@clickhouse/client" + steps: + - name: Checkout repository + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + + - name: Setup Node.js + uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 24 + registry-url: "https://registry.npmjs.org" + + - name: Install dependencies + run: npm ci + + - name: Set head pre-release version + id: version + run: | + BASE_VERSION=$(node -p "require('./packages/client-node/package.json').version") + HEAD_VERSION="${BASE_VERSION}-head.${GITHUB_SHA::7}.${GITHUB_RUN_ATTEMPT}" + echo "Setting version to: $HEAD_VERSION" + npm --workspace @clickhouse/client version --no-git-tag-version "$HEAD_VERSION" + echo "export default \"$HEAD_VERSION\";" > packages/client-node/src/version.ts + echo "version=$HEAD_VERSION" >> "$GITHUB_OUTPUT" + + - name: Build the package + run: npm --workspace @clickhouse/client run build + + - name: Smoke test the packed tarball + run: .scripts/smoke_test_pack.sh @clickhouse/client + + - name: Publish @clickhouse/client with head tag + run: | + npm --workspace @clickhouse/client publish \ + --access public \ + --provenance \ + --tag head + + publish: + name: "Publish @clickhouse/client" + if: github.ref == 'refs/heads/release' && github.event_name == 'workflow_dispatch' + runs-on: ubuntu-latest + environment: npm-publish + permissions: + contents: write # Required to push the release git tag + id-token: write # Required for npm OIDC authentication and provenance + outputs: + version: ${{ steps.version.outputs.version }} + packages: "@clickhouse/client" + steps: + - name: Checkout repository + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + + - name: Setup Node.js + uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: 24 + registry-url: "https://registry.npmjs.org" + + - name: Install dependencies + run: npm ci + + - name: Get the release version + id: version + run: | + BASE_VERSION=$(node -p "require('./packages/client-node/package.json').version") + echo "Using version: $BASE_VERSION" + echo "version=$BASE_VERSION" >> "$GITHUB_OUTPUT" + + - name: Build the package + run: npm --workspace @clickhouse/client run build + + - name: Smoke test the packed tarball + run: .scripts/smoke_test_pack.sh @clickhouse/client + + - name: Publish @clickhouse/client to the latest tag (implicit) + run: | + npm --workspace @clickhouse/client publish \ + --access public \ + --provenance + + - name: Create and push release git tag + env: + RELEASE_VERSION: ${{ steps.version.outputs.version }} + run: .scripts/push_release_tag.sh "client-${RELEASE_VERSION}" + + e2e: + name: e2e (node ${{ matrix.node }}) + needs: [head, publish] + # Runs whether the published version came from the head job or the manual + # latest publish. + if: | + always() && + (needs.head.result == 'success' || needs.publish.result == 'success') + runs-on: ubuntu-latest + strategy: + fail-fast: true + matrix: + node: [20, 22, 24, 26] + defaults: + run: + working-directory: tests/e2e/install + env: + PUBLISHED_VERSION: ${{ needs.head.outputs.version || needs.publish.outputs.version }} + PUBLISHED_PACKAGES: ${{ needs.head.outputs.packages || needs.publish.outputs.packages }} + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + + - name: Start ClickHouse (stable) in Docker + uses: hoverkraft-tech/compose-action@11beaa1c2dae4e8ed7b1665aa074723b6cecb0e4 # v3.0.0 + env: + CLICKHOUSE_VERSION: latest + with: + compose-file: "docker-compose.yml" + down-flags: "--volumes" + + - name: Setup NodeJS ${{ matrix.node }} + uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 + with: + node-version: ${{ matrix.node }} + registry-url: "https://registry.npmjs.org" + + - name: Wait for ${{ env.PUBLISHED_VERSION }} to be available on npm + run: | + set -euo pipefail + if [ -z "${PUBLISHED_VERSION}" ]; then + echo "PUBLISHED_VERSION is empty; cannot wait for npm publication." >&2 + exit 1 + fi + if [ -z "${PUBLISHED_PACKAGES}" ]; then + echo "PUBLISHED_PACKAGES is empty; cannot wait for npm publication." >&2 + exit 1 + fi + read -r -a packages <<< "${PUBLISHED_PACKAGES}" + # Poll the registry for up to ~5 minutes per package. New versions + # usually surface in seconds, but the registry CDN can lag. + max_attempts=60 + sleep_seconds=5 + for pkg in "${packages[@]}"; do + echo "Waiting for ${pkg}@${PUBLISHED_VERSION} to be available on npm..." + attempt=1 + while true; do + if npm view "${pkg}@${PUBLISHED_VERSION}" version >/dev/null 2>&1; then + echo " ${pkg}@${PUBLISHED_VERSION} is available." + break + fi + if [ "$attempt" -ge "$max_attempts" ]; then + echo "Timed out waiting for ${pkg}@${PUBLISHED_VERSION} on npm" >&2 + exit 1 + fi + echo " attempt ${attempt}/${max_attempts}: not available yet, sleeping ${sleep_seconds}s..." + attempt=$((attempt + 1)) + sleep "$sleep_seconds" + done + done + + - name: Install dependencies + run: npm install + + - name: Install the packages at the published version + run: | + set -euo pipefail + read -r -a packages <<< "${PUBLISHED_PACKAGES}" + specs=() + for pkg in "${packages[@]}"; do + specs+=("${pkg}@${PUBLISHED_VERSION}") + done + npm install "${specs[@]}" + + - name: Type check + run: npx tsc --noEmit + + - name: Run client code + env: + EXPECTED_VERSION: ${{ env.PUBLISHED_VERSION }} + run: node src/index.ts + + # End-to-end integration against the installed package and a live + # single-node ClickHouse (started above): create/insert/select, stream, + # and confirm a bad query surfaces as a ClickHouseError. + - name: Run integration tests against the published package + run: node src/integration.ts diff --git a/.github/workflows/publish-datatype-parser.yml b/.github/workflows/publish-datatype-parser.yml index cf3a39d85..8396f4dd7 100644 --- a/.github/workflows/publish-datatype-parser.yml +++ b/.github/workflows/publish-datatype-parser.yml @@ -1,10 +1,10 @@ name: "publish: datatype parser" # Independent publish + release for the standalone @clickhouse/datatype-parser -# package (the data-type string parser). It is NOT part of the npm workspace -# lockstep release driven by publish.yml — it carries its own version in +# package (the data-type string parser). Like the per-package client publish +# workflows (publish-client*.yml), it carries its own version in # packages/datatype-parser/package.json and ships on its own cadence. Triggered -# manually, and — like publish.yml — must be dispatched from the `release` +# manually, and — like those workflows — must be dispatched from the `release` # branch: the npm-publish environment is protected so only that branch may # deploy (the repo's human-in-the-loop release gate). Dispatches from any other # ref are skipped by the job-level `if` guard below. @@ -28,6 +28,14 @@ concurrency: on: workflow_dispatch: +env: + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" + jobs: publish: # The npm-publish environment only permits the release branch to deploy; @@ -129,7 +137,7 @@ jobs: strategy: fail-fast: true matrix: - node: [20, 22, 24] + node: [20, 22, 24, 26] env: PUBLISHED_VERSION: ${{ needs.publish.outputs.version }} steps: diff --git a/.github/workflows/publish-skill-rowbinary-parser.yml b/.github/workflows/publish-skill-rowbinary.yml similarity index 88% rename from .github/workflows/publish-skill-rowbinary-parser.yml rename to .github/workflows/publish-skill-rowbinary.yml index f4b94e6cc..f4fd63438 100644 --- a/.github/workflows/publish-skill-rowbinary-parser.yml +++ b/.github/workflows/publish-skill-rowbinary.yml @@ -1,10 +1,10 @@ -name: "publish: rowbinary parser" +name: "publish: rowbinary" # Independent publish + release for the standalone @clickhouse/rowbinary -# package (the RowBinary parser skill). It is NOT part of the npm workspace -# lockstep release driven by publish.yml — it carries its own version in -# skills/clickhouse-js-node-rowbinary-parser/package.json and ships on its own -# cadence. Triggered manually, and — like publish.yml — must be dispatched from +# package (the RowBinary codec skill). Like the per-package client publish +# workflows (publish-client*.yml), it carries its own version in +# skills/clickhouse-js-node-rowbinary/package.json and ships on its own +# cadence. Triggered manually, and — like those workflows — must be dispatched from # the `release` branch: the npm-publish environment is protected so only that # branch may deploy (the repo's human-in-the-loop release gate). Dispatches from # any other ref are skipped by the job-level `if` guard below. @@ -28,6 +28,14 @@ concurrency: on: workflow_dispatch: +env: + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" + jobs: publish: # The npm-publish environment only permits the release branch to deploy; @@ -41,7 +49,7 @@ jobs: id-token: write # Required for npm OIDC authentication and provenance defaults: run: - working-directory: skills/clickhouse-js-node-rowbinary-parser + working-directory: skills/clickhouse-js-node-rowbinary outputs: version: ${{ steps.version.outputs.version }} steps: @@ -89,7 +97,7 @@ jobs: # expose their parsers, from the exact artifact we are about to publish. node --input-type=module -e " import * as rb from '@clickhouse/rowbinary'; - import * as ints from '@clickhouse/rowbinary/integers'; + import * as ints from '@clickhouse/rowbinary/readers/integers'; if (typeof rb.readRows !== 'function') throw new Error('readRows missing from main export'); if (typeof ints.readUInt8 !== 'function') throw new Error('readUInt8 missing from subpath export'); console.log('OK: packed tarball imports cleanly'); @@ -124,7 +132,7 @@ jobs: strategy: fail-fast: true matrix: - node: [20, 22, 24] + node: [20, 22, 24, 26] env: PUBLISHED_VERSION: ${{ needs.publish.outputs.version }} steps: @@ -173,7 +181,7 @@ jobs: # expose their parsers to a downstream consumer. node --input-type=module -e " import * as rb from '@clickhouse/rowbinary'; - import * as ints from '@clickhouse/rowbinary/integers'; + import * as ints from '@clickhouse/rowbinary/readers/integers'; if (typeof rb.readRows !== 'function') throw new Error('readRows missing from main export'); if (typeof ints.readUInt8 !== 'function') throw new Error('readUInt8 missing from subpath export'); console.log('OK: @clickhouse/rowbinary@${PUBLISHED_VERSION} imports cleanly'); diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml deleted file mode 100644 index b71efe8d9..000000000 --- a/.github/workflows/publish.yml +++ /dev/null @@ -1,417 +0,0 @@ -name: "publish" - -# As NPM only supports a single workflow for publishing packages, -# this workflow is both triggered on push to the `release` branch and manually. -# When triggered manually, it will publish with the "latest" tag, -# and when triggered on push to `release`, it will publish with the "head" tag. -# The `main` branch is reserved for development and does not publish. -# For both it uses NPM OIDC authentication with provenance support. -# -# The manual ("latest") trigger takes a required `package` input and runs the -# matching per-package job, so a single package can be published on its own - -# for example, a final release of the deprecated `@clickhouse/client-common`. -# The automatic head publish runs one per-package job for every package (each -# computes its own `-head..` version from its package.json). -# -# Before each Node.js client publish, a smoke test packs the freshly built -# package into a tarball, installs that tarball into a throwaway app, and runs -# ESM + CJS checks against it (.scripts/smoke_test_pack.sh). This catches -# packaging problems before anything reaches npm. -# -# After a successful publish, the `e2e` job waits for the freshly published -# version to become available on the npm registry, installs that exact -# version into a tiny downstream project, and verifies it is usable. It only -# runs when `@clickhouse/client` was published (the head job or the client job). - -permissions: - contents: read - id-token: write # Required for npm OIDC authentication and provenance - -concurrency: - # Scope manual publishes by package so dispatching a release for one package - # does not cancel an in-progress release of another. `github.event.inputs` is - # empty for the head (push) trigger, so head runs still share a single group. - group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }}-${{ github.event.inputs.package }} - cancel-in-progress: true -on: - # for the latest workflow - workflow_dispatch: - inputs: - package: - description: "Package to publish" - required: true - type: choice - options: - - "@clickhouse/client" - - "@clickhouse/client-web" - # The common package is deprecated, but can still be published on its own. - - "@clickhouse/client-common" - # for the head workflow - push: - branches: - - release - # Only run the head publishing workflow when files relevant to the - # published packages change. The web and node packages bundle the - # common package sources, so any change under packages/** triggers an - # all-or-nothing publish of every package. - paths: - - "packages/**" - - "package.json" - - "package-lock.json" - - "tsconfig.base.json" - - "README.md" - - "LICENSE" - - "skills/**" - - ".github/workflows/publish.yml" - -jobs: - head_client: - name: "Publish @clickhouse/client (head)" - if: github.ref == 'refs/heads/release' && github.event_name == 'push' - runs-on: ubuntu-latest - environment: npm-publish - outputs: - version: ${{ steps.version.outputs.version }} - packages: "@clickhouse/client" - steps: - - name: Checkout repository - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 - - - name: Setup Node.js - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 - with: - node-version: 24 - registry-url: "https://registry.npmjs.org" - - - name: Install dependencies - run: npm ci - - - name: Set head pre-release version - id: version - run: | - BASE_VERSION=$(node -p "require('./packages/client-node/package.json').version") - HEAD_VERSION="${BASE_VERSION}-head.${GITHUB_SHA::7}.${GITHUB_RUN_ATTEMPT}" - echo "Setting version to: $HEAD_VERSION" - npm --workspace @clickhouse/client version --no-git-tag-version "$HEAD_VERSION" - echo "export default \"$HEAD_VERSION\";" > packages/client-node/src/version.ts - echo "version=$HEAD_VERSION" >> "$GITHUB_OUTPUT" - - - name: Build the package - run: npm --workspace @clickhouse/client run build - - - name: Smoke test the packed tarball - run: .scripts/smoke_test_pack.sh @clickhouse/client - - - name: Publish @clickhouse/client with head tag - run: | - npm --workspace @clickhouse/client publish \ - --access public \ - --provenance \ - --tag head - - head_client_web: - name: "Publish @clickhouse/client-web (head)" - if: github.ref == 'refs/heads/release' && github.event_name == 'push' - runs-on: ubuntu-latest - environment: npm-publish - steps: - - name: Checkout repository - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 - - - name: Setup Node.js - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 - with: - node-version: 24 - registry-url: "https://registry.npmjs.org" - - - name: Install dependencies - run: npm ci - - - name: Set head pre-release version - run: | - BASE_VERSION=$(node -p "require('./packages/client-web/package.json').version") - HEAD_VERSION="${BASE_VERSION}-head.${GITHUB_SHA::7}.${GITHUB_RUN_ATTEMPT}" - echo "Setting version to: $HEAD_VERSION" - npm --workspace @clickhouse/client-web version --no-git-tag-version "$HEAD_VERSION" - echo "export default \"$HEAD_VERSION\";" > packages/client-web/src/version.ts - - - name: Build the package - run: npm --workspace @clickhouse/client-web run build - - - name: Publish @clickhouse/client-web with head tag - run: | - npm --workspace @clickhouse/client-web publish \ - --access public \ - --provenance \ - --tag head - - head_client_common: - name: "Publish @clickhouse/client-common (head)" - if: github.ref == 'refs/heads/release' && github.event_name == 'push' - runs-on: ubuntu-latest - environment: npm-publish - steps: - - name: Checkout repository - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 - - - name: Setup Node.js - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 - with: - node-version: 24 - registry-url: "https://registry.npmjs.org" - - - name: Install dependencies - run: npm ci - - - name: Set head pre-release version - run: | - BASE_VERSION=$(node -p "require('./packages/client-common/package.json').version") - HEAD_VERSION="${BASE_VERSION}-head.${GITHUB_SHA::7}.${GITHUB_RUN_ATTEMPT}" - echo "Setting version to: $HEAD_VERSION" - npm --workspace @clickhouse/client-common version --no-git-tag-version "$HEAD_VERSION" - echo "export default \"$HEAD_VERSION\";" > packages/client-common/src/version.ts - - - name: Build the package - run: npm --workspace @clickhouse/client-common run build - - - name: Publish @clickhouse/client-common with head tag - run: | - npm --workspace @clickhouse/client-common publish \ - --access public \ - --provenance \ - --tag head - - publish_client: - name: "Publish @clickhouse/client" - if: github.ref == 'refs/heads/release' && github.event_name == 'workflow_dispatch' && inputs.package == '@clickhouse/client' - runs-on: ubuntu-latest - environment: npm-publish - permissions: - contents: write # Required to push the release git tag - id-token: write # Required for npm OIDC authentication and provenance - outputs: - version: ${{ steps.version.outputs.version }} - packages: "@clickhouse/client" - steps: - - name: Checkout repository - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 - - - name: Setup Node.js - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 - with: - node-version: 24 - registry-url: "https://registry.npmjs.org" - - - name: Install dependencies - run: npm ci - - - name: Get the release version - id: version - run: | - BASE_VERSION=$(node -p "require('./packages/client-node/package.json').version") - echo "Using version: $BASE_VERSION" - echo "version=$BASE_VERSION" >> "$GITHUB_OUTPUT" - - - name: Build the package - run: npm --workspace @clickhouse/client run build - - - name: Smoke test the packed tarball - run: .scripts/smoke_test_pack.sh @clickhouse/client - - - name: Publish @clickhouse/client to the latest tag (implicit) - run: | - npm --workspace @clickhouse/client publish \ - --access public \ - --provenance - - - name: Create and push release git tag - env: - RELEASE_VERSION: ${{ steps.version.outputs.version }} - run: .scripts/push_release_tag.sh "client-${RELEASE_VERSION}" - - publish_client_web: - name: "Publish @clickhouse/client-web" - if: github.ref == 'refs/heads/release' && github.event_name == 'workflow_dispatch' && inputs.package == '@clickhouse/client-web' - runs-on: ubuntu-latest - environment: npm-publish - permissions: - contents: write # Required to push the release git tag - id-token: write # Required for npm OIDC authentication and provenance - outputs: - version: ${{ steps.version.outputs.version }} - packages: "@clickhouse/client-web" - steps: - - name: Checkout repository - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 - - - name: Setup Node.js - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 - with: - node-version: 24 - registry-url: "https://registry.npmjs.org" - - - name: Install dependencies - run: npm ci - - - name: Get the release version - id: version - run: | - BASE_VERSION=$(node -p "require('./packages/client-web/package.json').version") - echo "Using version: $BASE_VERSION" - echo "version=$BASE_VERSION" >> "$GITHUB_OUTPUT" - - - name: Build the package - run: npm --workspace @clickhouse/client-web run build - - - name: Publish @clickhouse/client-web to the latest tag (implicit) - run: | - npm --workspace @clickhouse/client-web publish \ - --access public \ - --provenance - - - name: Create and push release git tag - env: - RELEASE_VERSION: ${{ steps.version.outputs.version }} - run: .scripts/push_release_tag.sh "client-web-${RELEASE_VERSION}" - - publish_client_common: - name: "Publish @clickhouse/client-common (deprecated)" - if: github.ref == 'refs/heads/release' && github.event_name == 'workflow_dispatch' && inputs.package == '@clickhouse/client-common' - runs-on: ubuntu-latest - environment: npm-publish - permissions: - contents: write # Required to push the release git tag - id-token: write # Required for npm OIDC authentication and provenance - outputs: - version: ${{ steps.version.outputs.version }} - packages: "@clickhouse/client-common" - steps: - - name: Checkout repository - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 - - - name: Setup Node.js - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 - with: - node-version: 24 - registry-url: "https://registry.npmjs.org" - - - name: Install dependencies - run: npm ci - - - name: Get the release version - id: version - run: | - BASE_VERSION=$(node -p "require('./packages/client-common/package.json').version") - echo "Using version: $BASE_VERSION" - echo "version=$BASE_VERSION" >> "$GITHUB_OUTPUT" - - - name: Build the package - run: npm --workspace @clickhouse/client-common run build - - - name: Publish @clickhouse/client-common to the latest tag (implicit) - run: | - npm --workspace @clickhouse/client-common publish \ - --access public \ - --provenance - - - name: Create and push release git tag - env: - RELEASE_VERSION: ${{ steps.version.outputs.version }} - run: .scripts/push_release_tag.sh "client-common-${RELEASE_VERSION}" - - e2e: - name: e2e (node ${{ matrix.node }}) - needs: [head_client, publish_client] - # The e2e project imports `@clickhouse/client`, so it only runs when the - # Node.js client was published - either via the head job or the client job. - if: | - always() && - (needs.head_client.result == 'success' || needs.publish_client.result == 'success') - runs-on: ubuntu-latest - strategy: - fail-fast: true - matrix: - node: [20, 22, 24] - defaults: - run: - working-directory: tests/e2e/install - env: - PUBLISHED_VERSION: ${{ needs.head_client.outputs.version || needs.publish_client.outputs.version }} - PUBLISHED_PACKAGES: ${{ needs.head_client.outputs.packages || needs.publish_client.outputs.packages }} - steps: - - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 - - - name: Start ClickHouse (stable) in Docker - uses: hoverkraft-tech/compose-action@11beaa1c2dae4e8ed7b1665aa074723b6cecb0e4 # v3.0.0 - env: - CLICKHOUSE_VERSION: latest - with: - compose-file: "docker-compose.yml" - down-flags: "--volumes" - - - name: Setup NodeJS ${{ matrix.node }} - uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0 - with: - node-version: ${{ matrix.node }} - registry-url: "https://registry.npmjs.org" - - - name: Wait for ${{ env.PUBLISHED_VERSION }} to be available on npm - run: | - set -euo pipefail - if [ -z "${PUBLISHED_VERSION}" ]; then - echo "PUBLISHED_VERSION is empty; cannot wait for npm publication." >&2 - exit 1 - fi - if [ -z "${PUBLISHED_PACKAGES}" ]; then - echo "PUBLISHED_PACKAGES is empty; cannot wait for npm publication." >&2 - exit 1 - fi - read -r -a packages <<< "${PUBLISHED_PACKAGES}" - # Poll the registry for up to ~5 minutes per package. New versions - # usually surface in seconds, but the registry CDN can lag. - max_attempts=60 - sleep_seconds=5 - for pkg in "${packages[@]}"; do - echo "Waiting for ${pkg}@${PUBLISHED_VERSION} to be available on npm..." - attempt=1 - while true; do - if npm view "${pkg}@${PUBLISHED_VERSION}" version >/dev/null 2>&1; then - echo " ${pkg}@${PUBLISHED_VERSION} is available." - break - fi - if [ "$attempt" -ge "$max_attempts" ]; then - echo "Timed out waiting for ${pkg}@${PUBLISHED_VERSION} on npm" >&2 - exit 1 - fi - echo " attempt ${attempt}/${max_attempts}: not available yet, sleeping ${sleep_seconds}s..." - attempt=$((attempt + 1)) - sleep "$sleep_seconds" - done - done - - - name: Install dependencies - run: npm install - - - name: Install the packages at the published version - run: | - set -euo pipefail - read -r -a packages <<< "${PUBLISHED_PACKAGES}" - specs=() - for pkg in "${packages[@]}"; do - specs+=("${pkg}@${PUBLISHED_VERSION}") - done - npm install "${specs[@]}" - - - name: Type check - run: npx tsc --noEmit - - - name: Run client code - env: - EXPECTED_VERSION: ${{ env.PUBLISHED_VERSION }} - run: node src/index.ts - - # End-to-end integration against the installed package and a live - # single-node ClickHouse (started above): create/insert/select, stream, - # and confirm a bad query surfaces as a ClickHouseError. - - name: Run integration tests against the published package - run: node src/integration.ts diff --git a/.github/workflows/tests-node.yml b/.github/workflows/tests-node.yml index 7a869b6d8..b509b334f 100644 --- a/.github/workflows/tests-node.yml +++ b/.github/workflows/tests-node.yml @@ -51,6 +51,12 @@ env: OTEL_RESOURCE_ATTRIBUTES: "service.namespace=clickhouse-js,deployment.environment=ci" VITEST_OTEL_ENABLED: "true" VITEST_COVERAGE: "true" + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" jobs: code-quality: @@ -100,7 +106,7 @@ jobs: strategy: fail-fast: false matrix: - node: [20, 22, 24] + node: [20, 22, 24, 26] steps: - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 @@ -142,7 +148,7 @@ jobs: strategy: fail-fast: false matrix: - node: [20, 22, 24] + node: [20, 22, 24, 26] steps: - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 @@ -189,7 +195,7 @@ jobs: strategy: fail-fast: false matrix: - node: [20, 22, 24] + node: [20, 22, 24, 26] clickhouse: [head, latest] log_level: [undefined, TRACE] include: @@ -263,7 +269,7 @@ jobs: strategy: fail-fast: false matrix: - node: [20, 22, 24] + node: [20, 22, 24, 26] clickhouse: [head, latest] steps: @@ -317,7 +323,7 @@ jobs: strategy: fail-fast: false matrix: - node: [20, 22, 24] + node: [20, 22, 24, 26] steps: - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 diff --git a/.github/workflows/tests-oss-dependents.yml b/.github/workflows/tests-oss-dependents.yml index eedcbd315..c48d22987 100644 --- a/.github/workflows/tests-oss-dependents.yml +++ b/.github/workflows/tests-oss-dependents.yml @@ -44,6 +44,14 @@ concurrency: group: "${{ github.workflow }}-${{ github.ref }}" cancel-in-progress: true +env: + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" + jobs: # Runnable reproductions of how the top OSS dependents use the client, resolved # against the workspace source via the @clickhouse/client* vitest aliases. This @@ -56,7 +64,7 @@ jobs: strategy: fail-fast: false matrix: - node: [20, 22, 24] + node: [20, 22, 24, 26] steps: - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 diff --git a/.github/workflows/tests-skill-rowbinary-parser.yml b/.github/workflows/tests-skill-rowbinary.yml similarity index 82% rename from .github/workflows/tests-skill-rowbinary-parser.yml rename to .github/workflows/tests-skill-rowbinary.yml index 3d73384a2..84801ee29 100644 --- a/.github/workflows/tests-skill-rowbinary-parser.yml +++ b/.github/workflows/tests-skill-rowbinary.yml @@ -1,4 +1,4 @@ -name: "skill: rowbinary parser" +name: "skill: rowbinary" permissions: {} on: @@ -7,28 +7,36 @@ on: branches: - main paths: - - .github/workflows/tests-skill-rowbinary-parser.yml - - skills/clickhouse-js-node-rowbinary-parser/** + - .github/workflows/tests-skill-rowbinary.yml + - skills/clickhouse-js-node-rowbinary/** # The skill depends on @clickhouse/datatype-parser; rerun against local # parser changes so a parser regression cannot pass this suite unnoticed. - packages/datatype-parser/** pull_request: paths: - - .github/workflows/tests-skill-rowbinary-parser.yml - - skills/clickhouse-js-node-rowbinary-parser/** + - .github/workflows/tests-skill-rowbinary.yml + - skills/clickhouse-js-node-rowbinary/** - packages/datatype-parser/** concurrency: group: "${{ github.workflow }}-${{ github.ref }}" cancel-in-progress: true +env: + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" + jobs: typecheck: timeout-minutes: 5 runs-on: ubuntu-latest defaults: run: - working-directory: skills/clickhouse-js-node-rowbinary-parser + working-directory: skills/clickhouse-js-node-rowbinary steps: - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 @@ -61,11 +69,11 @@ jobs: runs-on: ubuntu-latest defaults: run: - working-directory: skills/clickhouse-js-node-rowbinary-parser + working-directory: skills/clickhouse-js-node-rowbinary strategy: fail-fast: false matrix: - node: [20, 22, 24] + node: [20, 22, 24, 26] clickhouse: [head, latest] steps: - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 diff --git a/.github/workflows/tests-web.yml b/.github/workflows/tests-web.yml index b935ffa78..4fb2bae42 100644 --- a/.github/workflows/tests-web.yml +++ b/.github/workflows/tests-web.yml @@ -51,6 +51,12 @@ env: OTEL_RESOURCE_ATTRIBUTES: "service.namespace=clickhouse-js,deployment.environment=ci" VITEST_OTEL_ENABLED: "true" VITEST_COVERAGE: "true" + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" jobs: common-unit-tests: diff --git a/.github/workflows/upstream-sql-tests.yml b/.github/workflows/upstream-sql-tests.yml index 9ce98b251..1407cb354 100644 --- a/.github/workflows/upstream-sql-tests.yml +++ b/.github/workflows/upstream-sql-tests.yml @@ -6,9 +6,9 @@ on: workflow_dispatch: inputs: upstream_ref: - description: "ClickHouse/ClickHouse ref to check out" + description: "ClickHouse/ClickHouse ref to check out (defaults to the pinned commit)" required: false - default: "master" + default: "" type: string schedule: - cron: "0 5 * * *" @@ -29,6 +29,19 @@ concurrency: env: UPSTREAM_REPO: "ClickHouse/ClickHouse" + # Pinned ClickHouse commit for the upstream test suite. Floating `master` + # drifts ahead of the `head`/`latest` server images (e.g. new tests using + # settings the image doesn't know yet -> "Unknown setting ..."), which + # spuriously reds this workflow. This SHA is a known-good snapshot (last green + # run); bump it deliberately, or override per-run via the workflow_dispatch + # `upstream_ref` input. + UPSTREAM_REF: "edc2b4454dfed0453003168ee672a1249e7f46e9" + # Network resilience: npm's default of 2 fetch retries is not enough for + # the transient registry errors (ECONNRESET) we regularly hit in CI. + NPM_CONFIG_FETCH_RETRIES: "5" + NPM_CONFIG_FETCH_RETRY_MINTIMEOUT: "10000" + NPM_CONFIG_FETCH_RETRY_MAXTIMEOUT: "60000" + NPM_CONFIG_FETCH_TIMEOUT: "600000" jobs: upstream-sql-tests: @@ -37,7 +50,18 @@ jobs: strategy: fail-fast: false matrix: - clickhouse: [head, latest] + # Only the released `latest` image. The nightly `head` image rebuilds + # ~twice a day and routinely ships EXPLAIN/plan/index output changes and + # new settings ahead of any released server, spuriously reding this + # workflow (and every PR that touches the test runner). `latest` is a + # stable target; bleeding-edge coverage is not worth the daily noise. + clickhouse: [latest] + # passthrough: stream ClickHouse's own TabSeparated text (transport / + # session / settings coverage). rowbinary: decode + # RowBinaryWithNamesAndTypes through @clickhouse/rowbinary and re-render + # to TabSeparated, exercising the dynamic header->reader path. Each + # backend has its own allowlist (see the "Run upstream SQL tests" step). + backend: [passthrough, rowbinary] # Round-robin shards keep each job at roughly one minute so the # upstream SQL tests no longer dominate PR CI runtime. Bump # `shard` and `SHARD_TOTAL` together if the allowlist grows enough @@ -51,7 +75,7 @@ jobs: uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 with: repository: ${{ env.UPSTREAM_REPO }} - ref: ${{ github.event.inputs.upstream_ref || 'master' }} + ref: ${{ github.event.inputs.upstream_ref || env.UPSTREAM_REF }} path: tests/clickhouse-test-runner/.upstream/ClickHouse sparse-checkout: | tests/clickhouse-test @@ -100,14 +124,21 @@ jobs: CLICKHOUSE_CLIENT_CLI_LOG: ${{ github.workspace }}/upstream-run.log SHARD_INDEX: ${{ matrix.shard }} SHARD_TOTAL: 10 + TEST_RUNNER_BACKEND: ${{ matrix.backend }} run: | + # The rowbinary backend can only validate tests whose decoded types it + # can render back to TabSeparated, so it runs a dedicated (smaller) + # allowlist; passthrough runs the full one. + if [ "${TEST_RUNNER_BACKEND}" = "rowbinary" ]; then + export UPSTREAM_TEST_LIST="${{ github.workspace }}/tests/clickhouse-test-runner/rowbinary-allowlist.txt" + fi bash tests/clickhouse-test-runner/scripts/run-upstream-tests.sh --no-stateful - name: Upload test artifacts if: always() uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 with: - name: upstream-sql-tests-${{ matrix.clickhouse }}-shard-${{ matrix.shard }} + name: upstream-sql-tests-${{ matrix.backend }}-${{ matrix.clickhouse }}-shard-${{ matrix.shard }} retention-days: 14 if-no-files-found: ignore path: | diff --git a/AGENTS.md b/AGENTS.md index a24d8cf5e..a728c56e7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,129 +2,19 @@ > **Audience:** This file contains guidance for AI agents contributing to the `ClickHouse/clickhouse-js` repository itself. It is **not** intended for downstream projects that depend on `@clickhouse/client` or `@clickhouse/client-web` -1. When adding log messages, make sure to use eager log level checks to avoid unnecessary calculations for log messages that will not be emitted. For example: +This root file holds repo-wide guidance. Folder-specific guidance lives in nested `AGENTS.md` files next to the code they describe — read the one closest to the files you are editing: - ```ts - if (log_level <= ClickHouseLogLevel.WARN) { - log_writer.warn({ - message: "Example log message", - }); - } - ``` - -2. When adding new log messages with suggestions for users, make sure to create a unique documentation page under the `docs/` directory (use `docs/howto/` for task-style guides; see `docs/socket_hang_up_econnreset.md` as a reference) with a detailed explanation of the issue and how to resolve it. Then, include a link to that documentation page in the log message. For example: - - ```ts - if (some_condition) { - log_writer.warn({ - message: - "Example log message with suggestions for users. For more information, see https://github.com/ClickHouse/clickhouse-js/blob/main/docs/socket_hang_up_econnreset.md", - }); - } - ``` - -## Package structure and code duplication - -The source packages under [`packages/`](packages) are: - -- `client-common` — platform-agnostic shared code (config, query-param formatting, multipart - assembly, URL handling, result sets, etc.). It must not depend on Node.js-only or Web-only APIs. - The published `@clickhouse/client-common` package is **deprecated**: `client-node` and `client-web` - no longer depend on it and instead bundle its sources via the `src/common` symlink - (`packages/client-node/src/common` and `packages/client-web/src/common` both point to - `packages/client-common/src`), importing from it with relative paths (e.g. `./common/index`). -- `client-node` (`@clickhouse/client`) — the Node.js client. -- `client-web` (`@clickhouse/client-web`) — the Web/edge client. - -`client-node` and `client-web` are slated to be **separated into fully independent packages**. Because -of that, some logic is **intentionally duplicated** between the two connection implementations -(`packages/client-node/src/connection/node_base_connection.ts` and -`packages/client-web/src/connection/web_connection.ts`) rather than hoisted into `client-common` — for -example the per-request `use_multipart_params` resolution and the `param_*` multipart-part assembly -loop. **Do not flag this node/web duplication as something to consolidate**, and prefer keeping each -client self-contained over adding shared helpers that only exist to remove the duplication. Genuinely -platform-agnostic primitives (like `buildMultipartBody`) still belong in `client-common`. +- [`packages/AGENTS.md`](packages/AGENTS.md) — client source packages: log-message conventions, package structure, and intentional node/web duplication. +- [`examples/AGENTS.md`](examples/AGENTS.md) — the example corpus layout and conventions. +- [`skills/AGENTS.md`](skills/AGENTS.md) — shipped agent skills and how they are declared. + - [`skills/clickhouse-js-node-rowbinary/AGENTS.md`](skills/clickhouse-js-node-rowbinary/AGENTS.md) — `@clickhouse/rowbinary` reader/writer conventions (tests, no defensive validation). +- [`docs/AGENTS.md`](docs/AGENTS.md) — embedded troubleshooting / how-to pages. +- [`tests/clickhouse-test-runner/AGENTS.md`](tests/clickhouse-test-runner/AGENTS.md) — the upstream SQL test harness and allowlist strategy. ## Code intelligence (TypeScript LSP) The repository ships `typescript-language-server` as a root devDependency, so after `npm install` you can start a TypeScript language server with `npx typescript-language-server --stdio` from the repo root for precise go-to-definition, find-references, hover (signatures and JSDoc, including `@deprecated`), workspace symbol search, completions, and type diagnostics. Prefer it over text search when resolving symbols or usages across the `packages/*` workspaces. See [`.claude/skills/typescript-lsp/SKILL.md`](.claude/skills/typescript-lsp/SKILL.md) for verified capabilities and protocol notes. -## Examples - -The repository contains an [`examples`](examples) directory that is being refactored to be AI-agent-friendly. -The goals of the refactor are: - -1. Examples should be runnable right away, with no manual edits required to get them working against a - local ClickHouse instance (use `docker-compose up` from the repo root for the default setup). -2. Examples are organized by client flavor and tailored to the corresponding runtime: - - [`examples/node`](examples/node) — examples for the Node.js client (`@clickhouse/client`). These - may freely use Node.js-only APIs (file streams, TLS, `http`, `node:*` built-ins, etc.) and import - Node built-ins using the `node:` prefix (e.g., `node:fs`, `node:path`, `node:stream`). - - [`examples/web`](examples/web) — examples for the Web client (`@clickhouse/client-web`). These - must only use Web-platform APIs (e.g., `globalThis.crypto.randomUUID()` instead of Node's - `crypto` module) and must not depend on Node.js-only modules. -3. `examples/node` and `examples/web` are independent npm packages, each with its own `package.json`, - `tsconfig.json`, and ESLint config. Keep dependencies and configuration scoped to the relevant - subpackage. -4. General-purpose scenarios (configuration, ping, inserts, selects, parameters, sessions, etc.) should - exist in both subdirectories where applicable, with the only differences being the `import` - statement and any platform-specific adjustments. Examples that rely on Node.js-only APIs live only - under `examples/node`. -5. Within each subpackage, examples are split into intent-driven **use-case folders** so each folder - can back a focused AI agent skill: - - `coding/` — day-to-day client API usage (configure, ping, basic insert/select, parameter - binding, sessions, data types, custom JSON). - - `performance/` — async inserts, streaming with backpressure, file/Parquet streams, progress - streaming, server-side bulk moves. Mostly Node-only; `examples/web/performance/` exists for the - few perf scenarios that work in the browser (e.g. streaming `JSONEachRow`). - - `troubleshooting/` — cancellation, timeouts, long-running query progress, server error surfaces, - number-precision pitfalls. - - `security/` — TLS, RBAC, SQL-injection-safe parameter binding. - - `schema-and-deployments/` — `CREATE TABLE` examples for each deployment shape and - deployment-shaped connection strings. -6. A small number of examples are **intentionally duplicated** across folders so each folder is a - self-contained skill corpus. Each duplicated example has one _primary_ location; the secondary - copies are excluded from the Vitest runner via the per-package `vitest.config.ts`. When you edit - a duplicated example, update **all** copies. The current duplicates and their primary locations - are listed in [`examples/README.md`](examples/README.md#editing-duplicated-examples). - -## Skills - -- Each shipped skill must also be listed in the `agents.skills` array of - [`packages/client-node/package.json`](packages/client-node/package.json) so downstream tooling can - discover it. The [`Skills E2E`](.github/workflows/e2e-skills.yml) workflow - (`tests/e2e/skills/check.js`) asserts that the packaged tarball contains the declared skills. - -## Embedded docs - -The [`docs/`](docs) directory holds long-form troubleshooting / how-to pages that log messages and -skill references can link to (e.g. `docs/socket_hang_up_econnreset.md`, `docs/howto/`). Prefer -adding new pages here over linking out to external docs from log messages. - -## Upstream SQL test harness - -The [`tests/clickhouse-test-runner`](tests/clickhouse-test-runner) harness is a Node.js port of `clickhouse-client` that allows the official ClickHouse Python test runner (`tests/clickhouse-test`) to drive a subset of the upstream SQL test suite against `@clickhouse/client`. - -### What the harness does - -- Wraps `@clickhouse/client` in a tiny CLI (`bin/clickhouse` → `dist/main.js`) that mimics enough of the upstream `clickhouse-client` binary (same flags, `extract-from-config` shortcut, stdin/`--query` behavior) for the Python `tests/clickhouse-test` runner to drive it without modification. -- The runner is an npm workspace of the root `clickhouse-js` package, so `npm install` from the repo root links `@clickhouse/client` and `@clickhouse/client-common` from the local checkout instead of resolving them from the npm registry. Always install + build from the repo root (`npm install && npm run build`) so the harness exercises the code under review rather than the last published client. -- The CI matrix runs the harness against ClickHouse `latest` and `head` so that we exercise `@clickhouse/client` against both server versions and detect server regressions. The allowlist is also split into round-robin shards (`SHARD_INDEX` / `SHARD_TOTAL`) so each matrix job stays at roughly one minute; bump both the `shard` matrix values and the `SHARD_TOTAL` env value in the workflow together if per-shard runtime climbs back above ~1 minute. -- Reads the curated test list from [`upstream-allowlist.txt`](tests/clickhouse-test-runner/upstream-allowlist.txt) (one test name per line, `#` for comments) and forwards them as positional arguments to `tests/clickhouse-test`. -- The `SERVER_SETTINGS`/`CLIENT_ONLY_SETTINGS` allowlists in [`src/settings.ts`](tests/clickhouse-test-runner/src/settings.ts) are copied from the Java port and may need periodic resync as ClickHouse adds or reclassifies settings. - -See [`tests/clickhouse-test-runner/README.md`](tests/clickhouse-test-runner/README.md) for build, usage, and environment-variable documentation. When harness behavior changes (new wrapper flags, new short-circuited keys in `bin/clickhouse`, new entries in the settings allowlists), review the README and [`.github/workflows/upstream-sql-tests.yml`](.github/workflows/upstream-sql-tests.yml) to keep them in sync with the implementation. - -### Strategy for growing the allowlist - -The allowlist is grown in **batches of ~100 candidate tests at a time**, in upstream filename order, following this loop: - -1. **Pre-filter the candidate batch.** Skip non-SQL tests (`.sh`, `.py`, `.j2`) and tests tagged for unsupported infrastructure (`shard`, `distributed`, `replicated`, `zookeeper`, `kafka`, `s3`, `mysql`, `tls`, etc.). These will never pass through this harness as it stands today. -2. **Run each candidate through the harness** with `--no-stateful --no-long`. **Only keep tests that report `[ OK ]`**; drop failures and skips. -3. **Validate against the CI matrix before committing**, not just one local server version. The CI workflow runs `{ClickHouse latest, head} × {shard 1..N}` — a test that passes locally on `head` may fail on `latest` (or vice versa) and break CI. -4. **Beware substring/prefix expansion.** `tests/clickhouse-test` treats positional arguments as **substring/prefix matches** rather than exact names, so an allowlist entry like `00396_uuid` will silently pull in `00396_uuid_v7`, `00712_prewhere_with_alias` will pull in `00712_prewhere_with_alias_bug_2`, etc. When adding an entry whose name is a prefix of any other test in `0_stateless`, prefer the longest unambiguous form, or accept that the siblings come along and verify they all pass. -5. **Prune flakes promptly.** If a previously-passing test starts to flake on the nightly run, remove it (or its prefix-expanded siblings) from the allowlist rather than retrying — the allowlist exists to be a stable green signal, not a TODO list. - ## When reviewing code changes For every pull request review, make sure to provide an evaluation of the following aspects: @@ -139,11 +29,18 @@ For every pull request review, make sure to provide an evaluation of the followi 1. When reviewing code changes, it is important to consider the impact on the API quality and stability. For example, if the code changes involve modifying the library's public API surface (such as exported functions, classes, or types) or adding new public APIs, it is important to ensure that the changes are well-documented and do not break existing functionality for users of the library. -2. When introducing new features, fixing bugs, or making any change to observable behavior or the public API, you **must update [`CHANGELOG.md`](CHANGELOG.md) in the same PR** — do not defer it to "release time" or leave it only in the PR description. This satisfies the PR template checklist item ("A human-readable description of the changes was provided to include in CHANGELOG"). Follow the existing format exactly: - - Entries go under the **top-most version heading**. If the most recent `# x.y.z` heading corresponds to an **already-released** version (check `git tag`), open a **new** top-level `# x.y.z` heading that matches the unreleased version in `package.json` (e.g. the `version` field of [`packages/client-common/package.json`](packages/client-common/package.json)); otherwise append to the existing top heading. +2. When introducing new features, fixing bugs, or making any change to observable behavior or the public API, you **must update the changelog of every affected package in the same PR** — do not defer it to "release time" or leave it only in the PR description. This satisfies the PR template checklist item ("A human-readable description of the changes was provided to include in CHANGELOG"). Each package keeps its own changelog (the repository-wide [`CHANGELOG.md`](CHANGELOG.md) is **frozen** — do not add new entries there): + - `@clickhouse/client` → [`packages/client-node/CHANGELOG.md`](packages/client-node/CHANGELOG.md) + - `@clickhouse/client-web` → [`packages/client-web/CHANGELOG.md`](packages/client-web/CHANGELOG.md) + - `@clickhouse/client-common` (deprecated) → [`packages/client-common/CHANGELOG.md`](packages/client-common/CHANGELOG.md) + - `@clickhouse/datatype-parser` → [`packages/datatype-parser/CHANGELOG.md`](packages/datatype-parser/CHANGELOG.md) + - `@clickhouse/rowbinary` → [`skills/clickhouse-js-node-rowbinary/CHANGELOG.md`](skills/clickhouse-js-node-rowbinary/CHANGELOG.md) + + A change to shared code that is bundled into both clients (the common module) affects **both** `@clickhouse/client` and `@clickhouse/client-web`, so update both of their changelogs. Follow the existing format exactly: + - Entries go under the **top-most version heading** of that package's changelog. If the most recent `# x.y.z` heading corresponds to an **already-released** version (check `git tag`), open a **new** top-level `# x.y.z` heading that matches the unreleased version in that package's `package.json`; otherwise append to the existing top heading. - Group entries under lowercase section headings, reusing the ones already in the file: `## New features`, `## Improvements`, `## Bug fixes` (and `## Migration Notes` / `## Breaking changes` when relevant). - Write a concise, human-readable entry, add an example usage when it helps, and end it with a PR reference link, e.g. `([#825])` plus a matching `[#825]: https://github.com/ClickHouse/clickhouse-js/pull/` reference at the bottom of the section. - When a change is Node.js- or Web-only, say so explicitly in the entry (e.g. "(Node.js only)"). - - Run `npx prettier --write CHANGELOG.md` before committing. + - Run `npx prettier --write` on the changelog file(s) you touched before committing. 3. Additionally, make sure that the official documentation is in sync with the changes. diff --git a/CHANGELOG.md b/CHANGELOG.md index 096ef7c1b..b1ab17555 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,7 +1,23 @@ +> [!IMPORTANT] +> **This repository-wide changelog is frozen.** New entries now live in each +> package's own `CHANGELOG.md`: +> +> - `@clickhouse/client` → [`packages/client-node/CHANGELOG.md`](packages/client-node/CHANGELOG.md) +> - `@clickhouse/client-web` → [`packages/client-web/CHANGELOG.md`](packages/client-web/CHANGELOG.md) +> - `@clickhouse/client-common` (deprecated) → [`packages/client-common/CHANGELOG.md`](packages/client-common/CHANGELOG.md) +> - `@clickhouse/datatype-parser` → [`packages/datatype-parser/CHANGELOG.md`](packages/datatype-parser/CHANGELOG.md) +> - `@clickhouse/rowbinary` → [`skills/clickhouse-js-node-rowbinary-parser/CHANGELOG.md`](skills/clickhouse-js-node-rowbinary-parser/CHANGELOG.md) +> +> The history below (through `@clickhouse/client` 1.23.0) is retained for +> reference and was copied as-is into each client package's changelog as the +> starting point for the split. + # 1.23.0 ## Migration Notes +- Node.js 26.x was added to the CI matrix, and Node.js 18.x is no longer supported. The `engines.node` floor of `@clickhouse/client` (previously `>=16`) and `@clickhouse/datatype-parser` (previously `>=18.0.0`) was raised to `>=20`. Node.js 20.x, 22.x, 24.x, and 26.x are supported and exercised in CI. + - The `@clickhouse/client-common` package is deprecated. `@clickhouse/client` (Node.js) and `@clickhouse/client-web` (Web) no longer depend on it; the shared code is now bundled into each client package. Everything previously importable from `@clickhouse/client-common` should be imported from `@clickhouse/client` or `@clickhouse/client-web` instead. The `@clickhouse/client-common` package itself will no longer receive updates. ([#845]) - The `parseColumnType` function and its `SimpleColumnTypes` companion (exported from `@clickhouse/client`, `@clickhouse/client-web`, and `@clickhouse/client-common`) are deprecated and slated for removal in a future major version. They are superseded by the new standalone [`@clickhouse/datatype-parser`](https://www.npmjs.com/package/@clickhouse/datatype-parser) package (`parseDataType` plus its `Node` AST), which parses the full ClickHouse data-type grammar and emits an AST that mirrors the server's. ([#893]) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5e521a54e..2c3ef219b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -9,7 +9,7 @@ and help us maintain up-to-date documentation. You have installed: -- a compatible LTS version of Node.js: `v20.x`, `v22.x` or `v24.x` +- a compatible version of Node.js: `v20.x`, `v22.x`, `v24.x` or `v26.x` - NPM >= `9.x` ### Create a fork of the repository and clone it diff --git a/README.md b/README.md index 2506505f7..0c253c4a6 100644 --- a/README.md +++ b/README.md @@ -63,12 +63,12 @@ npm i @clickhouse/client-web Node.js must be available in the environment to run the Node.js client. The client is compatible with all the [maintained](https://github.com/nodejs/release#readme) Node.js releases. -| Node.js version | Supported? | -| --------------- | ----------- | -| 24.x | ✔ | -| 22.x | ✔ | -| 20.x | ✔ | -| 18.x | Best effort | +| Node.js version | Supported? | +| --------------- | ---------- | +| 26.x | ✔ | +| 24.x | ✔ | +| 22.x | ✔ | +| 20.x | ✔ | ### TypeScript @@ -110,6 +110,18 @@ See more examples in the [examples directory](./examples). See the [ClickHouse website](https://clickhouse.com/docs/integrations/javascript) for the full documentation. +## Changelog + +Each package keeps its own changelog: + +- `@clickhouse/client` — [`packages/client-node/CHANGELOG.md`](./packages/client-node/CHANGELOG.md) +- `@clickhouse/client-web` — [`packages/client-web/CHANGELOG.md`](./packages/client-web/CHANGELOG.md) +- `@clickhouse/client-common` (deprecated) — [`packages/client-common/CHANGELOG.md`](./packages/client-common/CHANGELOG.md) +- `@clickhouse/datatype-parser` — [`packages/datatype-parser/CHANGELOG.md`](./packages/datatype-parser/CHANGELOG.md) +- `@clickhouse/rowbinary` — [`skills/clickhouse-js-node-rowbinary/CHANGELOG.md`](./skills/clickhouse-js-node-rowbinary/CHANGELOG.md) + +History through `@clickhouse/client` 1.23.0 lives in the now-frozen repository-wide [`CHANGELOG.md`](./CHANGELOG.md). + ## AI Agent Skills This repository contains agent skills for working with the client: diff --git a/RELEASING.md b/RELEASING.md index f1ffa5f6d..c7d1655c8 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -1,55 +1,16 @@ # Release process -Tools required (for verifying a published build and promoting npm tags locally): +The release process now lives in the in-repo **`release` skill**, which is the +single source of truth and is kept in step with the GitHub Actions workflows: -- Node.js >= `20.x` -- npm >= `11.x` — newer than the npm bundled with Node.js `20.x`/`22.x` (`10.x`). Either upgrade npm in place (`npm install -g npm@latest`) or use Node.js `24.x`, which already ships npm `11.x`. +- [.claude/skills/release/SKILL.md](.claude/skills/release/SKILL.md) -Packages are versioned and released independently. Release one package at a time; to release several, repeat the steps below for each. Versions are bumped through the GitHub Actions workflows — there is no local version-bump script. +It covers all the packages — `@clickhouse/client`, `@clickhouse/client-web`, +the deprecated `@clickhouse/client-common`, and the standalone +`@clickhouse/datatype-parser` and `@clickhouse/rowbinary` — including version +bumping, syncing the protected `release` branch from `main`, the `npm-publish` +approval gate, and creating the GitHub Release from `CHANGELOG.md`. -## Bump the version - -Run the [`bump-version`](.github/workflows/bump-version.yml) workflow from the GitHub Actions tab. Select: - -- `package` — the package to release: `@clickhouse/client`, `@clickhouse/client-web`, or `@clickhouse/client-common` (the last is deprecated, but can still be cut a final standalone release). -- `bump_type` — `patch`, `minor`, or `major`. - -The workflow computes the next version, bumps that package's `package.json` and `src/version.ts`, and opens a release PR against `main`. - -Review and merge the PR into `main`. - -## Publish the `head` build - -The signed `head` build is published by the [`publish`](.github/workflows/publish.yml) workflow on push to the long-lived `release` branch — not on merge to `main`. After the bump PR is merged, update `release` from `main` (open a PR from `main` into `release` and merge it). That push triggers the `head` publish for every package. - -## Test the `head` build - -After the package is published it can be tested in a separate project by installing it with the `head` tag: - -```bash -npm install @clickhouse/client@head -``` - -and run a simple e2e test: https://github.com/ClickHouse/clickhouse-js/actions/workflows/npm.yml - -## Promote the `head` tag to `latest` - -Run this for the package(s) you released: - -```bash -npm dist-tag add @clickhouse/client@head latest -npm dist-tag add @clickhouse/client-web@head latest -npm dist-tag add @clickhouse/client-common@head latest -``` - -Mark the deprecated `@clickhouse/client-common` package as such on npm (it is no longer used by `@clickhouse/client` or `@clickhouse/client-web`; the shared code is bundled into each client package): - -```bash -npm deprecate @clickhouse/client-common "This package is deprecated and no longer used by @clickhouse/client or @clickhouse/client-web. Import everything from @clickhouse/client (Node.js) or @clickhouse/client-web (Web) instead." -``` - -Check that the packages have been published correctly: - -Then create a new release in GitHub for the published version and include the corresponding changelog notes. - -All done, thanks! +If you use Claude Code in this repo, just ask it to "release ``" and it +will drive the process. Otherwise, follow the steps in the skill document +directly. diff --git a/demo/logs/README.md b/demo/logs/README.md index f2cbf983b..58868fcf2 100644 --- a/demo/logs/README.md +++ b/demo/logs/README.md @@ -34,7 +34,7 @@ hot path. ## Prerequisites -- Node 18+ (built/tested on Node 24) +- Node 20+ (built/tested on Node 24) - A running ClickHouse. This demo ships a self-contained one — from **this directory** (`demo/logs`): diff --git a/docs/AGENTS.md b/docs/AGENTS.md new file mode 100644 index 000000000..b6e66ffdc --- /dev/null +++ b/docs/AGENTS.md @@ -0,0 +1,7 @@ +# Recommendations for AI agents — `docs/` + +Guidance for the embedded docs. See the [repo-root `AGENTS.md`](../AGENTS.md) for cross-cutting guidance. + +This directory holds long-form troubleshooting / how-to pages that log messages and +skill references can link to (e.g. [`socket_hang_up_econnreset.md`](socket_hang_up_econnreset.md), +[`howto/`](howto)). Prefer adding new pages here over linking out to external docs from log messages. diff --git a/examples/AGENTS.md b/examples/AGENTS.md new file mode 100644 index 000000000..cd8574f9c --- /dev/null +++ b/examples/AGENTS.md @@ -0,0 +1,39 @@ +# Recommendations for AI agents — `examples/` + +Guidance for the example corpus. See the [repo-root `AGENTS.md`](../AGENTS.md) for cross-cutting guidance. + +The [`examples`](.) directory is being refactored to be AI-agent-friendly. The goals of the refactor are: + +1. Examples should be runnable right away, with no manual edits required to get them working against a + local ClickHouse instance (use `docker-compose up` from the repo root for the default setup). +2. Examples are organized by client flavor and tailored to the corresponding runtime: + - [`examples/node`](node) — examples for the Node.js client (`@clickhouse/client`). These + may freely use Node.js-only APIs (file streams, TLS, `http`, `node:*` built-ins, etc.) and import + Node built-ins using the `node:` prefix (e.g., `node:fs`, `node:path`, `node:stream`). + - [`examples/web`](web) — examples for the Web client (`@clickhouse/client-web`). These + must only use Web-platform APIs (e.g., `globalThis.crypto.randomUUID()` instead of Node's + `crypto` module) and must not depend on Node.js-only modules. +3. `examples/node` and `examples/web` are independent npm packages, each with its own `package.json`, + `tsconfig.json`, and ESLint config. Keep dependencies and configuration scoped to the relevant + subpackage. +4. General-purpose scenarios (configuration, ping, inserts, selects, parameters, sessions, etc.) should + exist in both subdirectories where applicable, with the only differences being the `import` + statement and any platform-specific adjustments. Examples that rely on Node.js-only APIs live only + under `examples/node`. +5. Within each subpackage, examples are split into intent-driven **use-case folders** so each folder + can back a focused AI agent skill: + - `coding/` — day-to-day client API usage (configure, ping, basic insert/select, parameter + binding, sessions, data types, custom JSON). + - `performance/` — async inserts, streaming with backpressure, file/Parquet streams, progress + streaming, server-side bulk moves. Mostly Node-only; `examples/web/performance/` exists for the + few perf scenarios that work in the browser (e.g. streaming `JSONEachRow`). + - `troubleshooting/` — cancellation, timeouts, long-running query progress, server error surfaces, + number-precision pitfalls. + - `security/` — TLS, RBAC, SQL-injection-safe parameter binding. + - `schema-and-deployments/` — `CREATE TABLE` examples for each deployment shape and + deployment-shaped connection strings. +6. A small number of examples are **intentionally duplicated** across folders so each folder is a + self-contained skill corpus. Each duplicated example has one _primary_ location; the secondary + copies are excluded from the Vitest runner via the per-package `vitest.config.ts`. When you edit + a duplicated example, update **all** copies. The current duplicates and their primary locations + are listed in [`examples/README.md`](README.md#editing-duplicated-examples). diff --git a/examples/README.md b/examples/README.md index 0561012c7..423e9118f 100644 --- a/examples/README.md +++ b/examples/README.md @@ -159,7 +159,7 @@ deployment-shaped connection strings. Environment requirements for all examples: -- Node.js 18+ +- Node.js 20+ - NPM - Docker Compose diff --git a/package-lock.json b/package-lock.json index 20dbe4381..fe034ccdd 100644 --- a/package-lock.json +++ b/package-lock.json @@ -347,6 +347,18 @@ "resolved": "packages/datatype-parser", "link": true }, + "node_modules/@clickhouse/rowbinary": { + "version": "0.1.2", + "resolved": "https://registry.npmjs.org/@clickhouse/rowbinary/-/rowbinary-0.1.2.tgz", + "integrity": "sha512-BHc8DXdK+ORxQe5BhPC9bJbOk0zmJlXvdC7rhiMGFcECT+QyZLgAo4aQuwpRW7Ms90/ufict28JBmiON+rUcjw==", + "license": "Apache-2.0", + "dependencies": { + "@clickhouse/datatype-parser": "^0.1.2" + }, + "engines": { + "node": ">=20" + } + }, "node_modules/@emnapi/core": { "version": "1.10.0", "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.10.0.tgz", @@ -7710,7 +7722,7 @@ "simdjson": "^0.9.2" }, "engines": { - "node": ">=16" + "node": ">=20" } }, "packages/client-web": { @@ -7728,7 +7740,7 @@ "vitest": "^4.0.16" }, "engines": { - "node": ">=18.0.0" + "node": ">=20" } }, "packages/datatype-parser/node_modules/typescript": { @@ -7749,7 +7761,8 @@ "name": "@clickhouse/clickhouse-test-runner", "version": "1.23.0", "dependencies": { - "@clickhouse/client": "*" + "@clickhouse/client": "*", + "@clickhouse/rowbinary": "^0.1.2" }, "bin": { "clickhouse-js-test-runner": "dist/main.js" diff --git a/packages/AGENTS.md b/packages/AGENTS.md new file mode 100644 index 000000000..288a61ebd --- /dev/null +++ b/packages/AGENTS.md @@ -0,0 +1,48 @@ +# Recommendations for AI agents — `packages/` + +Guidance for the client source packages. See the [repo-root `AGENTS.md`](../AGENTS.md) for cross-cutting guidance (code intelligence, code-review expectations, changelog rules). + +## Log messages + +1. When adding log messages, make sure to use eager log level checks to avoid unnecessary calculations for log messages that will not be emitted. For example: + + ```ts + if (log_level <= ClickHouseLogLevel.WARN) { + log_writer.warn({ + message: "Example log message", + }); + } + ``` + +2. When adding new log messages with suggestions for users, make sure to create a unique documentation page under the [`docs/`](../docs) directory (use `docs/howto/` for task-style guides; see `docs/socket_hang_up_econnreset.md` as a reference) with a detailed explanation of the issue and how to resolve it. Then, include a link to that documentation page in the log message. For example: + + ```ts + if (some_condition) { + log_writer.warn({ + message: + "Example log message with suggestions for users. For more information, see https://github.com/ClickHouse/clickhouse-js/blob/main/docs/socket_hang_up_econnreset.md", + }); + } + ``` + +## Package structure and code duplication + +The source packages here are: + +- `client-common` — platform-agnostic shared code (config, query-param formatting, multipart + assembly, URL handling, result sets, etc.). It must not depend on Node.js-only or Web-only APIs. + The published `@clickhouse/client-common` package is **deprecated**: `client-node` and `client-web` + no longer depend on it and instead bundle its sources via the `src/common` symlink + (`client-node/src/common` and `client-web/src/common` both point to + `client-common/src`), importing from it with relative paths (e.g. `./common/index`). +- `client-node` (`@clickhouse/client`) — the Node.js client. +- `client-web` (`@clickhouse/client-web`) — the Web/edge client. + +`client-node` and `client-web` are slated to be **separated into fully independent packages**. Because +of that, some logic is **intentionally duplicated** between the two connection implementations +(`client-node/src/connection/node_base_connection.ts` and +`client-web/src/connection/web_connection.ts`) rather than hoisted into `client-common` — for +example the per-request `use_multipart_params` resolution and the `param_*` multipart-part assembly +loop. **Do not flag this node/web duplication as something to consolidate**, and prefer keeping each +client self-contained over adding shared helpers that only exist to remove the duplication. Genuinely +platform-agnostic primitives (like `buildMultipartBody`) still belong in `client-common`. diff --git a/packages/client-common/CHANGELOG.md b/packages/client-common/CHANGELOG.md new file mode 100644 index 000000000..7ae3e0373 --- /dev/null +++ b/packages/client-common/CHANGELOG.md @@ -0,0 +1,1342 @@ +# 1.23.0 + +## Migration Notes + +- Node.js 26.x was added to the CI matrix, and Node.js 18.x is no longer supported. The `engines.node` floor of `@clickhouse/client` (previously `>=16`) and `@clickhouse/datatype-parser` (previously `>=18.0.0`) was raised to `>=20`. Node.js 20.x, 22.x, 24.x, and 26.x are supported and exercised in CI. + +- The `@clickhouse/client-common` package is deprecated. `@clickhouse/client` (Node.js) and `@clickhouse/client-web` (Web) no longer depend on it; the shared code is now bundled into each client package. Everything previously importable from `@clickhouse/client-common` should be imported from `@clickhouse/client` or `@clickhouse/client-web` instead. The `@clickhouse/client-common` package itself will no longer receive updates. ([#845]) + +- The `parseColumnType` function and its `SimpleColumnTypes` companion (exported from `@clickhouse/client`, `@clickhouse/client-web`, and `@clickhouse/client-common`) are deprecated and slated for removal in a future major version. They are superseded by the new standalone [`@clickhouse/datatype-parser`](https://www.npmjs.com/package/@clickhouse/datatype-parser) package (`parseDataType` plus its `Node` AST), which parses the full ClickHouse data-type grammar and emits an AST that mirrors the server's. ([#893]) + +## New features + +- (Node.js) Added a RowBinary reader library and agent skill under [`skills/clickhouse-js-node-rowbinary-parser`](../../skills/clickhouse-js-node-rowbinary-parser). It ships type-specific, monomorphizable building blocks for decoding `RowBinary` / `RowBinaryWithNames` / `RowBinaryWithNamesAndTypes` streams (full-buffer and chunked), plus a skill that guides an agent to generate bespoke high-performance parsers from a query's column types. The skill is bundled into `@clickhouse/client` (registered in `agents.skills`) and is also published independently as the [`@clickhouse/rowbinary`](https://www.npmjs.com/package/@clickhouse/rowbinary) package. A matching RowBinary writer is planned. ([#864]) + +- Published the [`@clickhouse/datatype-parser`](https://www.npmjs.com/package/@clickhouse/datatype-parser) package: a small, dependency-free standalone parser for ClickHouse data-type strings (the kind sent in the types row of `RowBinaryWithNamesAndTypes`, e.g. `Array(Nullable(UInt64))`, `Tuple(a UInt8, b String)`, `Enum8('a' = 1)`). It is a faithful port of the server's `ParserDataType` and emits a JSON AST that is byte-identical to the server's `EXPLAIN AST json = 1` data-type subtree. It supersedes the deprecated `parseColumnType` (see Migration Notes). ([#893]) + +- (Node.js, `@experimental`) Added an additive `connection?: Connection` option to `createClient` that lets a caller plug an externally-built backend `Connection`-like object in place of the default HTTP(S) factory. Only supposed to be used for testing the `chDB` integration. ([#879]) + +- Added `ClickHouseSettingsInterface`, a package-neutral structural counterpart to `ClickHouseSettings`, exported from `@clickhouse/client`, `@clickhouse/client-web`, and `@clickhouse/client-common`. It is identical to `ClickHouseSettings` except that its index signature omits `SettingsMap` (a class with a private member, which TypeScript compares nominally). Because each client package now bundles its own copy of the common module, their `ClickHouseSettings` types are mutually unassignable; `ClickHouseSettingsInterface` is structurally identical across all three packages and assignable into each package's `ClickHouseSettings`, so a consumer that shares a single settings-producing helper across both the Node.js and Web clients can type it against this one type without casts. Values typed as `SettingsMap` cannot be carried through it — use `ClickHouseSettings` if you need them. ([#889]) + +# 1.22.0 + +## New features + +- (Node.js) The `compression.request` / `compression.response` client options now accept an explicit codec via an object, in addition to the existing boolean: `true` keeps gzip (backwards compatible), and `{ codec: "zstd" }` selects zstd. The object form is intentionally extensible for future codecs and codec-specific options. zstd typically yields a similar-or-better ratio than gzip at noticeably lower CPU cost (gzip/DEFLATE is comparatively CPU-heavy and decompressed single-threaded by the ClickHouse server), and it uses the built-in `zlib` zstd support, so it requires **Node.js >= 22.15.0** (`@clickhouse/client` throws a clear error at client creation otherwise). Response decompression is driven by the server's actual `Content-Encoding`, so it degrades gracefully. The request object form also accepts an optional `level` (`{ codec, level }`) to set the codec-specific compression level (zlib level for gzip, zstd compression level for zstd); the response compression level is controlled by the server. Supported only by `@clickhouse/client` (Node.js); `@clickhouse/client-web` rejects the `zstd` codec at client creation. + +- (Node.js) Brotli (`{ codec: "br" }`) is now supported for `compression.request` / `compression.response`, alongside gzip and zstd. Unlike zstd, Brotli is available on every supported Node.js version (no minimum-version requirement). The `compression.request` option is a per-codec discriminated union, so each codec exposes its own tuning option: a `level` for gzip/zstd, a `quality` for Brotli (`{ codec: "br", quality }`). When omitted, Brotli defaults to quality 4 for request bodies, since zlib's brotli default of 11 (max) is far too slow for a streaming insert path. Response decompression follows the server's `Content-Encoding`. Supported only by `@clickhouse/client` (Node.js). + +## Internal changes (`@clickhouse/client-common`) + +> These only affect code that imports the low-level connection primitives from the deprecated `@clickhouse/client-common` package directly (e.g. a custom `Connection` implementation). The `createClient` `compression` option is unchanged and fully backwards compatible — if you only use `@clickhouse/client` or `@clickhouse/client-web`, you are not affected. + +To carry the codec (and its optional compression level) instead of a bare on/off flag, the internal compression representation changed shape: + +- `CompressionSettings.compress_request` / `decompress_response` are no longer `boolean`. They are now a normalized codec object or `undefined` (disabled): `{ codec: "gzip" | "zstd"; level?: number } | { codec: "br"; quality?: number }` for the request, `{ codec: "gzip" | "zstd" | "br" }` for the response (response compression options are chosen by the server). `getConnectionParams` normalizes the public request option into this form (`true` → `{ codec: "gzip" }`). +- `withCompressionHeaders` now takes `request_compression_codec` / `response_compression_codec` (a `CompressionMethod | undefined`) instead of the boolean `enable_request_compression` / `enable_response_compression`; the codec value is also the `Content-Encoding` / `Accept-Encoding` it emits. +- `withHttpSettings` now takes the response codec object (`{ codec } | undefined`) instead of a `boolean`. +- New exported types: `CompressionMethod`, `RequestCompression`, `ResponseCompression`. + +Why: a single `boolean` could not express which codec to use or its level, and a separate level field on `CompressionSettings` would have mixed a codec-specific option into the shared type. Discriminating by codec keeps each codec's options on the codec it belongs to. + +## Documentation + +- Added two **tracer adapter recipes** to [`docs/howto/tracing.md`](../../docs/howto/tracing.md) and [`examples/node/coding/otel_tracing.ts`](../../examples/node/coding/otel_tracing.ts), demonstrating how common OpenTelemetry auto-instrumentation options compose as thin userland wrappers around the `tracer` API instead of being baked into the client: `requireParentSpan` (skip ClickHouse spans when there is no active parent span — e.g. background health checks) and suppressing the duplicate nested HTTP spans emitted by `@opentelemetry/instrumentation-http` (via `suppressTracing` from `@opentelemetry/core`). + +# 1.21.0 + +## New features + +- The tracer API (unreleased, introduced in [#776]) now follows the [OpenTelemetry database semantic conventions](https://opentelemetry.io/docs/specs/semconv/db/sql/) and matches the attribute vocabulary of the Rust client ([clickhouse-rs](https://github.com/ClickHouse/clickhouse-rs)); see [`docs/howto/tracing.md`](../../docs/howto/tracing.md) for the documentation. In particular ([#828]): + - Spans now carry `db.system.name` (instead of `db.system`), `server.address` + `server.port` (instead of a combined `host:port`), `clickhouse.request.query_id` / `clickhouse.request.session_id` (instead of `clickhouse.query_id` / `clickhouse.session_id`), `clickhouse.response.format` on `query` and `clickhouse.request.format` on `insert` (instead of `clickhouse.format`), and `db.operation.name` + `db.collection.name` on `insert` (instead of `clickhouse.table`). + - The span status is left unset on success (per the OTEL spec recommendation for client spans, previously set to `OK`); on failure, the span gets the `error.type` attribute (the error class name) and, for server-side errors, `clickhouse.error.code` (the numeric ClickHouse error code). + - Spans record response-side attributes: `db.response.status_code` (HTTP status) and, when the `X-ClickHouse-Summary` header is available, `clickhouse.summary.*` counters (`read_rows`, `written_rows`, etc.). + - `query()` now emits two spans: `clickhouse.query` covers the HTTP request lifetime and ends as soon as the response headers are received; a child `clickhouse.query.stream` span is handed to the `ResultSet` and tracks the stream consumption, ending when the response is fully read, closed, or fails - with the final `clickhouse.response.decoded_bytes` and (for row-streaming) `db.response.returned_rows` metrics. This separation makes it easy to distinguish the original request duration from a stream that may never end (e.g. tailing a live table). + - Fixed a span leak in the Web `ResultSet.stream()` path: if the underlying fetch response stream was aborted (e.g. due to a network error), the `clickhouse.query.stream` span was never ended. The TransformStream now handles both source-stream aborts and consumer-side cancellations via a `cancel` callback. + - The `insert` span records `clickhouse.request.sent_rows` for array-based inserts. + +- Added a `use_multipart_params_auto` client option (default: `false`). When enabled, `query()` automatically sends `query_params` as `multipart/form-data` body parts (the same mechanism as `use_multipart_params`) once their URL-encoded length exceeds 4096 characters, avoiding HTTP 414/400 errors from HTTP intermediaries (nginx, AWS ALB, CloudFront) caused by over-long URLs - for example, a large `IN` list or a high-dimensional vector embedding. Smaller parameter payloads remain in the URL query string, so existing behavior is unchanged unless the threshold is crossed. `use_multipart_params: true` still forces multipart for all queries regardless of size. This does not change the server's per-value size limit, which is governed by `http_max_field_value_size`. Supported on both `@clickhouse/client` and `@clickhouse/client-web`, and overridable per request via `use_multipart_params_auto` on `query()`. Ported from [clickhouse-connect#789](https://github.com/ClickHouse/clickhouse-connect/pull/789). ([#827]) + +```ts +const client = createClient({ use_multipart_params_auto: true }); + +await client.query({ + query: "SELECT * FROM events WHERE id IN {ids:Array(UInt64)}", + // Sent in the URL when small, auto-promoted to the multipart body when large + query_params: { ids: veryLargeArrayOfIds }, +}); +``` + +- Added a `use_multipart_params` client option (default: `false`). When enabled, `query()` sends `query_params` as `multipart/form-data` body parts (with the SQL moved into a `query` part) instead of URL query-string entries, avoiding HTTP 400 errors caused by over-long URLs when parameters contain large arrays (25K+ values). All other URL search params (database, query_id, settings, session_id, role) remain in the URL. Supported on both `@clickhouse/client` and `@clickhouse/client-web`, and overridable per request via `use_multipart_params` on `query()`. ([#825]) + +```ts +const client = createClient({ use_multipart_params: true }); + +await client.query({ + query: "SELECT * FROM events WHERE id IN {ids:Array(UInt64)}", + query_params: { ids: veryLargeArrayOfIds }, + // Per-request override is also supported: + // use_multipart_params: false, +}); +``` + +[#776]: https://github.com/ClickHouse/clickhouse-js/pull/776 +[#825]: https://github.com/ClickHouse/clickhouse-js/pull/825 +[#827]: https://github.com/ClickHouse/clickhouse-js/pull/827 +[#828]: https://github.com/ClickHouse/clickhouse-js/pull/828 +[#845]: https://github.com/ClickHouse/clickhouse-js/pull/845 +[#864]: https://github.com/ClickHouse/clickhouse-js/pull/864 +[#889]: https://github.com/ClickHouse/clickhouse-js/pull/889 +[#893]: https://github.com/ClickHouse/clickhouse-js/pull/893 + +## Bug Fixes + +- The client now checks the `X-ClickHouse-Exception-Code` response header to detect server errors even when the HTTP status code indicates success. In some scenarios (for example, when an exception occurs while streaming the response progress in headers, or with certain proxy setups), ClickHouse responds with HTTP 200 but sets the `X-ClickHouse-Exception-Code` header. Previously, such responses were treated as successful, and the exception text could surface as malformed response data; now the request is rejected with a parsed `ClickHouseError` (with the proper `code` and `type`), consistent with non-2xx error responses. This applies to both the Node.js and Web clients. ([#554], supersedes [#350], related issue: [#332]) + +[#554]: https://github.com/ClickHouse/clickhouse-js/pull/554 +[#350]: https://github.com/ClickHouse/clickhouse-js/pull/350 +[#332]: https://github.com/ClickHouse/clickhouse-js/issues/332 + +# 1.20.0 + +## New Features + +- Added an optional **tracer API** that the user can pass through the client config (`tracer`) and that gets called around key lifecycle operations (`query`, `command`, `exec`, `insert`, `ping`). The `ClickHouseTracer` interface is a structural subset of the OpenTelemetry `Tracer`/`Span` APIs, so a raw OTEL tracer (`trace.getTracer(...)`) can be passed to the client as-is - but the client itself ships no tracing dependency. Each operation runs inside `tracer.startActiveSpan(...)`, so auto-instrumented child spans nest under the ClickHouse operation spans; for OpenTelemetry, this requires the `AsyncLocalStorageContextManager` to be registered (the default in the OpenTelemetry Node.js SDK). Tracer exceptions are NOT caught, so a broken tracer will break client operations. See [`docs/howto/tracing.md`](../../docs/howto/tracing.md) for the full surface description, and [`examples/node/coding/otel_tracing.ts`](../../examples/node/coding/otel_tracing.ts) for a runnable Node.js example. ([#776]) + +```ts +import { createClient } from "@clickhouse/client"; +import { trace } from "@opentelemetry/api"; + +// a raw OpenTelemetry tracer is structurally compatible - no adapter needed +const client = createClient({ + url: "http://localhost:8123", + tracer: trace.getTracer("@clickhouse/client"), +}); +``` + +## Migration Notes + +- TypeScript: `ClickHouseLogLevel` is now exported as a literal numeric union type (`0 | 1 | 2 | 3 | 4 | 127`) instead of a TypeScript `enum` type. If you were assigning arbitrary `number` values to `ClickHouseLogLevel`, you may need to narrow/cast those values during migration. + +## Improvements + +- Added TypeScript typings for the remaining HTTP-specific ClickHouse settings, so they are now suggested by autocomplete when used in `clickhouse_settings`: `buffer_size`, `compress`, `decompress`, `quota_key`, and `stacktrace` (in addition to the existing `wait_end_of_query`, `default_format`, `session_timeout`, and `session_check`). + +```ts +await client.query({ + query: "SELECT 1", + clickhouse_settings: { + // Buffer the entire response on the server before sending it to the client + wait_end_of_query: 1, + buffer_size: "1048576", + }, +}); +``` + +## Bug Fixes + +- (Node.js only) Fixed a race condition in `ResultSet.json()` and `ResultSet.stream()` on `JSONEachRow` (and other streamable) result sets where calling `json()` on a fast/small response could throw `Stream has been already consumed` if the underlying stream ended between internal `readableEnded` checks. The consumption guard has been hardened: the stream is now shielded through a single `consume()` path that marks the result set as consumed in the appropriate branches, after format validation, so a successful `json()` call no longer races against the stream finishing. ([#603]) + +[#603]: https://github.com/ClickHouse/clickhouse-js/pull/603 + +# 1.19.0 + +## Improvements + +- Re-exported the `ResponseHeaders` type from `@clickhouse/client` and `@clickhouse/client-web`. Previously this type was only available from `@clickhouse/client-common`; it is now part of the public re-export surface of both flavored packages, alongside the other commonly used types. This is part of an ongoing effort to make `@clickhouse/client-common` an internal-only package so downstream consumers can depend solely on `@clickhouse/client` or `@clickhouse/client-web`. ([#758]) + +[#758]: https://github.com/ClickHouse/clickhouse-js/pull/758 +[#776]: https://github.com/ClickHouse/clickhouse-js/pull/776 + +## Bug Fixes + +- **Enum type parsing now correctly unescapes backslash escape sequences in enum names.** Previously, `parseEnumType` returned enum names with raw escape sequences (e.g., `f\'` instead of `f'`). Now it properly decodes escape sequences including `\'` (single quote), `\\` (backslash), `\n` (newline), `\t` (tab), and `\r` (carriage return). This matches the behavior of ClickHouse string literals and ensures consistency with how the client encodes strings when sending data to the server. If you were relying on the previous incorrect behavior where backslash escape sequences were preserved in enum names, you will need to update your code to handle properly unescaped values. + +Example: + +```ts +// Before (incorrect): +parseEnumType({ + columnType: "Enum8('f\\'' = 1)", + sourceType: "Enum8('f\\'' = 1)", +}); +// returned: { values: { 1: "f\\'" } } // with backslash + +// After (correct): +parseEnumType({ + columnType: "Enum8('f\\'' = 1)", + sourceType: "Enum8('f\\'' = 1)", +}); +// returns: { values: { 1: "f'" } } // unescaped +``` + +# 1.18.5 + +## Improvements + +- (Node.js only) Added `max_response_headers_size` client option that forwards the [`maxHeaderSize`](https://nodejs.org/api/http.html#httprequesturl-options-callback) option to the underlying `http(s).request` call. This raises the per-request limit on the total size of HTTP response headers received from the server (Node.js default is ~16 KB). It is most useful when running long-running queries with `send_progress_in_http_headers` enabled — the `X-ClickHouse-Progress` headers accumulate over the lifetime of the request and can exceed the default limit, causing the request to fail with `HPE_HEADER_OVERFLOW`. Setting this option avoids the need to use the global `--max-http-header-size` Node.js CLI flag or the `NODE_OPTIONS` environment variable. Has no effect for the Web client (which uses `fetch`) and no effect when a custom `http_agent` is configured with a request implementation that does not honor the option. + +```ts +const client = createClient({ + request_timeout: 400_000, + max_response_headers_size: 1024 * 1024, // accept up to 1 MiB of response headers + clickhouse_settings: { + send_progress_in_http_headers: 1, + http_headers_progress_interval_ms: "110000", + }, +}); +``` + +- The `@clickhouse/client` npm package now ships embedded AI-agent skills, `clickhouse-js-node-coding` and `clickhouse-js-node-troubleshooting`, under `node_modules/@clickhouse/client/skills/`. These skills are also declared in the `agents.skills` field of the package manifest for discovery tools that scan `node_modules`. This allows agentic coding tools to load focused, Node-client-specific coding and troubleshooting guidance without any additional setup. ([#682]) + +[#682]: https://github.com/ClickHouse/clickhouse-js/pull/682 + +# 1.18.4 + +A release-infrastructure-only version bump (no user-facing changes). See 1.18.5 for the next release with user-facing improvements. + +# 1.18.3 + +## Improvements + +- Added `keep_alive.eagerly_destroy_stale_sockets` option (Node.js only, default: `false`). When enabled, sockets that have been idle for longer than `idle_socket_ttl` are destroyed immediately before each request, rather than waiting for the idle timeout to fire. This helps reclaim stale sockets during event loop delays, where the timeout callback may not run on time. + +```ts +const client = createClient({ + keep_alive: { + enabled: true, + idle_socket_ttl: 2500, + eagerly_destroy_stale_sockets: true, + }, +}); +``` + +- Added auto-detection and warning when `request_timeout` is high (> 60 seconds) but progress headers are not configured. Long-running queries may fail with socket hang-up errors if they exceed the load balancer idle timeout. The client now warns users to enable `send_progress_in_http_headers` and `http_headers_progress_interval_ms` settings to prevent such issues. + +```ts +// This will now trigger a warning +const client = createClient({ + request_timeout: 120_000, // 120 seconds + // send_progress_in_http_headers is not configured +}); + +// ✓ Properly configured to avoid load balancer timeouts +const client = createClient({ + request_timeout: 400_000, + clickhouse_settings: { + send_progress_in_http_headers: 1, + http_headers_progress_interval_ms: "110000", // ~10s below LB timeout + }, +}); +``` + +# 1.18.2 + +## Improvements + +- Added a helping `WARN` level log message with a suggestion to check the `keep_alive` configuration if the client receives an `ECONNRESET` error from the server, which can happen when the server closes idle connections after a certain timeout, and the client tries to reuse such a connection from the pool. This can be especially helpful for new users who might not be aware of this aspect of HTTP connection management. The log message is only emitted if the `keep_alive` option is enabled in the client configuration, and it includes the server's keep-alive timeout value (if available) to assist with troubleshooting. ([#597](https://github.com/ClickHouse/clickhouse-js/pull/597)) + +How to reproduce the issue that triggers the log message: + +```ts +const client = createClient({ + // ... + keep_alive: { + enabled: true, + // ❌ DON'T SET THIS VALUE SO HIGH IN PRODUCTION + idle_socket_ttl: 1_000_000, + }, + log: { + level: ClickHouseLogLevel.WARN, // to see the warning logs + }, +}); + +for (let i = 0; i < 1000; i++) { + await client.ping({ + // To use a regular query instead of the /ping endpoint + // which might be configured differently on the server side + // and have different timeout settings. + select: true, + }); + + // Wait long enough to let the server close the idle connection, + // but not too long to let the client remove it from the pool, + // in other words try to hit the scenario when the race condition + // happens between the server closing the connection and the client + // trying to reuse it. + await sleep(SERVER_KEEP_ALIVE_TIMEOUT_MS - 100); +} +``` + +Example log message: + +```json +{ + "message": "Ping: idle socket TTL is greater than server keep-alive timeout, try setting idle socket TTL to a value lower than the server keep-alive timeout to prevent unexpected connection resets, see https://github.com/ClickHouse/clickhouse-js/blob/main/docs/howto/keep_alive_timeout.md for more details.", + "args": { + "operation": "Ping", + "connection_id": "8dc1c9bd-7895-49b1-8a95-276470151c65", + "query_id": "beee95af-2e83-4dcb-8e1e-045bd61f4985", + "request_id": "8dc1c9bd-7895-49b1-8a95-276470151c65:2", + "socket_id": "8dc1c9bd-7895-49b1-8a95-276470151c65:1", + "server_keep_alive_timeout_ms": 10000, + "idle_socket_ttl": 15000 + }, + "module": "HTTP Adapter" +} +``` + +# 1.18.1 + +## Improvements + +- Setting `log.level` default value to `ClickHouseLogLevel.WARN` instead of `ClickHouseLogLevel.OFF` to provide better visibility into potential issues without overwhelming users with too much information by default. + +```ts +const client = createClient({ + // ... + log: { + level: ClickHouseLogLevel.WARN, // default is now ClickHouseLogLevel.WARN instead of ClickHouseLogLevel.OFF + }, +}); +``` + +- Logging is now lazy, which means that the log messages will only be constructed if the log level is appropriate for the message. This can improve performance in cases where constructing the log message is expensive, and the log level is set to ignore such messages. See `ClickHouseLogLevel` enum for the complete list of log levels. ([#520]) + +```ts +const client = createClient({ + // ... + log: { + level: ClickHouseLogLevel.TRACE, // to log everything available down to the network level events + }, +}); +``` + +- Enhanced the logging of the HTTP request / socket lifecycle with additional trace messages and context such as Connection ID (UUID) and Request ID and Socket ID that embed the connection ID for ease of tracing the logs of a particular request across the connection lifecycle. To enable such logs, set the `log.level` config option to `ClickHouseLogLevel.TRACE`. ([#567]) + +```console +[2026-02-25T09:19:13.511Z][TRACE][@clickhouse/client][Connection] Insert: received 'close' event, 'free' listener removed +Arguments: { + operation: 'Insert', + connection_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c', + query_id: '9dfda627-39a2-41a6-9fc9-8f8716574826', + request_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:3', + socket_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:2', + event: 'close' +} +[2026-02-25T09:19:13.502Z][TRACE][@clickhouse/client][Connection] Query: reusing socket +Arguments: { + operation: 'Query', + connection_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c', + query_id: 'ad0127e8-b1c7-4ed6-9681-c0162f7a0ea9', + request_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:4', + socket_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:2', + usage_count: 1 +} +``` + +- A step towards structured logging: the client now passes rich context to the logger `args` parameter (e.g. `connection_id`, `query_id`, `request_id`, `socket_id`). ([#576]) + +## Deprecated API + +- The `drainStream` utility function is now deprecated, as the client will handle draining the stream internally when needed. Use `client.command()` instead, which will handle draining the stream internally when needed. ([#578]) + +- The `sleep` utility function is now deprecated, as it is not intended to be used outside of the client implementation. Use `setTimeout` directly or a more full-featured utility library if you need additional features like cancellation or timers management. ([#578]) + +[#520]: https://github.com/ClickHouse/clickhouse-js/pull/520 +[#567]: https://github.com/ClickHouse/clickhouse-js/pull/567 +[#576]: https://github.com/ClickHouse/clickhouse-js/pull/576 +[#578]: https://github.com/ClickHouse/clickhouse-js/pull/578 + +# 1.18.0 + +A beta version. See 1.18.1 for the stable release. + +# 1.17.0 + +## New features + +- Added `http_status_code` to query, insert, and exec commands ([#525], [Kinzeng]) +- Fixed `ignore_error_response` not getting passed when using `command` ([#536], [Kinzeng]) + +[#525]: https://github.com/ClickHouse/clickhouse-js/pull/525 +[#536]: https://github.com/ClickHouse/clickhouse-js/pull/536 + +# 1.16.0 + +## New features + +- Added support for the new [Disposable API] (a.k.a the `using` keyword) (#500) + +[Disposable API]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/using + +```ts +async function main() { + using resultSet = await client.query(…); + + // some code that can throw + // but thanks to `using` the resultSet will still get disposed + + // resultSet is also automatically disposed here by calling [Symbol.dispose] +} +``` + +Without the new `using` keyword it is required to wrap the code that might leak expensive resources like sockets and big buffers in ` try / finally` + +```ts +async function main() { + let client + try { + client = await createClient(…); + // some code that can throw + } finally { + if (client) { + await client.close() + } + } +} +``` + +# 1.15.0 + +## New features + +- Added support for [BigInt] values in query parameters. ([#487], @dalechyn) + +[#487]: https://github.com/ClickHouse/clickhouse-js/pull/487 +[BigInt]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt + +# 1.14.0 + +## New features + +- It is now possible to specify custom `parse` and `stringify` functions that will be used instead of the standard `JSON.parse` and `JSON.stringify` methods for JSON serialization/deserialization when working with `JSON*` family formats. See `ClickHouseClientConfigOptions.json`, and a new [custom_json_handling] example for more details. ([#481], [looskie]) +- (Node.js only) Added an `ignore_error_response` param to `ClickHouseClient.exec`, which allows callers to manually handle request errors on the application side. ([#483], [Kinzeng]) + +[#481]: https://github.com/ClickHouse/clickhouse-js/pull/481 +[#483]: https://github.com/ClickHouse/clickhouse-js/pull/483 +[looskie]: https://github.com/looskie +[Kinzeng]: https://github.com/Kinzeng +[custom_json_handling]: https://github.com/ClickHouse/clickhouse-js/blob/1.14.0/examples/custom_json_handling.ts + +# 1.13.0 + +## New features + +- Server-side exceptions that occur in the middle of the HTTP stream are now handled correctly. This requires [ClickHouse 25.11+](https://github.com/ClickHouse/ClickHouse/pull/88818). Previous ClickHouse versions are unaffected by this change. ([#478]) + +## Improvements + +- `TupleParam` constructor now accepts a readonly array to permit more usages. ([#465], [Malien]) + +## Bug fixes + +- Fixed boolean value formatting in query parameters. Boolean values within `Array`, `Tuple`, and `Map` types are now correctly formatted as `TRUE`/`FALSE` instead of `1`/`0` to ensure proper type compatibility with ClickHouse. ([#475], [baseballyama]) + +[#465]: https://github.com/ClickHouse/clickhouse-js/pull/465 +[#475]: https://github.com/ClickHouse/clickhouse-js/pull/475 +[#478]: https://github.com/ClickHouse/clickhouse-js/pull/478 +[Malien]: https://github.com/Malien +[baseballyama]: https://github.com/baseballyama + +# 1.12.1 + +## Improvements + +- Improved performance of `toSearchParams`. ([#449], [twk]) + +## Other + +- Added Node.js 24.x to the CI matrix. Node.js 18.x was removed from the CI due to [EOL](https://endoflife.date/nodejs). + +[#449]: https://github.com/ClickHouse/clickhouse-js/pull/449 +[twk]: https://github.com/twk + +# 1.12.0 + +## Types + +- Add missing `allow_experimental_join_condition` to `ClickHouseSettings` typing. ([#430], [looskie]) +- Fixed `JSONEachRowWithProgress` TypeScript flow after the breaking changes in [ClickHouse 25.1]. `RowOrProgress` now has an additional variant: `SpecialEventRow`. The library now additionally exports the `parseError` method, and newly added `isRow` / `isException` type guards. See the updated [JSONEachRowWithProgress example] ([#443]) +- Added missing `allow_experimental_variant_type` (24.1+), `allow_experimental_dynamic_type` (24.5+), `allow_experimental_json_type` (24.8+), `enable_json_type` (25.3+), `enable_time_time64_type` (25.6+) to `ClickHouseSettings` typing. ([#445]) + +## Improvements + +- Add a warning on a socket closed without fully consuming the stream (e.g., when using `query` or `exec` method). ([#441]) +- (Node.js only) An option to use a simple SELECT query for ping checks instead of `/ping` endpoint. See the new optional argument to the `ClickHouseClient.ping` method and `PingParams` typings. Note that the Web version always used a SELECT query by default, as the `/ping` endpoint does not support CORS, and that cannot be changed. ([#442]) + +## Other + +- The project now uses [Codecov] instead of SonarCloud for code coverage reports. ([#444]) + +[#430]: https://github.com/ClickHouse/clickhouse-js/pull/430 +[#441]: https://github.com/ClickHouse/clickhouse-js/pull/441 +[#442]: https://github.com/ClickHouse/clickhouse-js/pull/442 +[#443]: https://github.com/ClickHouse/clickhouse-js/pull/443 +[#444]: https://github.com/ClickHouse/clickhouse-js/pull/444 +[#445]: https://github.com/ClickHouse/clickhouse-js/pull/445 +[looskie]: https://github.com/looskie +[ClickHouse 25.1]: https://github.com/ClickHouse/ClickHouse/pull/74181 +[JSONEachRowWithProgress example]: https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/select_json_each_row_with_progress.ts +[Codecov]: https://codecov.io/gh/ClickHouse/clickhouse-js + +# 1.11.2 (Common, Node.js) + +A minor release to allow further investigation regarding uncaught error issues with [#410]. + +## Types + +- Added missing `lightweight_deletes_sync` typing to `ClickHouseSettings` ([#422], [pratimapatel2008]) + +## Improvements (Node.js) + +- Added a new configuration option: `capture_enhanced_stack_trace`; see the JS doc in the Node.js client package. Note that it is disabled by default due to a possible performance impact. ([#427]) +- Added more try-catch blocks to the Node.js connection layer. ([#427]) + +[#410]: https://github.com/ClickHouse/clickhouse-js/pull/410 +[#422]: https://github.com/ClickHouse/clickhouse-js/pull/422 +[#427]: https://github.com/ClickHouse/clickhouse-js/pull/427 +[pratimapatel2008]: https://github.com/pratimapatel2008 + +# 1.11.1 (Common, Node.js, Web) + +## Bug fixes + +- Fixed an issue with URLEncoded special characters in the URL configuration for username or password. ([#407](https://github.com/ClickHouse/clickhouse-js/issues/407)) + +## Improvements + +- Added support for streaming on 32-bit platforms. ([#403](https://github.com/ClickHouse/clickhouse-js/pull/403), [shevchenkonik](https://github.com/shevchenkonik)) + +# 1.11.0 (Common, Node.js, Web) + +## New features + +- It is now possible to provide custom HTTP headers when calling the `query`/`insert`/`command`/`exec` methods using the `http_headers` option. NB: `http_headers` specified this way will override `http_headers` set on the client instance level. ([#394](https://github.com/ClickHouse/clickhouse-js/issues/374), [@DylanRJohnston](https://github.com/DylanRJohnston)) +- (Web only) It is now possible to provide a custom `fetch` implementation to the client. ([#315](https://github.com/ClickHouse/clickhouse-js/issues/315), [@lucacasonato](https://github.com/lucacasonato)) + +# 1.10.1 (Common, Node.js, Web) + +## Bug fixes + +- Fixed `NULL` parameter binding with `Tuple`, `Array`, and `Map` types. ([#374](https://github.com/ClickHouse/clickhouse-js/issues/374)) + +## Improvements + +- `ClickHouseSettings` typings now include `session_timeout` and `session_check` settings. ([#370](https://github.com/ClickHouse/clickhouse-js/issues/370)) + +# 1.10.0 (Common, Node.js, Web) + +## New features + +- Added support for JWT authentication (ClickHouse Cloud feature) in both Node.js and Web API packages. JWT token can be set via `access_token` client configuration option. + + ```ts + const client = createClient({ + // ... + access_token: "", + }); + ``` + + Access token can also be configured via the URL params, e.g., `https://host:port?access_token=...`. + + It is also possible to override the access token for a particular request (see `BaseQueryParams.auth` for more details). + + NB: do not mix access token and username/password credentials in the configuration; the client will throw an error if both are set. + +# 1.9.1 (Node.js only) + +## Bug fixes + +- Fixed an uncaught exception that could happen in case of malformed ClickHouse response when response compression is enabled ([#363](https://github.com/ClickHouse/clickhouse-js/issues/363)) + +# 1.9.0 (Common, Node.js, Web) + +## New features + +- Added `input_format_json_throw_on_bad_escape_sequence` to the `ClickhouseSettings` type. ([#355](https://github.com/ClickHouse/clickhouse-js/pull/355), [@emmanuel-bonin](https://github.com/emmanuel-bonin)) +- The client now exports `TupleParam` wrapper class, allowing tuples to be properly used as query parameters. Added support for JS Map as a query parameter. ([#359](https://github.com/ClickHouse/clickhouse-js/pull/359)) + +## Improvements + +- The client will throw a more informative error if the buffered response is larger than the max allowed string length in V8, which is `2**29 - 24` bytes. ([#357](https://github.com/ClickHouse/clickhouse-js/pull/357)) + +# 1.8.1 (Node.js) + +## Bug fixes + +- When a custom HTTP agent is used, the HTTP or HTTPS request implementation is now correctly chosen based on the URL protocol. ([#352](https://github.com/ClickHouse/clickhouse-js/issues/352)) + +# 1.8.0 (Common, Node.js, Web) + +## New features + +- Added support for specifying roles via request query parameters. See [this example](examples/role.ts) for more details. ([@pulpdrew](https://github.com/pulpdrew), [#328](https://github.com/ClickHouse/clickhouse-js/pull/328)) + +# 1.7.0 (Common, Node.js, Web) + +## Bug fixes + +- (Web only) Fixed an issue where streaming large datasets could provide corrupted results. See [#333](https://github.com/ClickHouse/clickhouse-js/pull/333) (PR) for more details. + +## New features + +- Added `JSONEachRowWithProgress` format support, `ProgressRow` interface, and `isProgressRow` type guard. See [this Node.js example](../../examples/node/select_json_each_row_with_progress.ts) for more details. It should work similarly with the Web version. +- (Experimental) Exposed the `parseColumnType` function that takes a string representation of a ClickHouse type (e.g., `FixedString(16)`, `Nullable(Int32)`, etc.) and returns an AST-like object that represents the type. For example: + + ```ts + for (const type of [ + "Int32", + "Array(Nullable(String))", + `Map(Int32, DateTime64(9, 'UTC'))`, + ]) { + console.log(`##### Source ClickHouse type: ${type}`); + console.log(parseColumnType(type)); + } + ``` + + The above code will output: + + ``` + ##### Source ClickHouse type: Int32 + { type: 'Simple', columnType: 'Int32', sourceType: 'Int32' } + ##### Source ClickHouse type: Array(Nullable(String)) + { + type: 'Array', + value: { + type: 'Nullable', + sourceType: 'Nullable(String)', + value: { type: 'Simple', columnType: 'String', sourceType: 'String' } + }, + dimensions: 1, + sourceType: 'Array(Nullable(String))' + } + ##### Source ClickHouse type: Map(Int32, DateTime64(9, 'UTC')) + { + type: 'Map', + key: { type: 'Simple', columnType: 'Int32', sourceType: 'Int32' }, + value: { + type: 'DateTime64', + timezone: 'UTC', + precision: 9, + sourceType: "DateTime64(9, 'UTC')" + }, + sourceType: "Map(Int32, DateTime64(9, 'UTC'))" + } + ``` + + While the original intention was to use this function internally for `Native`/`RowBinaryWithNamesAndTypes` data formats headers parsing, it can be useful for other purposes as well (e.g., interfaces generation, or custom JSON serializers). + + NB: currently unsupported source types to parse: + - Geo + - (Simple)AggregateFunction + - Nested + - Old/new experimental JSON + - Dynamic + - Variant + +# 1.6.0 (Common, Node.js, Web) + +## New features + +- Added optional `real_time_microseconds` field to the `ClickHouseSummary` interface (see ) + +## Bug fixes + +- Fixed unhandled exceptions produced when calling `ResultSet.json` if the response data was not in fact a valid JSON. ([#311](https://github.com/ClickHouse/clickhouse-js/pull/311)) + +# 1.5.0 (Node.js) + +## New features + +- It is now possible to disable the automatic decompression of the response stream with the `exec` method. See `ExecParams.decompress_response_stream` for more details. ([#298](https://github.com/ClickHouse/clickhouse-js/issues/298)). + +# 1.4.1 (Node.js, Web) + +## Improvements + +- `ClickHouseClient` is now exported as a value from `@clickhouse/client` and `@clickhouse/client-web` packages, allowing for better integration in dependency injection frameworks that rely on IoC (e.g., [Nest.js](https://github.com/nestjs/nest), [tsyringe](https://github.com/microsoft/tsyringe)) ([@mathieu-bour](https://github.com/mathieu-bour), [#292](https://github.com/ClickHouse/clickhouse-js/issues/292)). + +## Bug fixes + +- Fixed a potential socket hang up issue that could happen under 100% CPU load ([#294](https://github.com/ClickHouse/clickhouse-js/issues/294)). + +# 1.4.0 (Node.js) + +## New features + +- (Node.js only) The `exec` method now accepts an optional `values` parameter, which allows you to pass the request body as a `Stream.Readable`. This can be useful in case of custom insert streaming with arbitrary ClickHouse data formats (which might not be explicitly supported and allowed by the client in the `insert` method yet). NB: in this case, you are expected to serialize the data in the stream in the required input format yourself. + +# 1.3.0 (Common, Node.js, Web) + +## New features + +- It is now possible to get the entire response headers object from the `query`/`insert`/`command`/`exec` methods. With `query`, you can access the `ResultSet.response_headers` property; other methods (`insert`/`command`/`exec`) return it as parts of their response objects as well. + For example: + + ```ts + const rs = await client.query({ + query: "SELECT * FROM system.numbers LIMIT 1", + format: "JSONEachRow", + }); + console.log(rs.response_headers["content-type"]); + ``` + + This will print: `application/x-ndjson; charset=UTF-8`. It can be used in a similar way with the other methods. + +## Improvements + +- Re-exported several constants from the `@clickhouse/client-common` package for convenience: + - `SupportedJSONFormats` + - `SupportedRawFormats` + - `StreamableFormats` + - `StreamableJSONFormats` + - `SingleDocumentJSONFormats` + - `RecordsJSONFormats` + +# 1.2.0 (Node.js) + +## New features + +- (Experimental) Added an option to provide a custom HTTP Agent in the client configuration via the `http_agent` option ([#283](https://github.com/ClickHouse/clickhouse-js/issues/283), related: [#278](https://github.com/ClickHouse/clickhouse-js/issues/278)). The following conditions apply if a custom HTTP Agent is provided: + - The `max_open_connections` and `tls` options will have _no effect_ and will be ignored by the client, as it is a part of the underlying HTTP Agent configuration. + - `keep_alive.enabled` will only regulate the default value of the `Connection` header (`true` -> `Connection: keep-alive`, `false` -> `Connection: close`). + - While the idle socket management will still work, it is now possible to disable it completely by setting the `keep_alive.idle_socket_ttl` value to `0`. +- (Experimental) Added a new client configuration option: `set_basic_auth_header`, which disables the `Authorization` header that is set by the client by default for every outgoing HTTP request. One of the possible scenarios when it is necessary to disable this header is when a custom HTTPS agent is used, and the server requires TLS authorization. For example: + + ```ts + const agent = new https.Agent({ + ca: fs.readFileSync("./ca.crt"), + }); + const client = createClient({ + url: "https://server.clickhouseconnect.test:8443", + http_agent: agent, + // With a custom HTTPS agent, the client won't use the default HTTPS connection implementation; the headers should be provided manually + http_headers: { + "X-ClickHouse-User": "default", + "X-ClickHouse-Key": "", + }, + // Authorization header conflicts with the TLS headers; disable it. + set_basic_auth_header: false, + }); + ``` + +NB: It is currently not possible to set the `set_basic_auth_header` option via the URL params. + +If you have feedback on these experimental features, please let us know by creating [an issue](https://github.com/ClickHouse/clickhouse-js/issues) in the repository. + +# 1.1.0 (Common, Node.js, Web) + +## New features + +- Added an option to override the credentials for a particular `query`/`command`/`exec`/`insert` request via the `BaseQueryParams.auth` setting; when set, the credentials will be taken from there instead of the username/password provided during the client instantiation ([#278](https://github.com/ClickHouse/clickhouse-js/issues/278)). +- Added an option to override the `session_id` for a particular `query`/`command`/`exec`/`insert` request via the `BaseQueryParams.session_id` setting; when set, it will be used instead of the session id provided during the client instantiation ([@holi0317](https://github.com/Holi0317), [#271](https://github.com/ClickHouse/clickhouse-js/issues/271)). + +## Bug fixes + +- Fixed the incorrect `ResponseJSON.totals` TypeScript type. Now it correctly matches the shape of the data (`T`, default = `unknown`) instead of the former `Record` definition ([#274](https://github.com/ClickHouse/clickhouse-js/issues/274)). + +# 1.0.2 (Common, Node.js, Web) + +## Bug fixes + +- The `command` method now drains the response stream properly, as the previous implementation could cause the `Keep-Alive` socket to close after each request. +- Removed an unnecessary error log in the `ResultSet.stream` method if the request was aborted or the result set was closed ([#263](https://github.com/ClickHouse/clickhouse-js/issues/263)). + +## Improvements + +- `ResultSet.stream` logs an error via the `Logger` instance, if the stream emits an error event instead of a simple `console.error` call. +- Minor adjustments to the `DefaultLogger` log messages formatting. +- Added missing `rows_before_limit_at_least` to the ResponseJSON type ([@0237h](https://github.com/0237h), [#267](https://github.com/ClickHouse/clickhouse-js/issues/267)). + +# 1.0.1 (Common, Node.js, Web) + +## Bug fixes + +- Fixed the regression where the default HTTP/HTTPS port numbers (80/443) could not be used with the URL configuration ([#258](https://github.com/ClickHouse/clickhouse-js/issues/258)). + +# 1.0.0 (Common, Node.js, Web) + +Formal stable release milestone with a lot of improvements and some [breaking changes](#breaking-changes-in-100). + +Major new features overview: + +- [Advanced TypeScript support for `query` + `ResultSet`](#advanced-typescript-support-for-query--resultset) +- [URL configuration](#url-configuration) + +From now on, the client will follow the [official semantic versioning](https://docs.npmjs.com/about-semantic-versioning) guidelines. + +## Deprecated API + +The following configuration parameters are marked as deprecated: + +- `host` configuration parameter is deprecated; use `url` instead. +- `additional_headers` configuration parameter is deprecated; use `http_headers` instead. + +The client will log a warning if any of these parameters are used. However, it is still allowed to use `host` instead of `url` and `additional_headers` instead of `http_headers` for now; this deprecation is not supposed to break the existing code. + +These parameters will be removed in the next major release (2.0.0). + +See "New features" section for more details. + +## Breaking changes in 1.0.0 + +- `compression.response` is now disabled by default in the client configuration options, as it cannot be used with readonly=1 users, and it was not clear from the ClickHouse error message what exact client option was causing the failing query in this case. If you'd like to continue using response compression, you should explicitly enable it in the client configuration. +- As the client now supports parsing [URL configuration](#url-configuration), you should specify `pathname` as a separate configuration option (as it would be considered as the `database` otherwise). +- (TypeScript only) `ResultSet` and `Row` are now more strictly typed, according to the format used during the `query` call. See [this section](#advanced-typescript-support-for-query--resultset) for more details. +- (TypeScript only) Both Node.js and Web versions now uniformly export correct `ClickHouseClient` and `ClickHouseClientConfigOptions` types, specific to each implementation. Exported `ClickHouseClient` now does not have a `Stream` type parameter, as it was unintended to expose it there. NB: you should still use `createClient` factory function provided in the package. + +## New features in 1.0.0 + +### Advanced TypeScript support for `query` + `ResultSet` + +Client will now try its best to figure out the shape of the data based on the DataFormat literal specified to the `query` call, as well as which methods are allowed to be called on the `ResultSet`. + +Live demo (see the full description below): + +[Screencast](https://github.com/ClickHouse/clickhouse-js/assets/3175289/b66afcb2-3a10-4411-af59-51d2754c417e) + +Complete reference: + +| Format | `ResultSet.json()` | `ResultSet.stream()` | Stream data | `Row.json()` | +| ------------------------------- | --------------------- | --------------------------- | ----------------- | --------------- | +| JSON | ResponseJSON\ | never | never | never | +| JSONObjectEachRow | Record\ | never | never | never | +| All other `JSON*EachRow` | Array\ | Stream\\>\> | Array\\> | T | +| CSV/TSV/CustomSeparated/Parquet | never | Stream\\>\> | Array\\> | never | + +By default, `T` (which represents `JSONType`) is still `unknown`. However, considering `JSONObjectsEachRow` example: prior to 1.0.0, you had to specify the entire type hint, including the shape of the data, manually: + +```ts +type Data = { foo: string }; + +const resultSet = await client.query({ + query: "SELECT * FROM my_table", + format: "JSONObjectsEachRow", +}); + +// pre-1.0.0, `resultOld` has type Record +const resultOld = resultSet.json>(); +// const resultOld = resultSet.json() // incorrect! The type hint should've been `Record` here. + +// 1.0.0, `resultNew` also has type Record; client inferred that it has to be a Record from the format literal. +const resultNew = resultSet.json(); +``` + +This is even more handy in case of streaming on the Node.js platform: + +```ts +const resultSet = await client.query({ + query: "SELECT * FROM my_table", + format: "JSONEachRow", +}); + +// pre-1.0.0 +// `streamOld` was just a regular Node.js Stream.Readable +const streamOld = resultSet.stream(); +// `rows` were `any`, needed an explicit type hint +streamNew.on("data", (rows: Row[]) => { + rows.forEach((row) => { + // without an explicit type hint to `rows`, calling `forEach` and other array methods resulted in TS compiler errors + const t = row.text; + const j = row.json(); // `j` needed a type hint here, otherwise, it's `unknown` + }); +}); + +// 1.0.0 +// `streamNew` is now StreamReadable (Node.js Stream.Readable with a bit more type hints); +// type hint for the further `json` calls can be added here (and removed from the `json` calls) +const streamNew = resultSet.stream(); +// `rows` are inferred as an Array> instead of `any` +streamNew.on("data", (rows) => { + // `row` is inferred as Row + rows.forEach((row) => { + // no explicit type hints required, you can use `forEach` straight away and TS compiler will be happy + const t = row.text; + const j = row.json(); // `j` will be of type Data + }); +}); + +// async iterator now also has type hints +// similarly to the `on(data)` example above, `rows` are inferred as Array> +for await (const rows of streamNew) { + // `row` is inferred as Row + rows.forEach((row) => { + const t = row.text; + const j = row.json(); // `j` will be of type Data + }); +} +``` + +Calling `ResultSet.stream` is not allowed for certain data formats, such as `JSON` and `JSONObjectsEachRow` (unlike `JSONEachRow` and the rest of `JSON*EachRow`, these formats return a single object). In these cases, the client throws an error. However, it was previously not reflected on the type level; now, calling `stream` on these formats will result in a TS compiler error. For example: + +```ts +const resultSet = await client.query("SELECT * FROM table", { + format: "JSON", +}); +const stream = resultSet.stream(); // `stream` is `never` +``` + +Calling `ResultSet.json` also does not make sense on `CSV` and similar "raw" formats, and the client throws. Again, now, it is typed properly: + +```ts +const resultSet = await client.query("SELECT * FROM table", { + format: "CSV", +}); +// `json` is `never`; same if you stream CSV, and call `Row.json` - it will be `never`, too. +const json = resultSet.json(); +``` + +Currently, there is one known limitation: as the general shape of the data and the methods allowed for calling are inferred from the format literal, there might be situations where it will fail to do so, for example: + +```ts +// assuming that `queryParams` has `JSONObjectsEachRow` format inside +async function runQuery( + queryParams: QueryParams, +): Promise> { + const resultSet = await client.query(queryParams); + // type hint here will provide a union of all known shapes instead of a specific one + // inferred shapes: Data[] | ResponseJSON | Record + return resultSet.json(); +} +``` + +In this case, as it is _likely_ that you already know the desired format in advance (otherwise, returning a specific shape like `Record` would've been incorrect), consider helping the client a bit: + +```ts +async function runQuery( + queryParams: QueryParams, +): Promise> { + const resultSet = await client.query({ + ...queryParams, + format: "JSONObjectsEachRow", + }); + // TS understands that it is a Record now + return resultSet.json(); +} +``` + +If you are interested in more details, see the [related test](../../packages/client-node/__tests__/integration/node_query_format_types.test.ts) (featuring a great ESLint plugin [expect-types](https://github.com/JoshuaKGoldberg/eslint-plugin-expect-type)) in the client package. + +### URL configuration + +- Added `url` configuration parameter. It is intended to replace the deprecated `host`, which was already supposed to be passed as a valid URL. +- It is now possible to configure most of the client instance parameters with a URL. The URL format is `http[s]://[username:password@]hostname:port[/database][?param1=value1¶m2=value2]`. In almost every case, the name of a particular parameter reflects its path in the config options interface, with a few exceptions. The following parameters are supported: + +| Parameter | Type | +| ------------------------------------------- | ----------------------------------------------------------------- | +| `pathname` | an arbitrary string. | +| `application_id` | an arbitrary string. | +| `session_id` | an arbitrary string. | +| `request_timeout` | non-negative number. | +| `max_open_connections` | non-negative number, greater than zero. | +| `compression_request` | boolean. See below [1]. | +| `compression_response` | boolean. | +| `log_level` | allowed values: `OFF`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`. | +| `keep_alive_enabled` | boolean. | +| `clickhouse_setting_*` or `ch_*` | see below [2]. | +| `http_header_*` | see below [3]. | +| (Node.js only) `keep_alive_idle_socket_ttl` | non-negative number. | + +[1] For booleans, valid values will be `true`/`1` and `false`/`0`. + +[2] Any parameter prefixed with `clickhouse_setting_` or `ch_` will have this prefix removed and the rest added to client's `clickhouse_settings`. For example, `?ch_async_insert=1&ch_wait_for_async_insert=1` will be the same as: + +```ts +createClient({ + clickhouse_settings: { + async_insert: 1, + wait_for_async_insert: 1, + }, +}); +``` + +Note: boolean values for `clickhouse_settings` should be passed as `1`/`0` in the URL. + +[3] Similar to [2], but for `http_header` configuration. For example, `?http_header_x-clickhouse-auth=foobar` will be an equivalent of: + +```ts +createClient({ + http_headers: { + "x-clickhouse-auth": "foobar", + }, +}); +``` + +**Important: URL will _always_ overwrite the hardcoded values and a warning will be logged in this case.** + +Currently not supported via URL: + +- `log.LoggerClass` +- (Node.js only) `tls_ca_cert`, `tls_cert`, `tls_key`. + +See also: [URL configuration example](../../examples/url_configuration.ts). + +### Performance + +- (Node.js only) Improved performance when decoding the entire set of rows with _streamable_ JSON formats (such as `JSONEachRow` or `JSONCompactEachRow`) by calling the `ResultSet.json()` method. NB: The actual streaming performance when consuming the `ResultSet.stream()` hasn't changed. Only the `ResultSet.json()` method used a suboptimal stream processing in some instances, and now `ResultSet.json()` just consumes the same stream transformer provided by the `ResultSet.stream()` method (see [#253](https://github.com/ClickHouse/clickhouse-js/pull/253) for more details). + +### Miscellaneous + +- Added `http_headers` configuration parameter as a direct replacement for `additional_headers`. Functionally, it is the same, and the change is purely cosmetic, as we'd like to leave an option to implement TCP connection in the future open. + +## 0.3.1 (Common, Node.js, Web) + +### Bug fixes + +- Fixed an issue where query parameters containing tabs or newline characters were not encoded properly. + +## 0.3.0 (Node.js only) + +This release primarily focuses on improving the Keep-Alive mechanism's reliability on the client side. + +### New features + +- Idle sockets timeout rework; now, the client attaches internal timers to idling sockets, and forcefully removes them from the pool if it considers that a particular socket is idling for too long. The intention of this additional sockets housekeeping is to eliminate "Socket hang-up" errors that could previously still occur on certain configurations. Now, the client does not rely on KeepAlive agent when it comes to removing the idling sockets; in most cases, the server will not close the socket before the client does. +- There is a new `keep_alive.idle_socket_ttl` configuration parameter. The default value is `2500` (milliseconds), which is considered to be safe, as [ClickHouse versions prior to 23.11 had `keep_alive_timeout` set to 3 seconds by default](https://github.com/ClickHouse/ClickHouse/commit/1685cdcb89fe110b45497c7ff27ce73cc03e82d1), and `keep_alive.idle_socket_ttl` is supposed to be slightly less than that to allow the client to remove the sockets that are about to expire before the server does so. +- Logging improvements: more internal logs on failing requests; all client methods except ping will log an error on failure now. A failed ping will log a warning, since the underlying error is returned as a part of its result. Client logging still needs to be enabled explicitly by specifying the desired `log.level` config option, as the log level is `OFF` by default. Currently, the client logs the following events, depending on the selected `log.level` value: + - `TRACE` - low-level information about the Keep-Alive sockets lifecycle. + - `DEBUG` - response information (without authorization headers and host info). + - `INFO` - still mostly unused, will print the current log level when the client is initialized. + - `WARN` - non-fatal errors; failed `ping` request is logged as a warning, as the underlying error is included in the returned result. + - `ERROR` - fatal errors from `query`/`insert`/`exec`/`command` methods, such as a failed request. + +### Breaking changes + +- `keep_alive.retry_on_expired_socket` and `keep_alive.socket_ttl` configuration parameters are removed. +- The `max_open_connections` configuration parameter is now 10 by default, as we should not rely on the KeepAlive agent's defaults. +- Fixed the default `request_timeout` configuration value (now it is correctly set to `30_000`, previously `300_000` (milliseconds)). + +### Bug fixes + +- Fixed a bug with Ping that could lead to an unhandled "Socket hang-up" propagation. +- Ensure proper `Connection` header value considering Keep-Alive settings. If Keep-Alive is disabled, its value is now forced to ["close"](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Connection#close). + +## 0.3.0-beta.1 (Node.js only) + +See [0.3.0](#030-nodejs-only). + +## 0.2.10 (Common, Node.js, Web) + +### New features + +- If `InsertParams.values` is an empty array, no request is sent to the server and `ClickHouseClient.insert` short-circuits itself. In this scenario, the newly added `InsertResult.executed` flag will be `false`, and `InsertResult.query_id` will be an empty string. + +### Bug fixes + +- Client no longer produces `Code: 354. inflate failed: buffer error` exception if request compression is enabled and `InsertParams.values` is an empty array (see above). + +## 0.2.9 (Common, Node.js, Web) + +### New features + +- It is now possible to set additional HTTP headers for outgoing ClickHouse requests. This might be useful if, for example, you use a reverse proxy with authorization. ([@teawithfruit](https://github.com/teawithfruit), [#224](https://github.com/ClickHouse/clickhouse-js/pull/224)) + +```ts +const client = createClient({ + additional_headers: { + "X-ClickHouse-User": "clickhouse_user", + "X-ClickHouse-Key": "clickhouse_password", + }, +}); +``` + +## 0.2.8 (Common, Node.js, Web) + +### New features + +- (Web only) Allow to modify Keep-Alive setting (previously always disabled). + Keep-Alive setting **is now enabled by default** for the Web version. + +```ts +import { createClient } from "@clickhouse/client-web"; +const client = createClient({ keep_alive: { enabled: true } }); +``` + +- (Node.js & Web) It is now possible to either specify a list of columns to insert the data into or a list of excluded columns: + +```ts +// Generated query: INSERT INTO mytable (message) FORMAT JSONEachRow +await client.insert({ + table: "mytable", + format: "JSONEachRow", + values: [{ message: "foo" }], + columns: ["message"], +}); + +// Generated query: INSERT INTO mytable (* EXCEPT (message)) FORMAT JSONEachRow +await client.insert({ + table: "mytable", + format: "JSONEachRow", + values: [{ id: 42 }], + columns: { except: ["message"] }, +}); +``` + +See also the new examples: + +- [Including specific columns](../../examples/insert_specific_columns.ts) or [excluding certain ones instead](../../examples/insert_exclude_columns.ts) +- [Leveraging this feature](../../examples/insert_ephemeral_columns.ts) when working with + [ephemeral columns](https://clickhouse.com/docs/en/sql-reference/statements/create/table#ephemeral) + ([#217](https://github.com/ClickHouse/clickhouse-js/issues/217)) + +## 0.2.7 (Common, Node.js, Web) + +### New features + +- (Node.js only) `X-ClickHouse-Summary` response header is now parsed when working with `insert`/`exec`/`command` methods. + See the [related test](../../packages/client-node/__tests__/integration/node_summary.test.ts) for more details. + NB: it is guaranteed to be correct only for non-streaming scenarios. + Web version does not currently support this due to CORS limitations. ([#210](https://github.com/ClickHouse/clickhouse-js/issues/210)) + +### Bug fixes + +- Drain insert response stream in Web version - required to properly work with `async_insert`, especially in the Cloudflare Workers context. + +## 0.2.6 (Common, Node.js) + +### New features + +- Added [Parquet format](https://clickhouse.com/docs/en/integrations/data-formats/parquet) streaming support. + See the new examples: + [insert from a file](../../examples/node/insert_file_stream_parquet.ts), + [select into a file](../../examples/node/select_parquet_as_file.ts). + +## 0.2.5 (Common, Node.js, Web) + +### Bug fixes + +- `pathname` segment from `host` client configuration parameter is now handled properly when making requests. + See this [comment](https://github.com/ClickHouse/clickhouse-js/issues/164#issuecomment-1785166626) for more details. + +## 0.2.4 (Node.js only) + +No changes in web/common modules. + +### Bug fixes + +- (Node.js only) Fixed an issue where streaming large datasets could provide corrupted results. See [#171](https://github.com/ClickHouse/clickhouse-js/issues/171) (issue) and [#204](https://github.com/ClickHouse/clickhouse-js/pull/204) (PR) for more details. + +## 0.2.3 (Node.js only) + +No changes in web/common modules. + +### Bug fixes + +- (Node.js only) Fixed an issue where the underlying socket was closed every time after using `insert` with a `keep_alive` option enabled, which led to performance limitations. See [#202](https://github.com/ClickHouse/clickhouse-js/issues/202) for more details. ([@varrocs](https://github.com/varrocs)) + +## 0.2.2 (Common, Node.js & Web) + +### New features + +- Added `default_format` setting, which allows to perform `exec` calls without `FORMAT` clause. + +## 0.2.1 (Common, Node.js & Web) + +### Breaking changes + +Date objects in query parameters are now serialized as time-zone-agnostic Unix timestamps (NNNNNNNNNN[.NNN], optionally with millisecond-precision) instead of datetime strings without time zones (YYYY-MM-DD HH:MM:SS[.MMM]). This means the server will receive the same absolute timestamp the client sent even if the client's time zone and the database server's time zone differ. Previously, if the server used one time zone and the client used another, Date objects would be encoded in the client's time zone and decoded in the server's time zone and create a mismatch. + +For instance, if the server used UTC (GMT) and the client used PST (GMT-8), a Date object for "2023-01-01 13:00:00 **PST**" would be encoded as "2023-01-01 13:00:00.000" and decoded as "2023-01-01 13:00:00 **UTC**" (which is 2023-01-01 **05**:00:00 PST). Now, "2023-01-01 13:00:00 PST" is encoded as "1672606800000" and decoded as "2023-01-01 **21**:00:00 UTC", the same time the client sent. + +## 0.2.0 (web platform support) + +Introduces web client (using native [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) +and [WebStream](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API) APIs) +without Node.js modules in the common interfaces. No polyfills are required. + +Web client is confirmed to work with Chrome/Firefox/CloudFlare workers. + +It is now possible to implement new custom connections on top of `@clickhouse/client-common`. + +The client was refactored into three packages: + +- `@clickhouse/client-common`: all possible platform-independent code, types and interfaces +- `@clickhouse/client-web`: new web (or non-Node.js env) connection, uses native fetch. +- `@clickhouse/client`: Node.js connection as it was before. + +### Node.js client breaking changes + +- Changed `ping` method behavior: it will not throw now. + Instead, either `{ success: true }` or `{ success: false, error: Error }` is returned. +- Log level configuration parameter is now explicit instead of `CLICKHOUSE_LOG_LEVEL` environment variable. + Default is `OFF`. +- `query` return type signature changed to is `BaseResultSet` (no functional changes) +- `exec` return type signature changed to `ExecResult` (no functional changes) +- `insert` params argument type changed to `InsertParams` (no functional changes) +- Experimental `schema` module is removed + +### Web client known limitations + +- Streaming for select queries works, but it is disabled for inserts (on the type level as well). +- KeepAlive is disabled and not configurable yet. +- Request compression is disabled and configuration is ignored. Response compression works. +- No logging support yet. + +## 0.1.1 + +## New features + +- Expired socket detection on the client side when using Keep-Alive. If a potentially expired socket is detected, + and retry is enabled in the configuration, both socket and request will be immediately destroyed (before sending the data), + and the client will recreate the request. See `ClickHouseClientConfigOptions.keep_alive` for more details. Disabled by default. +- Allow disabling Keep-Alive feature entirely. +- `TRACE` log level. + +## Examples + +#### Disable Keep-Alive feature + +```ts +const client = createClient({ + keep_alive: { + enabled: false, + }, +}); +``` + +#### Retry on expired socket + +```ts +const client = createClient({ + keep_alive: { + enabled: true, + // should be slightly less than the `keep_alive_timeout` setting in server's `config.xml` + // default is 3s there, so 2500 milliseconds seems to be a safe client value in this scenario + // another example: if your configuration has `keep_alive_timeout` set to 60s, you could put 59_000 here + socket_ttl: 2500, + retry_on_expired_socket: true, + }, +}); +``` + +## 0.1.0 + +## Breaking changes + +- `connect_timeout` client setting is removed, as it was unused in the code. + +## New features + +- `command` method is introduced as an alternative to `exec`. + `command` does not expect user to consume the response stream, and it is destroyed immediately. + Essentially, this is a shortcut to `exec` that destroys the stream under the hood. + Consider using `command` instead of `exec` for DDLs and other custom commands which do not provide any valuable output. + +Example: + +```ts +// incorrect: stream is not consumed and not destroyed, request will be timed out eventually +await client.exec("CREATE TABLE foo (id String) ENGINE Memory"); + +// correct: stream does not contain any information and just destroyed +const { stream } = await client.exec( + "CREATE TABLE foo (id String) ENGINE Memory", +); +stream.destroy(); + +// correct: same as exec + stream.destroy() +await client.command("CREATE TABLE foo (id String) ENGINE Memory"); +``` + +### Bug fixes + +- Fixed delays on subsequent requests after calling `insert` that happened due to unclosed stream instance when using low number of `max_open_connections`. See [#161](https://github.com/ClickHouse/clickhouse-js/issues/161) for more details. +- Request timeouts internal logic rework (see [#168](https://github.com/ClickHouse/clickhouse-js/pull/168)) + +## 0.0.16 + +- Fix NULL parameter binding. + As HTTP interface expects `\N` instead of `'NULL'` string, it is now correctly handled for both `null` + and _explicitly_ `undefined` parameters. See the [test scenarios](https://github.com/ClickHouse/clickhouse-js/blob/f1500e188600d85ddd5ee7d2a80846071c8cf23e/__tests__/integration/select_query_binding.test.ts#L273-L303) for more details. + +## 0.0.15 + +### Bug fixes + +- Fix Node.JS 19.x/20.x timeout error (@olexiyb) + +## 0.0.14 + +### New features + +- Added support for `JSONStrings`, `JSONCompact`, `JSONCompactStrings`, `JSONColumnsWithMetadata` formats (@andrewzolotukhin). + +## 0.0.13 + +### New features + +- `query_id` can be now overridden for all main client's methods: `query`, `exec`, `insert`. + +## 0.0.12 + +### New features + +- `ResultSet.query_id` contains a unique query identifier that might be useful for retrieving query metrics from `system.query_log` +- `User-Agent` HTTP header is set according to the [language client spec](https://docs.google.com/document/d/1924Dvy79KXIhfqKpi1EBVY3133pIdoMwgCQtZ-uhEKs/edit#heading=h.ah33hoz5xei2). + For example, for client version 0.0.12 and Node.js runtime v19.0.4 on Linux platform, it will be `clickhouse-js/0.0.12 (lv:nodejs/19.0.4; os:linux)`. + If `ClickHouseClientConfigOptions.application` is set, it will be prepended to the generated `User-Agent`. + +### Breaking changes + +- `client.insert` now returns `{ query_id: string }` instead of `void` +- `client.exec` now returns `{ stream: Stream.Readable, query_id: string }` instead of just `Stream.Readable` + +## 0.0.11, 2022-12-08 + +### Breaking changes + +- `log.enabled` flag was removed from the client configuration. +- Use `CLICKHOUSE_LOG_LEVEL` environment variable instead. Possible values: `OFF`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`. + Currently, there are only debug messages, but we will log more in the future. + +For more details, see PR [#110](https://github.com/ClickHouse/clickhouse-js/pull/110) + +## 0.0.10, 2022-11-14 + +### New features + +- Remove request listeners synchronously. + [#123](https://github.com/ClickHouse/clickhouse-js/issues/123) + +## 0.0.9, 2022-10-25 + +### New features + +- Added ClickHouse session_id support. + [#121](https://github.com/ClickHouse/clickhouse-js/pull/121) + +## 0.0.8, 2022-10-18 + +### New features + +- Added SSL/TLS support (basic and mutual). + [#52](https://github.com/ClickHouse/clickhouse-js/issues/52) + +## 0.0.7, 2022-10-18 + +### Bug fixes + +- Allow semicolons in select clause. + [#116](https://github.com/ClickHouse/clickhouse-js/issues/116) + +## 0.0.6, 2022-10-07 + +### New features + +- Add JSONObjectEachRow input/output and JSON input formats. + [#113](https://github.com/ClickHouse/clickhouse-js/pull/113) + +## 0.0.5, 2022-10-04 + +### Breaking changes + +- Rows abstraction was renamed to ResultSet. +- now, every iteration over `ResultSet.stream()` yields `Row[]` instead of a single `Row`. + Please check out [an example](https://github.com/ClickHouse/clickhouse-js/blob/c86c31dada8f4845cd4e6843645177c99bc53a9d/examples/select_streaming_on_data.ts) + and [this PR](https://github.com/ClickHouse/clickhouse-js/pull/109) for more details. + These changes allowed us to significantly reduce overhead on select result set streaming. + +### New features + +- [split2](https://www.npmjs.com/package/split2) is no longer a package dependency. diff --git a/packages/client-common/package.json b/packages/client-common/package.json index 187b11982..e5893e09a 100644 --- a/packages/client-common/package.json +++ b/packages/client-common/package.json @@ -17,7 +17,8 @@ "main": "dist/index.js", "types": "dist/index.d.ts", "files": [ - "dist" + "dist", + "CHANGELOG.md" ], "scripts": { "pack": "npm pack", diff --git a/packages/client-node/CHANGELOG.md b/packages/client-node/CHANGELOG.md new file mode 100644 index 000000000..7ae3e0373 --- /dev/null +++ b/packages/client-node/CHANGELOG.md @@ -0,0 +1,1342 @@ +# 1.23.0 + +## Migration Notes + +- Node.js 26.x was added to the CI matrix, and Node.js 18.x is no longer supported. The `engines.node` floor of `@clickhouse/client` (previously `>=16`) and `@clickhouse/datatype-parser` (previously `>=18.0.0`) was raised to `>=20`. Node.js 20.x, 22.x, 24.x, and 26.x are supported and exercised in CI. + +- The `@clickhouse/client-common` package is deprecated. `@clickhouse/client` (Node.js) and `@clickhouse/client-web` (Web) no longer depend on it; the shared code is now bundled into each client package. Everything previously importable from `@clickhouse/client-common` should be imported from `@clickhouse/client` or `@clickhouse/client-web` instead. The `@clickhouse/client-common` package itself will no longer receive updates. ([#845]) + +- The `parseColumnType` function and its `SimpleColumnTypes` companion (exported from `@clickhouse/client`, `@clickhouse/client-web`, and `@clickhouse/client-common`) are deprecated and slated for removal in a future major version. They are superseded by the new standalone [`@clickhouse/datatype-parser`](https://www.npmjs.com/package/@clickhouse/datatype-parser) package (`parseDataType` plus its `Node` AST), which parses the full ClickHouse data-type grammar and emits an AST that mirrors the server's. ([#893]) + +## New features + +- (Node.js) Added a RowBinary reader library and agent skill under [`skills/clickhouse-js-node-rowbinary-parser`](../../skills/clickhouse-js-node-rowbinary-parser). It ships type-specific, monomorphizable building blocks for decoding `RowBinary` / `RowBinaryWithNames` / `RowBinaryWithNamesAndTypes` streams (full-buffer and chunked), plus a skill that guides an agent to generate bespoke high-performance parsers from a query's column types. The skill is bundled into `@clickhouse/client` (registered in `agents.skills`) and is also published independently as the [`@clickhouse/rowbinary`](https://www.npmjs.com/package/@clickhouse/rowbinary) package. A matching RowBinary writer is planned. ([#864]) + +- Published the [`@clickhouse/datatype-parser`](https://www.npmjs.com/package/@clickhouse/datatype-parser) package: a small, dependency-free standalone parser for ClickHouse data-type strings (the kind sent in the types row of `RowBinaryWithNamesAndTypes`, e.g. `Array(Nullable(UInt64))`, `Tuple(a UInt8, b String)`, `Enum8('a' = 1)`). It is a faithful port of the server's `ParserDataType` and emits a JSON AST that is byte-identical to the server's `EXPLAIN AST json = 1` data-type subtree. It supersedes the deprecated `parseColumnType` (see Migration Notes). ([#893]) + +- (Node.js, `@experimental`) Added an additive `connection?: Connection` option to `createClient` that lets a caller plug an externally-built backend `Connection`-like object in place of the default HTTP(S) factory. Only supposed to be used for testing the `chDB` integration. ([#879]) + +- Added `ClickHouseSettingsInterface`, a package-neutral structural counterpart to `ClickHouseSettings`, exported from `@clickhouse/client`, `@clickhouse/client-web`, and `@clickhouse/client-common`. It is identical to `ClickHouseSettings` except that its index signature omits `SettingsMap` (a class with a private member, which TypeScript compares nominally). Because each client package now bundles its own copy of the common module, their `ClickHouseSettings` types are mutually unassignable; `ClickHouseSettingsInterface` is structurally identical across all three packages and assignable into each package's `ClickHouseSettings`, so a consumer that shares a single settings-producing helper across both the Node.js and Web clients can type it against this one type without casts. Values typed as `SettingsMap` cannot be carried through it — use `ClickHouseSettings` if you need them. ([#889]) + +# 1.22.0 + +## New features + +- (Node.js) The `compression.request` / `compression.response` client options now accept an explicit codec via an object, in addition to the existing boolean: `true` keeps gzip (backwards compatible), and `{ codec: "zstd" }` selects zstd. The object form is intentionally extensible for future codecs and codec-specific options. zstd typically yields a similar-or-better ratio than gzip at noticeably lower CPU cost (gzip/DEFLATE is comparatively CPU-heavy and decompressed single-threaded by the ClickHouse server), and it uses the built-in `zlib` zstd support, so it requires **Node.js >= 22.15.0** (`@clickhouse/client` throws a clear error at client creation otherwise). Response decompression is driven by the server's actual `Content-Encoding`, so it degrades gracefully. The request object form also accepts an optional `level` (`{ codec, level }`) to set the codec-specific compression level (zlib level for gzip, zstd compression level for zstd); the response compression level is controlled by the server. Supported only by `@clickhouse/client` (Node.js); `@clickhouse/client-web` rejects the `zstd` codec at client creation. + +- (Node.js) Brotli (`{ codec: "br" }`) is now supported for `compression.request` / `compression.response`, alongside gzip and zstd. Unlike zstd, Brotli is available on every supported Node.js version (no minimum-version requirement). The `compression.request` option is a per-codec discriminated union, so each codec exposes its own tuning option: a `level` for gzip/zstd, a `quality` for Brotli (`{ codec: "br", quality }`). When omitted, Brotli defaults to quality 4 for request bodies, since zlib's brotli default of 11 (max) is far too slow for a streaming insert path. Response decompression follows the server's `Content-Encoding`. Supported only by `@clickhouse/client` (Node.js). + +## Internal changes (`@clickhouse/client-common`) + +> These only affect code that imports the low-level connection primitives from the deprecated `@clickhouse/client-common` package directly (e.g. a custom `Connection` implementation). The `createClient` `compression` option is unchanged and fully backwards compatible — if you only use `@clickhouse/client` or `@clickhouse/client-web`, you are not affected. + +To carry the codec (and its optional compression level) instead of a bare on/off flag, the internal compression representation changed shape: + +- `CompressionSettings.compress_request` / `decompress_response` are no longer `boolean`. They are now a normalized codec object or `undefined` (disabled): `{ codec: "gzip" | "zstd"; level?: number } | { codec: "br"; quality?: number }` for the request, `{ codec: "gzip" | "zstd" | "br" }` for the response (response compression options are chosen by the server). `getConnectionParams` normalizes the public request option into this form (`true` → `{ codec: "gzip" }`). +- `withCompressionHeaders` now takes `request_compression_codec` / `response_compression_codec` (a `CompressionMethod | undefined`) instead of the boolean `enable_request_compression` / `enable_response_compression`; the codec value is also the `Content-Encoding` / `Accept-Encoding` it emits. +- `withHttpSettings` now takes the response codec object (`{ codec } | undefined`) instead of a `boolean`. +- New exported types: `CompressionMethod`, `RequestCompression`, `ResponseCompression`. + +Why: a single `boolean` could not express which codec to use or its level, and a separate level field on `CompressionSettings` would have mixed a codec-specific option into the shared type. Discriminating by codec keeps each codec's options on the codec it belongs to. + +## Documentation + +- Added two **tracer adapter recipes** to [`docs/howto/tracing.md`](../../docs/howto/tracing.md) and [`examples/node/coding/otel_tracing.ts`](../../examples/node/coding/otel_tracing.ts), demonstrating how common OpenTelemetry auto-instrumentation options compose as thin userland wrappers around the `tracer` API instead of being baked into the client: `requireParentSpan` (skip ClickHouse spans when there is no active parent span — e.g. background health checks) and suppressing the duplicate nested HTTP spans emitted by `@opentelemetry/instrumentation-http` (via `suppressTracing` from `@opentelemetry/core`). + +# 1.21.0 + +## New features + +- The tracer API (unreleased, introduced in [#776]) now follows the [OpenTelemetry database semantic conventions](https://opentelemetry.io/docs/specs/semconv/db/sql/) and matches the attribute vocabulary of the Rust client ([clickhouse-rs](https://github.com/ClickHouse/clickhouse-rs)); see [`docs/howto/tracing.md`](../../docs/howto/tracing.md) for the documentation. In particular ([#828]): + - Spans now carry `db.system.name` (instead of `db.system`), `server.address` + `server.port` (instead of a combined `host:port`), `clickhouse.request.query_id` / `clickhouse.request.session_id` (instead of `clickhouse.query_id` / `clickhouse.session_id`), `clickhouse.response.format` on `query` and `clickhouse.request.format` on `insert` (instead of `clickhouse.format`), and `db.operation.name` + `db.collection.name` on `insert` (instead of `clickhouse.table`). + - The span status is left unset on success (per the OTEL spec recommendation for client spans, previously set to `OK`); on failure, the span gets the `error.type` attribute (the error class name) and, for server-side errors, `clickhouse.error.code` (the numeric ClickHouse error code). + - Spans record response-side attributes: `db.response.status_code` (HTTP status) and, when the `X-ClickHouse-Summary` header is available, `clickhouse.summary.*` counters (`read_rows`, `written_rows`, etc.). + - `query()` now emits two spans: `clickhouse.query` covers the HTTP request lifetime and ends as soon as the response headers are received; a child `clickhouse.query.stream` span is handed to the `ResultSet` and tracks the stream consumption, ending when the response is fully read, closed, or fails - with the final `clickhouse.response.decoded_bytes` and (for row-streaming) `db.response.returned_rows` metrics. This separation makes it easy to distinguish the original request duration from a stream that may never end (e.g. tailing a live table). + - Fixed a span leak in the Web `ResultSet.stream()` path: if the underlying fetch response stream was aborted (e.g. due to a network error), the `clickhouse.query.stream` span was never ended. The TransformStream now handles both source-stream aborts and consumer-side cancellations via a `cancel` callback. + - The `insert` span records `clickhouse.request.sent_rows` for array-based inserts. + +- Added a `use_multipart_params_auto` client option (default: `false`). When enabled, `query()` automatically sends `query_params` as `multipart/form-data` body parts (the same mechanism as `use_multipart_params`) once their URL-encoded length exceeds 4096 characters, avoiding HTTP 414/400 errors from HTTP intermediaries (nginx, AWS ALB, CloudFront) caused by over-long URLs - for example, a large `IN` list or a high-dimensional vector embedding. Smaller parameter payloads remain in the URL query string, so existing behavior is unchanged unless the threshold is crossed. `use_multipart_params: true` still forces multipart for all queries regardless of size. This does not change the server's per-value size limit, which is governed by `http_max_field_value_size`. Supported on both `@clickhouse/client` and `@clickhouse/client-web`, and overridable per request via `use_multipart_params_auto` on `query()`. Ported from [clickhouse-connect#789](https://github.com/ClickHouse/clickhouse-connect/pull/789). ([#827]) + +```ts +const client = createClient({ use_multipart_params_auto: true }); + +await client.query({ + query: "SELECT * FROM events WHERE id IN {ids:Array(UInt64)}", + // Sent in the URL when small, auto-promoted to the multipart body when large + query_params: { ids: veryLargeArrayOfIds }, +}); +``` + +- Added a `use_multipart_params` client option (default: `false`). When enabled, `query()` sends `query_params` as `multipart/form-data` body parts (with the SQL moved into a `query` part) instead of URL query-string entries, avoiding HTTP 400 errors caused by over-long URLs when parameters contain large arrays (25K+ values). All other URL search params (database, query_id, settings, session_id, role) remain in the URL. Supported on both `@clickhouse/client` and `@clickhouse/client-web`, and overridable per request via `use_multipart_params` on `query()`. ([#825]) + +```ts +const client = createClient({ use_multipart_params: true }); + +await client.query({ + query: "SELECT * FROM events WHERE id IN {ids:Array(UInt64)}", + query_params: { ids: veryLargeArrayOfIds }, + // Per-request override is also supported: + // use_multipart_params: false, +}); +``` + +[#776]: https://github.com/ClickHouse/clickhouse-js/pull/776 +[#825]: https://github.com/ClickHouse/clickhouse-js/pull/825 +[#827]: https://github.com/ClickHouse/clickhouse-js/pull/827 +[#828]: https://github.com/ClickHouse/clickhouse-js/pull/828 +[#845]: https://github.com/ClickHouse/clickhouse-js/pull/845 +[#864]: https://github.com/ClickHouse/clickhouse-js/pull/864 +[#889]: https://github.com/ClickHouse/clickhouse-js/pull/889 +[#893]: https://github.com/ClickHouse/clickhouse-js/pull/893 + +## Bug Fixes + +- The client now checks the `X-ClickHouse-Exception-Code` response header to detect server errors even when the HTTP status code indicates success. In some scenarios (for example, when an exception occurs while streaming the response progress in headers, or with certain proxy setups), ClickHouse responds with HTTP 200 but sets the `X-ClickHouse-Exception-Code` header. Previously, such responses were treated as successful, and the exception text could surface as malformed response data; now the request is rejected with a parsed `ClickHouseError` (with the proper `code` and `type`), consistent with non-2xx error responses. This applies to both the Node.js and Web clients. ([#554], supersedes [#350], related issue: [#332]) + +[#554]: https://github.com/ClickHouse/clickhouse-js/pull/554 +[#350]: https://github.com/ClickHouse/clickhouse-js/pull/350 +[#332]: https://github.com/ClickHouse/clickhouse-js/issues/332 + +# 1.20.0 + +## New Features + +- Added an optional **tracer API** that the user can pass through the client config (`tracer`) and that gets called around key lifecycle operations (`query`, `command`, `exec`, `insert`, `ping`). The `ClickHouseTracer` interface is a structural subset of the OpenTelemetry `Tracer`/`Span` APIs, so a raw OTEL tracer (`trace.getTracer(...)`) can be passed to the client as-is - but the client itself ships no tracing dependency. Each operation runs inside `tracer.startActiveSpan(...)`, so auto-instrumented child spans nest under the ClickHouse operation spans; for OpenTelemetry, this requires the `AsyncLocalStorageContextManager` to be registered (the default in the OpenTelemetry Node.js SDK). Tracer exceptions are NOT caught, so a broken tracer will break client operations. See [`docs/howto/tracing.md`](../../docs/howto/tracing.md) for the full surface description, and [`examples/node/coding/otel_tracing.ts`](../../examples/node/coding/otel_tracing.ts) for a runnable Node.js example. ([#776]) + +```ts +import { createClient } from "@clickhouse/client"; +import { trace } from "@opentelemetry/api"; + +// a raw OpenTelemetry tracer is structurally compatible - no adapter needed +const client = createClient({ + url: "http://localhost:8123", + tracer: trace.getTracer("@clickhouse/client"), +}); +``` + +## Migration Notes + +- TypeScript: `ClickHouseLogLevel` is now exported as a literal numeric union type (`0 | 1 | 2 | 3 | 4 | 127`) instead of a TypeScript `enum` type. If you were assigning arbitrary `number` values to `ClickHouseLogLevel`, you may need to narrow/cast those values during migration. + +## Improvements + +- Added TypeScript typings for the remaining HTTP-specific ClickHouse settings, so they are now suggested by autocomplete when used in `clickhouse_settings`: `buffer_size`, `compress`, `decompress`, `quota_key`, and `stacktrace` (in addition to the existing `wait_end_of_query`, `default_format`, `session_timeout`, and `session_check`). + +```ts +await client.query({ + query: "SELECT 1", + clickhouse_settings: { + // Buffer the entire response on the server before sending it to the client + wait_end_of_query: 1, + buffer_size: "1048576", + }, +}); +``` + +## Bug Fixes + +- (Node.js only) Fixed a race condition in `ResultSet.json()` and `ResultSet.stream()` on `JSONEachRow` (and other streamable) result sets where calling `json()` on a fast/small response could throw `Stream has been already consumed` if the underlying stream ended between internal `readableEnded` checks. The consumption guard has been hardened: the stream is now shielded through a single `consume()` path that marks the result set as consumed in the appropriate branches, after format validation, so a successful `json()` call no longer races against the stream finishing. ([#603]) + +[#603]: https://github.com/ClickHouse/clickhouse-js/pull/603 + +# 1.19.0 + +## Improvements + +- Re-exported the `ResponseHeaders` type from `@clickhouse/client` and `@clickhouse/client-web`. Previously this type was only available from `@clickhouse/client-common`; it is now part of the public re-export surface of both flavored packages, alongside the other commonly used types. This is part of an ongoing effort to make `@clickhouse/client-common` an internal-only package so downstream consumers can depend solely on `@clickhouse/client` or `@clickhouse/client-web`. ([#758]) + +[#758]: https://github.com/ClickHouse/clickhouse-js/pull/758 +[#776]: https://github.com/ClickHouse/clickhouse-js/pull/776 + +## Bug Fixes + +- **Enum type parsing now correctly unescapes backslash escape sequences in enum names.** Previously, `parseEnumType` returned enum names with raw escape sequences (e.g., `f\'` instead of `f'`). Now it properly decodes escape sequences including `\'` (single quote), `\\` (backslash), `\n` (newline), `\t` (tab), and `\r` (carriage return). This matches the behavior of ClickHouse string literals and ensures consistency with how the client encodes strings when sending data to the server. If you were relying on the previous incorrect behavior where backslash escape sequences were preserved in enum names, you will need to update your code to handle properly unescaped values. + +Example: + +```ts +// Before (incorrect): +parseEnumType({ + columnType: "Enum8('f\\'' = 1)", + sourceType: "Enum8('f\\'' = 1)", +}); +// returned: { values: { 1: "f\\'" } } // with backslash + +// After (correct): +parseEnumType({ + columnType: "Enum8('f\\'' = 1)", + sourceType: "Enum8('f\\'' = 1)", +}); +// returns: { values: { 1: "f'" } } // unescaped +``` + +# 1.18.5 + +## Improvements + +- (Node.js only) Added `max_response_headers_size` client option that forwards the [`maxHeaderSize`](https://nodejs.org/api/http.html#httprequesturl-options-callback) option to the underlying `http(s).request` call. This raises the per-request limit on the total size of HTTP response headers received from the server (Node.js default is ~16 KB). It is most useful when running long-running queries with `send_progress_in_http_headers` enabled — the `X-ClickHouse-Progress` headers accumulate over the lifetime of the request and can exceed the default limit, causing the request to fail with `HPE_HEADER_OVERFLOW`. Setting this option avoids the need to use the global `--max-http-header-size` Node.js CLI flag or the `NODE_OPTIONS` environment variable. Has no effect for the Web client (which uses `fetch`) and no effect when a custom `http_agent` is configured with a request implementation that does not honor the option. + +```ts +const client = createClient({ + request_timeout: 400_000, + max_response_headers_size: 1024 * 1024, // accept up to 1 MiB of response headers + clickhouse_settings: { + send_progress_in_http_headers: 1, + http_headers_progress_interval_ms: "110000", + }, +}); +``` + +- The `@clickhouse/client` npm package now ships embedded AI-agent skills, `clickhouse-js-node-coding` and `clickhouse-js-node-troubleshooting`, under `node_modules/@clickhouse/client/skills/`. These skills are also declared in the `agents.skills` field of the package manifest for discovery tools that scan `node_modules`. This allows agentic coding tools to load focused, Node-client-specific coding and troubleshooting guidance without any additional setup. ([#682]) + +[#682]: https://github.com/ClickHouse/clickhouse-js/pull/682 + +# 1.18.4 + +A release-infrastructure-only version bump (no user-facing changes). See 1.18.5 for the next release with user-facing improvements. + +# 1.18.3 + +## Improvements + +- Added `keep_alive.eagerly_destroy_stale_sockets` option (Node.js only, default: `false`). When enabled, sockets that have been idle for longer than `idle_socket_ttl` are destroyed immediately before each request, rather than waiting for the idle timeout to fire. This helps reclaim stale sockets during event loop delays, where the timeout callback may not run on time. + +```ts +const client = createClient({ + keep_alive: { + enabled: true, + idle_socket_ttl: 2500, + eagerly_destroy_stale_sockets: true, + }, +}); +``` + +- Added auto-detection and warning when `request_timeout` is high (> 60 seconds) but progress headers are not configured. Long-running queries may fail with socket hang-up errors if they exceed the load balancer idle timeout. The client now warns users to enable `send_progress_in_http_headers` and `http_headers_progress_interval_ms` settings to prevent such issues. + +```ts +// This will now trigger a warning +const client = createClient({ + request_timeout: 120_000, // 120 seconds + // send_progress_in_http_headers is not configured +}); + +// ✓ Properly configured to avoid load balancer timeouts +const client = createClient({ + request_timeout: 400_000, + clickhouse_settings: { + send_progress_in_http_headers: 1, + http_headers_progress_interval_ms: "110000", // ~10s below LB timeout + }, +}); +``` + +# 1.18.2 + +## Improvements + +- Added a helping `WARN` level log message with a suggestion to check the `keep_alive` configuration if the client receives an `ECONNRESET` error from the server, which can happen when the server closes idle connections after a certain timeout, and the client tries to reuse such a connection from the pool. This can be especially helpful for new users who might not be aware of this aspect of HTTP connection management. The log message is only emitted if the `keep_alive` option is enabled in the client configuration, and it includes the server's keep-alive timeout value (if available) to assist with troubleshooting. ([#597](https://github.com/ClickHouse/clickhouse-js/pull/597)) + +How to reproduce the issue that triggers the log message: + +```ts +const client = createClient({ + // ... + keep_alive: { + enabled: true, + // ❌ DON'T SET THIS VALUE SO HIGH IN PRODUCTION + idle_socket_ttl: 1_000_000, + }, + log: { + level: ClickHouseLogLevel.WARN, // to see the warning logs + }, +}); + +for (let i = 0; i < 1000; i++) { + await client.ping({ + // To use a regular query instead of the /ping endpoint + // which might be configured differently on the server side + // and have different timeout settings. + select: true, + }); + + // Wait long enough to let the server close the idle connection, + // but not too long to let the client remove it from the pool, + // in other words try to hit the scenario when the race condition + // happens between the server closing the connection and the client + // trying to reuse it. + await sleep(SERVER_KEEP_ALIVE_TIMEOUT_MS - 100); +} +``` + +Example log message: + +```json +{ + "message": "Ping: idle socket TTL is greater than server keep-alive timeout, try setting idle socket TTL to a value lower than the server keep-alive timeout to prevent unexpected connection resets, see https://github.com/ClickHouse/clickhouse-js/blob/main/docs/howto/keep_alive_timeout.md for more details.", + "args": { + "operation": "Ping", + "connection_id": "8dc1c9bd-7895-49b1-8a95-276470151c65", + "query_id": "beee95af-2e83-4dcb-8e1e-045bd61f4985", + "request_id": "8dc1c9bd-7895-49b1-8a95-276470151c65:2", + "socket_id": "8dc1c9bd-7895-49b1-8a95-276470151c65:1", + "server_keep_alive_timeout_ms": 10000, + "idle_socket_ttl": 15000 + }, + "module": "HTTP Adapter" +} +``` + +# 1.18.1 + +## Improvements + +- Setting `log.level` default value to `ClickHouseLogLevel.WARN` instead of `ClickHouseLogLevel.OFF` to provide better visibility into potential issues without overwhelming users with too much information by default. + +```ts +const client = createClient({ + // ... + log: { + level: ClickHouseLogLevel.WARN, // default is now ClickHouseLogLevel.WARN instead of ClickHouseLogLevel.OFF + }, +}); +``` + +- Logging is now lazy, which means that the log messages will only be constructed if the log level is appropriate for the message. This can improve performance in cases where constructing the log message is expensive, and the log level is set to ignore such messages. See `ClickHouseLogLevel` enum for the complete list of log levels. ([#520]) + +```ts +const client = createClient({ + // ... + log: { + level: ClickHouseLogLevel.TRACE, // to log everything available down to the network level events + }, +}); +``` + +- Enhanced the logging of the HTTP request / socket lifecycle with additional trace messages and context such as Connection ID (UUID) and Request ID and Socket ID that embed the connection ID for ease of tracing the logs of a particular request across the connection lifecycle. To enable such logs, set the `log.level` config option to `ClickHouseLogLevel.TRACE`. ([#567]) + +```console +[2026-02-25T09:19:13.511Z][TRACE][@clickhouse/client][Connection] Insert: received 'close' event, 'free' listener removed +Arguments: { + operation: 'Insert', + connection_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c', + query_id: '9dfda627-39a2-41a6-9fc9-8f8716574826', + request_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:3', + socket_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:2', + event: 'close' +} +[2026-02-25T09:19:13.502Z][TRACE][@clickhouse/client][Connection] Query: reusing socket +Arguments: { + operation: 'Query', + connection_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c', + query_id: 'ad0127e8-b1c7-4ed6-9681-c0162f7a0ea9', + request_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:4', + socket_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:2', + usage_count: 1 +} +``` + +- A step towards structured logging: the client now passes rich context to the logger `args` parameter (e.g. `connection_id`, `query_id`, `request_id`, `socket_id`). ([#576]) + +## Deprecated API + +- The `drainStream` utility function is now deprecated, as the client will handle draining the stream internally when needed. Use `client.command()` instead, which will handle draining the stream internally when needed. ([#578]) + +- The `sleep` utility function is now deprecated, as it is not intended to be used outside of the client implementation. Use `setTimeout` directly or a more full-featured utility library if you need additional features like cancellation or timers management. ([#578]) + +[#520]: https://github.com/ClickHouse/clickhouse-js/pull/520 +[#567]: https://github.com/ClickHouse/clickhouse-js/pull/567 +[#576]: https://github.com/ClickHouse/clickhouse-js/pull/576 +[#578]: https://github.com/ClickHouse/clickhouse-js/pull/578 + +# 1.18.0 + +A beta version. See 1.18.1 for the stable release. + +# 1.17.0 + +## New features + +- Added `http_status_code` to query, insert, and exec commands ([#525], [Kinzeng]) +- Fixed `ignore_error_response` not getting passed when using `command` ([#536], [Kinzeng]) + +[#525]: https://github.com/ClickHouse/clickhouse-js/pull/525 +[#536]: https://github.com/ClickHouse/clickhouse-js/pull/536 + +# 1.16.0 + +## New features + +- Added support for the new [Disposable API] (a.k.a the `using` keyword) (#500) + +[Disposable API]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/using + +```ts +async function main() { + using resultSet = await client.query(…); + + // some code that can throw + // but thanks to `using` the resultSet will still get disposed + + // resultSet is also automatically disposed here by calling [Symbol.dispose] +} +``` + +Without the new `using` keyword it is required to wrap the code that might leak expensive resources like sockets and big buffers in ` try / finally` + +```ts +async function main() { + let client + try { + client = await createClient(…); + // some code that can throw + } finally { + if (client) { + await client.close() + } + } +} +``` + +# 1.15.0 + +## New features + +- Added support for [BigInt] values in query parameters. ([#487], @dalechyn) + +[#487]: https://github.com/ClickHouse/clickhouse-js/pull/487 +[BigInt]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt + +# 1.14.0 + +## New features + +- It is now possible to specify custom `parse` and `stringify` functions that will be used instead of the standard `JSON.parse` and `JSON.stringify` methods for JSON serialization/deserialization when working with `JSON*` family formats. See `ClickHouseClientConfigOptions.json`, and a new [custom_json_handling] example for more details. ([#481], [looskie]) +- (Node.js only) Added an `ignore_error_response` param to `ClickHouseClient.exec`, which allows callers to manually handle request errors on the application side. ([#483], [Kinzeng]) + +[#481]: https://github.com/ClickHouse/clickhouse-js/pull/481 +[#483]: https://github.com/ClickHouse/clickhouse-js/pull/483 +[looskie]: https://github.com/looskie +[Kinzeng]: https://github.com/Kinzeng +[custom_json_handling]: https://github.com/ClickHouse/clickhouse-js/blob/1.14.0/examples/custom_json_handling.ts + +# 1.13.0 + +## New features + +- Server-side exceptions that occur in the middle of the HTTP stream are now handled correctly. This requires [ClickHouse 25.11+](https://github.com/ClickHouse/ClickHouse/pull/88818). Previous ClickHouse versions are unaffected by this change. ([#478]) + +## Improvements + +- `TupleParam` constructor now accepts a readonly array to permit more usages. ([#465], [Malien]) + +## Bug fixes + +- Fixed boolean value formatting in query parameters. Boolean values within `Array`, `Tuple`, and `Map` types are now correctly formatted as `TRUE`/`FALSE` instead of `1`/`0` to ensure proper type compatibility with ClickHouse. ([#475], [baseballyama]) + +[#465]: https://github.com/ClickHouse/clickhouse-js/pull/465 +[#475]: https://github.com/ClickHouse/clickhouse-js/pull/475 +[#478]: https://github.com/ClickHouse/clickhouse-js/pull/478 +[Malien]: https://github.com/Malien +[baseballyama]: https://github.com/baseballyama + +# 1.12.1 + +## Improvements + +- Improved performance of `toSearchParams`. ([#449], [twk]) + +## Other + +- Added Node.js 24.x to the CI matrix. Node.js 18.x was removed from the CI due to [EOL](https://endoflife.date/nodejs). + +[#449]: https://github.com/ClickHouse/clickhouse-js/pull/449 +[twk]: https://github.com/twk + +# 1.12.0 + +## Types + +- Add missing `allow_experimental_join_condition` to `ClickHouseSettings` typing. ([#430], [looskie]) +- Fixed `JSONEachRowWithProgress` TypeScript flow after the breaking changes in [ClickHouse 25.1]. `RowOrProgress` now has an additional variant: `SpecialEventRow`. The library now additionally exports the `parseError` method, and newly added `isRow` / `isException` type guards. See the updated [JSONEachRowWithProgress example] ([#443]) +- Added missing `allow_experimental_variant_type` (24.1+), `allow_experimental_dynamic_type` (24.5+), `allow_experimental_json_type` (24.8+), `enable_json_type` (25.3+), `enable_time_time64_type` (25.6+) to `ClickHouseSettings` typing. ([#445]) + +## Improvements + +- Add a warning on a socket closed without fully consuming the stream (e.g., when using `query` or `exec` method). ([#441]) +- (Node.js only) An option to use a simple SELECT query for ping checks instead of `/ping` endpoint. See the new optional argument to the `ClickHouseClient.ping` method and `PingParams` typings. Note that the Web version always used a SELECT query by default, as the `/ping` endpoint does not support CORS, and that cannot be changed. ([#442]) + +## Other + +- The project now uses [Codecov] instead of SonarCloud for code coverage reports. ([#444]) + +[#430]: https://github.com/ClickHouse/clickhouse-js/pull/430 +[#441]: https://github.com/ClickHouse/clickhouse-js/pull/441 +[#442]: https://github.com/ClickHouse/clickhouse-js/pull/442 +[#443]: https://github.com/ClickHouse/clickhouse-js/pull/443 +[#444]: https://github.com/ClickHouse/clickhouse-js/pull/444 +[#445]: https://github.com/ClickHouse/clickhouse-js/pull/445 +[looskie]: https://github.com/looskie +[ClickHouse 25.1]: https://github.com/ClickHouse/ClickHouse/pull/74181 +[JSONEachRowWithProgress example]: https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/select_json_each_row_with_progress.ts +[Codecov]: https://codecov.io/gh/ClickHouse/clickhouse-js + +# 1.11.2 (Common, Node.js) + +A minor release to allow further investigation regarding uncaught error issues with [#410]. + +## Types + +- Added missing `lightweight_deletes_sync` typing to `ClickHouseSettings` ([#422], [pratimapatel2008]) + +## Improvements (Node.js) + +- Added a new configuration option: `capture_enhanced_stack_trace`; see the JS doc in the Node.js client package. Note that it is disabled by default due to a possible performance impact. ([#427]) +- Added more try-catch blocks to the Node.js connection layer. ([#427]) + +[#410]: https://github.com/ClickHouse/clickhouse-js/pull/410 +[#422]: https://github.com/ClickHouse/clickhouse-js/pull/422 +[#427]: https://github.com/ClickHouse/clickhouse-js/pull/427 +[pratimapatel2008]: https://github.com/pratimapatel2008 + +# 1.11.1 (Common, Node.js, Web) + +## Bug fixes + +- Fixed an issue with URLEncoded special characters in the URL configuration for username or password. ([#407](https://github.com/ClickHouse/clickhouse-js/issues/407)) + +## Improvements + +- Added support for streaming on 32-bit platforms. ([#403](https://github.com/ClickHouse/clickhouse-js/pull/403), [shevchenkonik](https://github.com/shevchenkonik)) + +# 1.11.0 (Common, Node.js, Web) + +## New features + +- It is now possible to provide custom HTTP headers when calling the `query`/`insert`/`command`/`exec` methods using the `http_headers` option. NB: `http_headers` specified this way will override `http_headers` set on the client instance level. ([#394](https://github.com/ClickHouse/clickhouse-js/issues/374), [@DylanRJohnston](https://github.com/DylanRJohnston)) +- (Web only) It is now possible to provide a custom `fetch` implementation to the client. ([#315](https://github.com/ClickHouse/clickhouse-js/issues/315), [@lucacasonato](https://github.com/lucacasonato)) + +# 1.10.1 (Common, Node.js, Web) + +## Bug fixes + +- Fixed `NULL` parameter binding with `Tuple`, `Array`, and `Map` types. ([#374](https://github.com/ClickHouse/clickhouse-js/issues/374)) + +## Improvements + +- `ClickHouseSettings` typings now include `session_timeout` and `session_check` settings. ([#370](https://github.com/ClickHouse/clickhouse-js/issues/370)) + +# 1.10.0 (Common, Node.js, Web) + +## New features + +- Added support for JWT authentication (ClickHouse Cloud feature) in both Node.js and Web API packages. JWT token can be set via `access_token` client configuration option. + + ```ts + const client = createClient({ + // ... + access_token: "", + }); + ``` + + Access token can also be configured via the URL params, e.g., `https://host:port?access_token=...`. + + It is also possible to override the access token for a particular request (see `BaseQueryParams.auth` for more details). + + NB: do not mix access token and username/password credentials in the configuration; the client will throw an error if both are set. + +# 1.9.1 (Node.js only) + +## Bug fixes + +- Fixed an uncaught exception that could happen in case of malformed ClickHouse response when response compression is enabled ([#363](https://github.com/ClickHouse/clickhouse-js/issues/363)) + +# 1.9.0 (Common, Node.js, Web) + +## New features + +- Added `input_format_json_throw_on_bad_escape_sequence` to the `ClickhouseSettings` type. ([#355](https://github.com/ClickHouse/clickhouse-js/pull/355), [@emmanuel-bonin](https://github.com/emmanuel-bonin)) +- The client now exports `TupleParam` wrapper class, allowing tuples to be properly used as query parameters. Added support for JS Map as a query parameter. ([#359](https://github.com/ClickHouse/clickhouse-js/pull/359)) + +## Improvements + +- The client will throw a more informative error if the buffered response is larger than the max allowed string length in V8, which is `2**29 - 24` bytes. ([#357](https://github.com/ClickHouse/clickhouse-js/pull/357)) + +# 1.8.1 (Node.js) + +## Bug fixes + +- When a custom HTTP agent is used, the HTTP or HTTPS request implementation is now correctly chosen based on the URL protocol. ([#352](https://github.com/ClickHouse/clickhouse-js/issues/352)) + +# 1.8.0 (Common, Node.js, Web) + +## New features + +- Added support for specifying roles via request query parameters. See [this example](examples/role.ts) for more details. ([@pulpdrew](https://github.com/pulpdrew), [#328](https://github.com/ClickHouse/clickhouse-js/pull/328)) + +# 1.7.0 (Common, Node.js, Web) + +## Bug fixes + +- (Web only) Fixed an issue where streaming large datasets could provide corrupted results. See [#333](https://github.com/ClickHouse/clickhouse-js/pull/333) (PR) for more details. + +## New features + +- Added `JSONEachRowWithProgress` format support, `ProgressRow` interface, and `isProgressRow` type guard. See [this Node.js example](../../examples/node/select_json_each_row_with_progress.ts) for more details. It should work similarly with the Web version. +- (Experimental) Exposed the `parseColumnType` function that takes a string representation of a ClickHouse type (e.g., `FixedString(16)`, `Nullable(Int32)`, etc.) and returns an AST-like object that represents the type. For example: + + ```ts + for (const type of [ + "Int32", + "Array(Nullable(String))", + `Map(Int32, DateTime64(9, 'UTC'))`, + ]) { + console.log(`##### Source ClickHouse type: ${type}`); + console.log(parseColumnType(type)); + } + ``` + + The above code will output: + + ``` + ##### Source ClickHouse type: Int32 + { type: 'Simple', columnType: 'Int32', sourceType: 'Int32' } + ##### Source ClickHouse type: Array(Nullable(String)) + { + type: 'Array', + value: { + type: 'Nullable', + sourceType: 'Nullable(String)', + value: { type: 'Simple', columnType: 'String', sourceType: 'String' } + }, + dimensions: 1, + sourceType: 'Array(Nullable(String))' + } + ##### Source ClickHouse type: Map(Int32, DateTime64(9, 'UTC')) + { + type: 'Map', + key: { type: 'Simple', columnType: 'Int32', sourceType: 'Int32' }, + value: { + type: 'DateTime64', + timezone: 'UTC', + precision: 9, + sourceType: "DateTime64(9, 'UTC')" + }, + sourceType: "Map(Int32, DateTime64(9, 'UTC'))" + } + ``` + + While the original intention was to use this function internally for `Native`/`RowBinaryWithNamesAndTypes` data formats headers parsing, it can be useful for other purposes as well (e.g., interfaces generation, or custom JSON serializers). + + NB: currently unsupported source types to parse: + - Geo + - (Simple)AggregateFunction + - Nested + - Old/new experimental JSON + - Dynamic + - Variant + +# 1.6.0 (Common, Node.js, Web) + +## New features + +- Added optional `real_time_microseconds` field to the `ClickHouseSummary` interface (see ) + +## Bug fixes + +- Fixed unhandled exceptions produced when calling `ResultSet.json` if the response data was not in fact a valid JSON. ([#311](https://github.com/ClickHouse/clickhouse-js/pull/311)) + +# 1.5.0 (Node.js) + +## New features + +- It is now possible to disable the automatic decompression of the response stream with the `exec` method. See `ExecParams.decompress_response_stream` for more details. ([#298](https://github.com/ClickHouse/clickhouse-js/issues/298)). + +# 1.4.1 (Node.js, Web) + +## Improvements + +- `ClickHouseClient` is now exported as a value from `@clickhouse/client` and `@clickhouse/client-web` packages, allowing for better integration in dependency injection frameworks that rely on IoC (e.g., [Nest.js](https://github.com/nestjs/nest), [tsyringe](https://github.com/microsoft/tsyringe)) ([@mathieu-bour](https://github.com/mathieu-bour), [#292](https://github.com/ClickHouse/clickhouse-js/issues/292)). + +## Bug fixes + +- Fixed a potential socket hang up issue that could happen under 100% CPU load ([#294](https://github.com/ClickHouse/clickhouse-js/issues/294)). + +# 1.4.0 (Node.js) + +## New features + +- (Node.js only) The `exec` method now accepts an optional `values` parameter, which allows you to pass the request body as a `Stream.Readable`. This can be useful in case of custom insert streaming with arbitrary ClickHouse data formats (which might not be explicitly supported and allowed by the client in the `insert` method yet). NB: in this case, you are expected to serialize the data in the stream in the required input format yourself. + +# 1.3.0 (Common, Node.js, Web) + +## New features + +- It is now possible to get the entire response headers object from the `query`/`insert`/`command`/`exec` methods. With `query`, you can access the `ResultSet.response_headers` property; other methods (`insert`/`command`/`exec`) return it as parts of their response objects as well. + For example: + + ```ts + const rs = await client.query({ + query: "SELECT * FROM system.numbers LIMIT 1", + format: "JSONEachRow", + }); + console.log(rs.response_headers["content-type"]); + ``` + + This will print: `application/x-ndjson; charset=UTF-8`. It can be used in a similar way with the other methods. + +## Improvements + +- Re-exported several constants from the `@clickhouse/client-common` package for convenience: + - `SupportedJSONFormats` + - `SupportedRawFormats` + - `StreamableFormats` + - `StreamableJSONFormats` + - `SingleDocumentJSONFormats` + - `RecordsJSONFormats` + +# 1.2.0 (Node.js) + +## New features + +- (Experimental) Added an option to provide a custom HTTP Agent in the client configuration via the `http_agent` option ([#283](https://github.com/ClickHouse/clickhouse-js/issues/283), related: [#278](https://github.com/ClickHouse/clickhouse-js/issues/278)). The following conditions apply if a custom HTTP Agent is provided: + - The `max_open_connections` and `tls` options will have _no effect_ and will be ignored by the client, as it is a part of the underlying HTTP Agent configuration. + - `keep_alive.enabled` will only regulate the default value of the `Connection` header (`true` -> `Connection: keep-alive`, `false` -> `Connection: close`). + - While the idle socket management will still work, it is now possible to disable it completely by setting the `keep_alive.idle_socket_ttl` value to `0`. +- (Experimental) Added a new client configuration option: `set_basic_auth_header`, which disables the `Authorization` header that is set by the client by default for every outgoing HTTP request. One of the possible scenarios when it is necessary to disable this header is when a custom HTTPS agent is used, and the server requires TLS authorization. For example: + + ```ts + const agent = new https.Agent({ + ca: fs.readFileSync("./ca.crt"), + }); + const client = createClient({ + url: "https://server.clickhouseconnect.test:8443", + http_agent: agent, + // With a custom HTTPS agent, the client won't use the default HTTPS connection implementation; the headers should be provided manually + http_headers: { + "X-ClickHouse-User": "default", + "X-ClickHouse-Key": "", + }, + // Authorization header conflicts with the TLS headers; disable it. + set_basic_auth_header: false, + }); + ``` + +NB: It is currently not possible to set the `set_basic_auth_header` option via the URL params. + +If you have feedback on these experimental features, please let us know by creating [an issue](https://github.com/ClickHouse/clickhouse-js/issues) in the repository. + +# 1.1.0 (Common, Node.js, Web) + +## New features + +- Added an option to override the credentials for a particular `query`/`command`/`exec`/`insert` request via the `BaseQueryParams.auth` setting; when set, the credentials will be taken from there instead of the username/password provided during the client instantiation ([#278](https://github.com/ClickHouse/clickhouse-js/issues/278)). +- Added an option to override the `session_id` for a particular `query`/`command`/`exec`/`insert` request via the `BaseQueryParams.session_id` setting; when set, it will be used instead of the session id provided during the client instantiation ([@holi0317](https://github.com/Holi0317), [#271](https://github.com/ClickHouse/clickhouse-js/issues/271)). + +## Bug fixes + +- Fixed the incorrect `ResponseJSON.totals` TypeScript type. Now it correctly matches the shape of the data (`T`, default = `unknown`) instead of the former `Record` definition ([#274](https://github.com/ClickHouse/clickhouse-js/issues/274)). + +# 1.0.2 (Common, Node.js, Web) + +## Bug fixes + +- The `command` method now drains the response stream properly, as the previous implementation could cause the `Keep-Alive` socket to close after each request. +- Removed an unnecessary error log in the `ResultSet.stream` method if the request was aborted or the result set was closed ([#263](https://github.com/ClickHouse/clickhouse-js/issues/263)). + +## Improvements + +- `ResultSet.stream` logs an error via the `Logger` instance, if the stream emits an error event instead of a simple `console.error` call. +- Minor adjustments to the `DefaultLogger` log messages formatting. +- Added missing `rows_before_limit_at_least` to the ResponseJSON type ([@0237h](https://github.com/0237h), [#267](https://github.com/ClickHouse/clickhouse-js/issues/267)). + +# 1.0.1 (Common, Node.js, Web) + +## Bug fixes + +- Fixed the regression where the default HTTP/HTTPS port numbers (80/443) could not be used with the URL configuration ([#258](https://github.com/ClickHouse/clickhouse-js/issues/258)). + +# 1.0.0 (Common, Node.js, Web) + +Formal stable release milestone with a lot of improvements and some [breaking changes](#breaking-changes-in-100). + +Major new features overview: + +- [Advanced TypeScript support for `query` + `ResultSet`](#advanced-typescript-support-for-query--resultset) +- [URL configuration](#url-configuration) + +From now on, the client will follow the [official semantic versioning](https://docs.npmjs.com/about-semantic-versioning) guidelines. + +## Deprecated API + +The following configuration parameters are marked as deprecated: + +- `host` configuration parameter is deprecated; use `url` instead. +- `additional_headers` configuration parameter is deprecated; use `http_headers` instead. + +The client will log a warning if any of these parameters are used. However, it is still allowed to use `host` instead of `url` and `additional_headers` instead of `http_headers` for now; this deprecation is not supposed to break the existing code. + +These parameters will be removed in the next major release (2.0.0). + +See "New features" section for more details. + +## Breaking changes in 1.0.0 + +- `compression.response` is now disabled by default in the client configuration options, as it cannot be used with readonly=1 users, and it was not clear from the ClickHouse error message what exact client option was causing the failing query in this case. If you'd like to continue using response compression, you should explicitly enable it in the client configuration. +- As the client now supports parsing [URL configuration](#url-configuration), you should specify `pathname` as a separate configuration option (as it would be considered as the `database` otherwise). +- (TypeScript only) `ResultSet` and `Row` are now more strictly typed, according to the format used during the `query` call. See [this section](#advanced-typescript-support-for-query--resultset) for more details. +- (TypeScript only) Both Node.js and Web versions now uniformly export correct `ClickHouseClient` and `ClickHouseClientConfigOptions` types, specific to each implementation. Exported `ClickHouseClient` now does not have a `Stream` type parameter, as it was unintended to expose it there. NB: you should still use `createClient` factory function provided in the package. + +## New features in 1.0.0 + +### Advanced TypeScript support for `query` + `ResultSet` + +Client will now try its best to figure out the shape of the data based on the DataFormat literal specified to the `query` call, as well as which methods are allowed to be called on the `ResultSet`. + +Live demo (see the full description below): + +[Screencast](https://github.com/ClickHouse/clickhouse-js/assets/3175289/b66afcb2-3a10-4411-af59-51d2754c417e) + +Complete reference: + +| Format | `ResultSet.json()` | `ResultSet.stream()` | Stream data | `Row.json()` | +| ------------------------------- | --------------------- | --------------------------- | ----------------- | --------------- | +| JSON | ResponseJSON\ | never | never | never | +| JSONObjectEachRow | Record\ | never | never | never | +| All other `JSON*EachRow` | Array\ | Stream\\>\> | Array\\> | T | +| CSV/TSV/CustomSeparated/Parquet | never | Stream\\>\> | Array\\> | never | + +By default, `T` (which represents `JSONType`) is still `unknown`. However, considering `JSONObjectsEachRow` example: prior to 1.0.0, you had to specify the entire type hint, including the shape of the data, manually: + +```ts +type Data = { foo: string }; + +const resultSet = await client.query({ + query: "SELECT * FROM my_table", + format: "JSONObjectsEachRow", +}); + +// pre-1.0.0, `resultOld` has type Record +const resultOld = resultSet.json>(); +// const resultOld = resultSet.json() // incorrect! The type hint should've been `Record` here. + +// 1.0.0, `resultNew` also has type Record; client inferred that it has to be a Record from the format literal. +const resultNew = resultSet.json(); +``` + +This is even more handy in case of streaming on the Node.js platform: + +```ts +const resultSet = await client.query({ + query: "SELECT * FROM my_table", + format: "JSONEachRow", +}); + +// pre-1.0.0 +// `streamOld` was just a regular Node.js Stream.Readable +const streamOld = resultSet.stream(); +// `rows` were `any`, needed an explicit type hint +streamNew.on("data", (rows: Row[]) => { + rows.forEach((row) => { + // without an explicit type hint to `rows`, calling `forEach` and other array methods resulted in TS compiler errors + const t = row.text; + const j = row.json(); // `j` needed a type hint here, otherwise, it's `unknown` + }); +}); + +// 1.0.0 +// `streamNew` is now StreamReadable (Node.js Stream.Readable with a bit more type hints); +// type hint for the further `json` calls can be added here (and removed from the `json` calls) +const streamNew = resultSet.stream(); +// `rows` are inferred as an Array> instead of `any` +streamNew.on("data", (rows) => { + // `row` is inferred as Row + rows.forEach((row) => { + // no explicit type hints required, you can use `forEach` straight away and TS compiler will be happy + const t = row.text; + const j = row.json(); // `j` will be of type Data + }); +}); + +// async iterator now also has type hints +// similarly to the `on(data)` example above, `rows` are inferred as Array> +for await (const rows of streamNew) { + // `row` is inferred as Row + rows.forEach((row) => { + const t = row.text; + const j = row.json(); // `j` will be of type Data + }); +} +``` + +Calling `ResultSet.stream` is not allowed for certain data formats, such as `JSON` and `JSONObjectsEachRow` (unlike `JSONEachRow` and the rest of `JSON*EachRow`, these formats return a single object). In these cases, the client throws an error. However, it was previously not reflected on the type level; now, calling `stream` on these formats will result in a TS compiler error. For example: + +```ts +const resultSet = await client.query("SELECT * FROM table", { + format: "JSON", +}); +const stream = resultSet.stream(); // `stream` is `never` +``` + +Calling `ResultSet.json` also does not make sense on `CSV` and similar "raw" formats, and the client throws. Again, now, it is typed properly: + +```ts +const resultSet = await client.query("SELECT * FROM table", { + format: "CSV", +}); +// `json` is `never`; same if you stream CSV, and call `Row.json` - it will be `never`, too. +const json = resultSet.json(); +``` + +Currently, there is one known limitation: as the general shape of the data and the methods allowed for calling are inferred from the format literal, there might be situations where it will fail to do so, for example: + +```ts +// assuming that `queryParams` has `JSONObjectsEachRow` format inside +async function runQuery( + queryParams: QueryParams, +): Promise> { + const resultSet = await client.query(queryParams); + // type hint here will provide a union of all known shapes instead of a specific one + // inferred shapes: Data[] | ResponseJSON | Record + return resultSet.json(); +} +``` + +In this case, as it is _likely_ that you already know the desired format in advance (otherwise, returning a specific shape like `Record` would've been incorrect), consider helping the client a bit: + +```ts +async function runQuery( + queryParams: QueryParams, +): Promise> { + const resultSet = await client.query({ + ...queryParams, + format: "JSONObjectsEachRow", + }); + // TS understands that it is a Record now + return resultSet.json(); +} +``` + +If you are interested in more details, see the [related test](../../packages/client-node/__tests__/integration/node_query_format_types.test.ts) (featuring a great ESLint plugin [expect-types](https://github.com/JoshuaKGoldberg/eslint-plugin-expect-type)) in the client package. + +### URL configuration + +- Added `url` configuration parameter. It is intended to replace the deprecated `host`, which was already supposed to be passed as a valid URL. +- It is now possible to configure most of the client instance parameters with a URL. The URL format is `http[s]://[username:password@]hostname:port[/database][?param1=value1¶m2=value2]`. In almost every case, the name of a particular parameter reflects its path in the config options interface, with a few exceptions. The following parameters are supported: + +| Parameter | Type | +| ------------------------------------------- | ----------------------------------------------------------------- | +| `pathname` | an arbitrary string. | +| `application_id` | an arbitrary string. | +| `session_id` | an arbitrary string. | +| `request_timeout` | non-negative number. | +| `max_open_connections` | non-negative number, greater than zero. | +| `compression_request` | boolean. See below [1]. | +| `compression_response` | boolean. | +| `log_level` | allowed values: `OFF`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`. | +| `keep_alive_enabled` | boolean. | +| `clickhouse_setting_*` or `ch_*` | see below [2]. | +| `http_header_*` | see below [3]. | +| (Node.js only) `keep_alive_idle_socket_ttl` | non-negative number. | + +[1] For booleans, valid values will be `true`/`1` and `false`/`0`. + +[2] Any parameter prefixed with `clickhouse_setting_` or `ch_` will have this prefix removed and the rest added to client's `clickhouse_settings`. For example, `?ch_async_insert=1&ch_wait_for_async_insert=1` will be the same as: + +```ts +createClient({ + clickhouse_settings: { + async_insert: 1, + wait_for_async_insert: 1, + }, +}); +``` + +Note: boolean values for `clickhouse_settings` should be passed as `1`/`0` in the URL. + +[3] Similar to [2], but for `http_header` configuration. For example, `?http_header_x-clickhouse-auth=foobar` will be an equivalent of: + +```ts +createClient({ + http_headers: { + "x-clickhouse-auth": "foobar", + }, +}); +``` + +**Important: URL will _always_ overwrite the hardcoded values and a warning will be logged in this case.** + +Currently not supported via URL: + +- `log.LoggerClass` +- (Node.js only) `tls_ca_cert`, `tls_cert`, `tls_key`. + +See also: [URL configuration example](../../examples/url_configuration.ts). + +### Performance + +- (Node.js only) Improved performance when decoding the entire set of rows with _streamable_ JSON formats (such as `JSONEachRow` or `JSONCompactEachRow`) by calling the `ResultSet.json()` method. NB: The actual streaming performance when consuming the `ResultSet.stream()` hasn't changed. Only the `ResultSet.json()` method used a suboptimal stream processing in some instances, and now `ResultSet.json()` just consumes the same stream transformer provided by the `ResultSet.stream()` method (see [#253](https://github.com/ClickHouse/clickhouse-js/pull/253) for more details). + +### Miscellaneous + +- Added `http_headers` configuration parameter as a direct replacement for `additional_headers`. Functionally, it is the same, and the change is purely cosmetic, as we'd like to leave an option to implement TCP connection in the future open. + +## 0.3.1 (Common, Node.js, Web) + +### Bug fixes + +- Fixed an issue where query parameters containing tabs or newline characters were not encoded properly. + +## 0.3.0 (Node.js only) + +This release primarily focuses on improving the Keep-Alive mechanism's reliability on the client side. + +### New features + +- Idle sockets timeout rework; now, the client attaches internal timers to idling sockets, and forcefully removes them from the pool if it considers that a particular socket is idling for too long. The intention of this additional sockets housekeeping is to eliminate "Socket hang-up" errors that could previously still occur on certain configurations. Now, the client does not rely on KeepAlive agent when it comes to removing the idling sockets; in most cases, the server will not close the socket before the client does. +- There is a new `keep_alive.idle_socket_ttl` configuration parameter. The default value is `2500` (milliseconds), which is considered to be safe, as [ClickHouse versions prior to 23.11 had `keep_alive_timeout` set to 3 seconds by default](https://github.com/ClickHouse/ClickHouse/commit/1685cdcb89fe110b45497c7ff27ce73cc03e82d1), and `keep_alive.idle_socket_ttl` is supposed to be slightly less than that to allow the client to remove the sockets that are about to expire before the server does so. +- Logging improvements: more internal logs on failing requests; all client methods except ping will log an error on failure now. A failed ping will log a warning, since the underlying error is returned as a part of its result. Client logging still needs to be enabled explicitly by specifying the desired `log.level` config option, as the log level is `OFF` by default. Currently, the client logs the following events, depending on the selected `log.level` value: + - `TRACE` - low-level information about the Keep-Alive sockets lifecycle. + - `DEBUG` - response information (without authorization headers and host info). + - `INFO` - still mostly unused, will print the current log level when the client is initialized. + - `WARN` - non-fatal errors; failed `ping` request is logged as a warning, as the underlying error is included in the returned result. + - `ERROR` - fatal errors from `query`/`insert`/`exec`/`command` methods, such as a failed request. + +### Breaking changes + +- `keep_alive.retry_on_expired_socket` and `keep_alive.socket_ttl` configuration parameters are removed. +- The `max_open_connections` configuration parameter is now 10 by default, as we should not rely on the KeepAlive agent's defaults. +- Fixed the default `request_timeout` configuration value (now it is correctly set to `30_000`, previously `300_000` (milliseconds)). + +### Bug fixes + +- Fixed a bug with Ping that could lead to an unhandled "Socket hang-up" propagation. +- Ensure proper `Connection` header value considering Keep-Alive settings. If Keep-Alive is disabled, its value is now forced to ["close"](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Connection#close). + +## 0.3.0-beta.1 (Node.js only) + +See [0.3.0](#030-nodejs-only). + +## 0.2.10 (Common, Node.js, Web) + +### New features + +- If `InsertParams.values` is an empty array, no request is sent to the server and `ClickHouseClient.insert` short-circuits itself. In this scenario, the newly added `InsertResult.executed` flag will be `false`, and `InsertResult.query_id` will be an empty string. + +### Bug fixes + +- Client no longer produces `Code: 354. inflate failed: buffer error` exception if request compression is enabled and `InsertParams.values` is an empty array (see above). + +## 0.2.9 (Common, Node.js, Web) + +### New features + +- It is now possible to set additional HTTP headers for outgoing ClickHouse requests. This might be useful if, for example, you use a reverse proxy with authorization. ([@teawithfruit](https://github.com/teawithfruit), [#224](https://github.com/ClickHouse/clickhouse-js/pull/224)) + +```ts +const client = createClient({ + additional_headers: { + "X-ClickHouse-User": "clickhouse_user", + "X-ClickHouse-Key": "clickhouse_password", + }, +}); +``` + +## 0.2.8 (Common, Node.js, Web) + +### New features + +- (Web only) Allow to modify Keep-Alive setting (previously always disabled). + Keep-Alive setting **is now enabled by default** for the Web version. + +```ts +import { createClient } from "@clickhouse/client-web"; +const client = createClient({ keep_alive: { enabled: true } }); +``` + +- (Node.js & Web) It is now possible to either specify a list of columns to insert the data into or a list of excluded columns: + +```ts +// Generated query: INSERT INTO mytable (message) FORMAT JSONEachRow +await client.insert({ + table: "mytable", + format: "JSONEachRow", + values: [{ message: "foo" }], + columns: ["message"], +}); + +// Generated query: INSERT INTO mytable (* EXCEPT (message)) FORMAT JSONEachRow +await client.insert({ + table: "mytable", + format: "JSONEachRow", + values: [{ id: 42 }], + columns: { except: ["message"] }, +}); +``` + +See also the new examples: + +- [Including specific columns](../../examples/insert_specific_columns.ts) or [excluding certain ones instead](../../examples/insert_exclude_columns.ts) +- [Leveraging this feature](../../examples/insert_ephemeral_columns.ts) when working with + [ephemeral columns](https://clickhouse.com/docs/en/sql-reference/statements/create/table#ephemeral) + ([#217](https://github.com/ClickHouse/clickhouse-js/issues/217)) + +## 0.2.7 (Common, Node.js, Web) + +### New features + +- (Node.js only) `X-ClickHouse-Summary` response header is now parsed when working with `insert`/`exec`/`command` methods. + See the [related test](../../packages/client-node/__tests__/integration/node_summary.test.ts) for more details. + NB: it is guaranteed to be correct only for non-streaming scenarios. + Web version does not currently support this due to CORS limitations. ([#210](https://github.com/ClickHouse/clickhouse-js/issues/210)) + +### Bug fixes + +- Drain insert response stream in Web version - required to properly work with `async_insert`, especially in the Cloudflare Workers context. + +## 0.2.6 (Common, Node.js) + +### New features + +- Added [Parquet format](https://clickhouse.com/docs/en/integrations/data-formats/parquet) streaming support. + See the new examples: + [insert from a file](../../examples/node/insert_file_stream_parquet.ts), + [select into a file](../../examples/node/select_parquet_as_file.ts). + +## 0.2.5 (Common, Node.js, Web) + +### Bug fixes + +- `pathname` segment from `host` client configuration parameter is now handled properly when making requests. + See this [comment](https://github.com/ClickHouse/clickhouse-js/issues/164#issuecomment-1785166626) for more details. + +## 0.2.4 (Node.js only) + +No changes in web/common modules. + +### Bug fixes + +- (Node.js only) Fixed an issue where streaming large datasets could provide corrupted results. See [#171](https://github.com/ClickHouse/clickhouse-js/issues/171) (issue) and [#204](https://github.com/ClickHouse/clickhouse-js/pull/204) (PR) for more details. + +## 0.2.3 (Node.js only) + +No changes in web/common modules. + +### Bug fixes + +- (Node.js only) Fixed an issue where the underlying socket was closed every time after using `insert` with a `keep_alive` option enabled, which led to performance limitations. See [#202](https://github.com/ClickHouse/clickhouse-js/issues/202) for more details. ([@varrocs](https://github.com/varrocs)) + +## 0.2.2 (Common, Node.js & Web) + +### New features + +- Added `default_format` setting, which allows to perform `exec` calls without `FORMAT` clause. + +## 0.2.1 (Common, Node.js & Web) + +### Breaking changes + +Date objects in query parameters are now serialized as time-zone-agnostic Unix timestamps (NNNNNNNNNN[.NNN], optionally with millisecond-precision) instead of datetime strings without time zones (YYYY-MM-DD HH:MM:SS[.MMM]). This means the server will receive the same absolute timestamp the client sent even if the client's time zone and the database server's time zone differ. Previously, if the server used one time zone and the client used another, Date objects would be encoded in the client's time zone and decoded in the server's time zone and create a mismatch. + +For instance, if the server used UTC (GMT) and the client used PST (GMT-8), a Date object for "2023-01-01 13:00:00 **PST**" would be encoded as "2023-01-01 13:00:00.000" and decoded as "2023-01-01 13:00:00 **UTC**" (which is 2023-01-01 **05**:00:00 PST). Now, "2023-01-01 13:00:00 PST" is encoded as "1672606800000" and decoded as "2023-01-01 **21**:00:00 UTC", the same time the client sent. + +## 0.2.0 (web platform support) + +Introduces web client (using native [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) +and [WebStream](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API) APIs) +without Node.js modules in the common interfaces. No polyfills are required. + +Web client is confirmed to work with Chrome/Firefox/CloudFlare workers. + +It is now possible to implement new custom connections on top of `@clickhouse/client-common`. + +The client was refactored into three packages: + +- `@clickhouse/client-common`: all possible platform-independent code, types and interfaces +- `@clickhouse/client-web`: new web (or non-Node.js env) connection, uses native fetch. +- `@clickhouse/client`: Node.js connection as it was before. + +### Node.js client breaking changes + +- Changed `ping` method behavior: it will not throw now. + Instead, either `{ success: true }` or `{ success: false, error: Error }` is returned. +- Log level configuration parameter is now explicit instead of `CLICKHOUSE_LOG_LEVEL` environment variable. + Default is `OFF`. +- `query` return type signature changed to is `BaseResultSet` (no functional changes) +- `exec` return type signature changed to `ExecResult` (no functional changes) +- `insert` params argument type changed to `InsertParams` (no functional changes) +- Experimental `schema` module is removed + +### Web client known limitations + +- Streaming for select queries works, but it is disabled for inserts (on the type level as well). +- KeepAlive is disabled and not configurable yet. +- Request compression is disabled and configuration is ignored. Response compression works. +- No logging support yet. + +## 0.1.1 + +## New features + +- Expired socket detection on the client side when using Keep-Alive. If a potentially expired socket is detected, + and retry is enabled in the configuration, both socket and request will be immediately destroyed (before sending the data), + and the client will recreate the request. See `ClickHouseClientConfigOptions.keep_alive` for more details. Disabled by default. +- Allow disabling Keep-Alive feature entirely. +- `TRACE` log level. + +## Examples + +#### Disable Keep-Alive feature + +```ts +const client = createClient({ + keep_alive: { + enabled: false, + }, +}); +``` + +#### Retry on expired socket + +```ts +const client = createClient({ + keep_alive: { + enabled: true, + // should be slightly less than the `keep_alive_timeout` setting in server's `config.xml` + // default is 3s there, so 2500 milliseconds seems to be a safe client value in this scenario + // another example: if your configuration has `keep_alive_timeout` set to 60s, you could put 59_000 here + socket_ttl: 2500, + retry_on_expired_socket: true, + }, +}); +``` + +## 0.1.0 + +## Breaking changes + +- `connect_timeout` client setting is removed, as it was unused in the code. + +## New features + +- `command` method is introduced as an alternative to `exec`. + `command` does not expect user to consume the response stream, and it is destroyed immediately. + Essentially, this is a shortcut to `exec` that destroys the stream under the hood. + Consider using `command` instead of `exec` for DDLs and other custom commands which do not provide any valuable output. + +Example: + +```ts +// incorrect: stream is not consumed and not destroyed, request will be timed out eventually +await client.exec("CREATE TABLE foo (id String) ENGINE Memory"); + +// correct: stream does not contain any information and just destroyed +const { stream } = await client.exec( + "CREATE TABLE foo (id String) ENGINE Memory", +); +stream.destroy(); + +// correct: same as exec + stream.destroy() +await client.command("CREATE TABLE foo (id String) ENGINE Memory"); +``` + +### Bug fixes + +- Fixed delays on subsequent requests after calling `insert` that happened due to unclosed stream instance when using low number of `max_open_connections`. See [#161](https://github.com/ClickHouse/clickhouse-js/issues/161) for more details. +- Request timeouts internal logic rework (see [#168](https://github.com/ClickHouse/clickhouse-js/pull/168)) + +## 0.0.16 + +- Fix NULL parameter binding. + As HTTP interface expects `\N` instead of `'NULL'` string, it is now correctly handled for both `null` + and _explicitly_ `undefined` parameters. See the [test scenarios](https://github.com/ClickHouse/clickhouse-js/blob/f1500e188600d85ddd5ee7d2a80846071c8cf23e/__tests__/integration/select_query_binding.test.ts#L273-L303) for more details. + +## 0.0.15 + +### Bug fixes + +- Fix Node.JS 19.x/20.x timeout error (@olexiyb) + +## 0.0.14 + +### New features + +- Added support for `JSONStrings`, `JSONCompact`, `JSONCompactStrings`, `JSONColumnsWithMetadata` formats (@andrewzolotukhin). + +## 0.0.13 + +### New features + +- `query_id` can be now overridden for all main client's methods: `query`, `exec`, `insert`. + +## 0.0.12 + +### New features + +- `ResultSet.query_id` contains a unique query identifier that might be useful for retrieving query metrics from `system.query_log` +- `User-Agent` HTTP header is set according to the [language client spec](https://docs.google.com/document/d/1924Dvy79KXIhfqKpi1EBVY3133pIdoMwgCQtZ-uhEKs/edit#heading=h.ah33hoz5xei2). + For example, for client version 0.0.12 and Node.js runtime v19.0.4 on Linux platform, it will be `clickhouse-js/0.0.12 (lv:nodejs/19.0.4; os:linux)`. + If `ClickHouseClientConfigOptions.application` is set, it will be prepended to the generated `User-Agent`. + +### Breaking changes + +- `client.insert` now returns `{ query_id: string }` instead of `void` +- `client.exec` now returns `{ stream: Stream.Readable, query_id: string }` instead of just `Stream.Readable` + +## 0.0.11, 2022-12-08 + +### Breaking changes + +- `log.enabled` flag was removed from the client configuration. +- Use `CLICKHOUSE_LOG_LEVEL` environment variable instead. Possible values: `OFF`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`. + Currently, there are only debug messages, but we will log more in the future. + +For more details, see PR [#110](https://github.com/ClickHouse/clickhouse-js/pull/110) + +## 0.0.10, 2022-11-14 + +### New features + +- Remove request listeners synchronously. + [#123](https://github.com/ClickHouse/clickhouse-js/issues/123) + +## 0.0.9, 2022-10-25 + +### New features + +- Added ClickHouse session_id support. + [#121](https://github.com/ClickHouse/clickhouse-js/pull/121) + +## 0.0.8, 2022-10-18 + +### New features + +- Added SSL/TLS support (basic and mutual). + [#52](https://github.com/ClickHouse/clickhouse-js/issues/52) + +## 0.0.7, 2022-10-18 + +### Bug fixes + +- Allow semicolons in select clause. + [#116](https://github.com/ClickHouse/clickhouse-js/issues/116) + +## 0.0.6, 2022-10-07 + +### New features + +- Add JSONObjectEachRow input/output and JSON input formats. + [#113](https://github.com/ClickHouse/clickhouse-js/pull/113) + +## 0.0.5, 2022-10-04 + +### Breaking changes + +- Rows abstraction was renamed to ResultSet. +- now, every iteration over `ResultSet.stream()` yields `Row[]` instead of a single `Row`. + Please check out [an example](https://github.com/ClickHouse/clickhouse-js/blob/c86c31dada8f4845cd4e6843645177c99bc53a9d/examples/select_streaming_on_data.ts) + and [this PR](https://github.com/ClickHouse/clickhouse-js/pull/109) for more details. + These changes allowed us to significantly reduce overhead on select result set streaming. + +### New features + +- [split2](https://www.npmjs.com/package/split2) is no longer a package dependency. diff --git a/packages/client-node/package.json b/packages/client-node/package.json index d275b92de..d8e0941e9 100644 --- a/packages/client-node/package.json +++ b/packages/client-node/package.json @@ -15,13 +15,14 @@ }, "private": false, "engines": { - "node": ">=16" + "node": ">=20" }, "main": "dist/index.js", "types": "dist/index.d.ts", "files": [ "dist", - "skills" + "skills", + "CHANGELOG.md" ], "agents": { "skills": [ @@ -34,14 +35,14 @@ "path": "./skills/clickhouse-js-node-troubleshooting" }, { - "name": "clickhouse-js-node-rowbinary-parser", - "path": "./skills/clickhouse-js-node-rowbinary-parser" + "name": "clickhouse-js-node-rowbinary", + "path": "./skills/clickhouse-js-node-rowbinary" } ] }, "scripts": { "pack": "npm pack", - "prepack": "rm -rf skills && cp ../../README.md ../../LICENSE . && cp -r ../../skills . && RBP=skills/clickhouse-js-node-rowbinary-parser && rm -rf $RBP/tests $RBP/node_modules $RBP/dist $RBP/package.json $RBP/package-lock.json $RBP/tsconfig.json $RBP/tsconfig.build.json $RBP/vitest.config.ts $RBP/.gitignore $RBP/LICENSE $RBP/eval_result*.md", + "prepack": "rm -rf skills && cp ../../README.md ../../LICENSE . && cp -r ../../skills . && RBP=skills/clickhouse-js-node-rowbinary && rm -rf $RBP/tests $RBP/node_modules $RBP/dist $RBP/package.json $RBP/package-lock.json $RBP/tsconfig.json $RBP/tsconfig.build.json $RBP/vitest.config.ts $RBP/.gitignore $RBP/LICENSE $RBP/eval_result*.md", "typecheck": "tsc --noEmit", "lint": "eslint --max-warnings=0 .", "lint:fix": "eslint . --fix", diff --git a/packages/client-web/CHANGELOG.md b/packages/client-web/CHANGELOG.md new file mode 100644 index 000000000..7ae3e0373 --- /dev/null +++ b/packages/client-web/CHANGELOG.md @@ -0,0 +1,1342 @@ +# 1.23.0 + +## Migration Notes + +- Node.js 26.x was added to the CI matrix, and Node.js 18.x is no longer supported. The `engines.node` floor of `@clickhouse/client` (previously `>=16`) and `@clickhouse/datatype-parser` (previously `>=18.0.0`) was raised to `>=20`. Node.js 20.x, 22.x, 24.x, and 26.x are supported and exercised in CI. + +- The `@clickhouse/client-common` package is deprecated. `@clickhouse/client` (Node.js) and `@clickhouse/client-web` (Web) no longer depend on it; the shared code is now bundled into each client package. Everything previously importable from `@clickhouse/client-common` should be imported from `@clickhouse/client` or `@clickhouse/client-web` instead. The `@clickhouse/client-common` package itself will no longer receive updates. ([#845]) + +- The `parseColumnType` function and its `SimpleColumnTypes` companion (exported from `@clickhouse/client`, `@clickhouse/client-web`, and `@clickhouse/client-common`) are deprecated and slated for removal in a future major version. They are superseded by the new standalone [`@clickhouse/datatype-parser`](https://www.npmjs.com/package/@clickhouse/datatype-parser) package (`parseDataType` plus its `Node` AST), which parses the full ClickHouse data-type grammar and emits an AST that mirrors the server's. ([#893]) + +## New features + +- (Node.js) Added a RowBinary reader library and agent skill under [`skills/clickhouse-js-node-rowbinary-parser`](../../skills/clickhouse-js-node-rowbinary-parser). It ships type-specific, monomorphizable building blocks for decoding `RowBinary` / `RowBinaryWithNames` / `RowBinaryWithNamesAndTypes` streams (full-buffer and chunked), plus a skill that guides an agent to generate bespoke high-performance parsers from a query's column types. The skill is bundled into `@clickhouse/client` (registered in `agents.skills`) and is also published independently as the [`@clickhouse/rowbinary`](https://www.npmjs.com/package/@clickhouse/rowbinary) package. A matching RowBinary writer is planned. ([#864]) + +- Published the [`@clickhouse/datatype-parser`](https://www.npmjs.com/package/@clickhouse/datatype-parser) package: a small, dependency-free standalone parser for ClickHouse data-type strings (the kind sent in the types row of `RowBinaryWithNamesAndTypes`, e.g. `Array(Nullable(UInt64))`, `Tuple(a UInt8, b String)`, `Enum8('a' = 1)`). It is a faithful port of the server's `ParserDataType` and emits a JSON AST that is byte-identical to the server's `EXPLAIN AST json = 1` data-type subtree. It supersedes the deprecated `parseColumnType` (see Migration Notes). ([#893]) + +- (Node.js, `@experimental`) Added an additive `connection?: Connection` option to `createClient` that lets a caller plug an externally-built backend `Connection`-like object in place of the default HTTP(S) factory. Only supposed to be used for testing the `chDB` integration. ([#879]) + +- Added `ClickHouseSettingsInterface`, a package-neutral structural counterpart to `ClickHouseSettings`, exported from `@clickhouse/client`, `@clickhouse/client-web`, and `@clickhouse/client-common`. It is identical to `ClickHouseSettings` except that its index signature omits `SettingsMap` (a class with a private member, which TypeScript compares nominally). Because each client package now bundles its own copy of the common module, their `ClickHouseSettings` types are mutually unassignable; `ClickHouseSettingsInterface` is structurally identical across all three packages and assignable into each package's `ClickHouseSettings`, so a consumer that shares a single settings-producing helper across both the Node.js and Web clients can type it against this one type without casts. Values typed as `SettingsMap` cannot be carried through it — use `ClickHouseSettings` if you need them. ([#889]) + +# 1.22.0 + +## New features + +- (Node.js) The `compression.request` / `compression.response` client options now accept an explicit codec via an object, in addition to the existing boolean: `true` keeps gzip (backwards compatible), and `{ codec: "zstd" }` selects zstd. The object form is intentionally extensible for future codecs and codec-specific options. zstd typically yields a similar-or-better ratio than gzip at noticeably lower CPU cost (gzip/DEFLATE is comparatively CPU-heavy and decompressed single-threaded by the ClickHouse server), and it uses the built-in `zlib` zstd support, so it requires **Node.js >= 22.15.0** (`@clickhouse/client` throws a clear error at client creation otherwise). Response decompression is driven by the server's actual `Content-Encoding`, so it degrades gracefully. The request object form also accepts an optional `level` (`{ codec, level }`) to set the codec-specific compression level (zlib level for gzip, zstd compression level for zstd); the response compression level is controlled by the server. Supported only by `@clickhouse/client` (Node.js); `@clickhouse/client-web` rejects the `zstd` codec at client creation. + +- (Node.js) Brotli (`{ codec: "br" }`) is now supported for `compression.request` / `compression.response`, alongside gzip and zstd. Unlike zstd, Brotli is available on every supported Node.js version (no minimum-version requirement). The `compression.request` option is a per-codec discriminated union, so each codec exposes its own tuning option: a `level` for gzip/zstd, a `quality` for Brotli (`{ codec: "br", quality }`). When omitted, Brotli defaults to quality 4 for request bodies, since zlib's brotli default of 11 (max) is far too slow for a streaming insert path. Response decompression follows the server's `Content-Encoding`. Supported only by `@clickhouse/client` (Node.js). + +## Internal changes (`@clickhouse/client-common`) + +> These only affect code that imports the low-level connection primitives from the deprecated `@clickhouse/client-common` package directly (e.g. a custom `Connection` implementation). The `createClient` `compression` option is unchanged and fully backwards compatible — if you only use `@clickhouse/client` or `@clickhouse/client-web`, you are not affected. + +To carry the codec (and its optional compression level) instead of a bare on/off flag, the internal compression representation changed shape: + +- `CompressionSettings.compress_request` / `decompress_response` are no longer `boolean`. They are now a normalized codec object or `undefined` (disabled): `{ codec: "gzip" | "zstd"; level?: number } | { codec: "br"; quality?: number }` for the request, `{ codec: "gzip" | "zstd" | "br" }` for the response (response compression options are chosen by the server). `getConnectionParams` normalizes the public request option into this form (`true` → `{ codec: "gzip" }`). +- `withCompressionHeaders` now takes `request_compression_codec` / `response_compression_codec` (a `CompressionMethod | undefined`) instead of the boolean `enable_request_compression` / `enable_response_compression`; the codec value is also the `Content-Encoding` / `Accept-Encoding` it emits. +- `withHttpSettings` now takes the response codec object (`{ codec } | undefined`) instead of a `boolean`. +- New exported types: `CompressionMethod`, `RequestCompression`, `ResponseCompression`. + +Why: a single `boolean` could not express which codec to use or its level, and a separate level field on `CompressionSettings` would have mixed a codec-specific option into the shared type. Discriminating by codec keeps each codec's options on the codec it belongs to. + +## Documentation + +- Added two **tracer adapter recipes** to [`docs/howto/tracing.md`](../../docs/howto/tracing.md) and [`examples/node/coding/otel_tracing.ts`](../../examples/node/coding/otel_tracing.ts), demonstrating how common OpenTelemetry auto-instrumentation options compose as thin userland wrappers around the `tracer` API instead of being baked into the client: `requireParentSpan` (skip ClickHouse spans when there is no active parent span — e.g. background health checks) and suppressing the duplicate nested HTTP spans emitted by `@opentelemetry/instrumentation-http` (via `suppressTracing` from `@opentelemetry/core`). + +# 1.21.0 + +## New features + +- The tracer API (unreleased, introduced in [#776]) now follows the [OpenTelemetry database semantic conventions](https://opentelemetry.io/docs/specs/semconv/db/sql/) and matches the attribute vocabulary of the Rust client ([clickhouse-rs](https://github.com/ClickHouse/clickhouse-rs)); see [`docs/howto/tracing.md`](../../docs/howto/tracing.md) for the documentation. In particular ([#828]): + - Spans now carry `db.system.name` (instead of `db.system`), `server.address` + `server.port` (instead of a combined `host:port`), `clickhouse.request.query_id` / `clickhouse.request.session_id` (instead of `clickhouse.query_id` / `clickhouse.session_id`), `clickhouse.response.format` on `query` and `clickhouse.request.format` on `insert` (instead of `clickhouse.format`), and `db.operation.name` + `db.collection.name` on `insert` (instead of `clickhouse.table`). + - The span status is left unset on success (per the OTEL spec recommendation for client spans, previously set to `OK`); on failure, the span gets the `error.type` attribute (the error class name) and, for server-side errors, `clickhouse.error.code` (the numeric ClickHouse error code). + - Spans record response-side attributes: `db.response.status_code` (HTTP status) and, when the `X-ClickHouse-Summary` header is available, `clickhouse.summary.*` counters (`read_rows`, `written_rows`, etc.). + - `query()` now emits two spans: `clickhouse.query` covers the HTTP request lifetime and ends as soon as the response headers are received; a child `clickhouse.query.stream` span is handed to the `ResultSet` and tracks the stream consumption, ending when the response is fully read, closed, or fails - with the final `clickhouse.response.decoded_bytes` and (for row-streaming) `db.response.returned_rows` metrics. This separation makes it easy to distinguish the original request duration from a stream that may never end (e.g. tailing a live table). + - Fixed a span leak in the Web `ResultSet.stream()` path: if the underlying fetch response stream was aborted (e.g. due to a network error), the `clickhouse.query.stream` span was never ended. The TransformStream now handles both source-stream aborts and consumer-side cancellations via a `cancel` callback. + - The `insert` span records `clickhouse.request.sent_rows` for array-based inserts. + +- Added a `use_multipart_params_auto` client option (default: `false`). When enabled, `query()` automatically sends `query_params` as `multipart/form-data` body parts (the same mechanism as `use_multipart_params`) once their URL-encoded length exceeds 4096 characters, avoiding HTTP 414/400 errors from HTTP intermediaries (nginx, AWS ALB, CloudFront) caused by over-long URLs - for example, a large `IN` list or a high-dimensional vector embedding. Smaller parameter payloads remain in the URL query string, so existing behavior is unchanged unless the threshold is crossed. `use_multipart_params: true` still forces multipart for all queries regardless of size. This does not change the server's per-value size limit, which is governed by `http_max_field_value_size`. Supported on both `@clickhouse/client` and `@clickhouse/client-web`, and overridable per request via `use_multipart_params_auto` on `query()`. Ported from [clickhouse-connect#789](https://github.com/ClickHouse/clickhouse-connect/pull/789). ([#827]) + +```ts +const client = createClient({ use_multipart_params_auto: true }); + +await client.query({ + query: "SELECT * FROM events WHERE id IN {ids:Array(UInt64)}", + // Sent in the URL when small, auto-promoted to the multipart body when large + query_params: { ids: veryLargeArrayOfIds }, +}); +``` + +- Added a `use_multipart_params` client option (default: `false`). When enabled, `query()` sends `query_params` as `multipart/form-data` body parts (with the SQL moved into a `query` part) instead of URL query-string entries, avoiding HTTP 400 errors caused by over-long URLs when parameters contain large arrays (25K+ values). All other URL search params (database, query_id, settings, session_id, role) remain in the URL. Supported on both `@clickhouse/client` and `@clickhouse/client-web`, and overridable per request via `use_multipart_params` on `query()`. ([#825]) + +```ts +const client = createClient({ use_multipart_params: true }); + +await client.query({ + query: "SELECT * FROM events WHERE id IN {ids:Array(UInt64)}", + query_params: { ids: veryLargeArrayOfIds }, + // Per-request override is also supported: + // use_multipart_params: false, +}); +``` + +[#776]: https://github.com/ClickHouse/clickhouse-js/pull/776 +[#825]: https://github.com/ClickHouse/clickhouse-js/pull/825 +[#827]: https://github.com/ClickHouse/clickhouse-js/pull/827 +[#828]: https://github.com/ClickHouse/clickhouse-js/pull/828 +[#845]: https://github.com/ClickHouse/clickhouse-js/pull/845 +[#864]: https://github.com/ClickHouse/clickhouse-js/pull/864 +[#889]: https://github.com/ClickHouse/clickhouse-js/pull/889 +[#893]: https://github.com/ClickHouse/clickhouse-js/pull/893 + +## Bug Fixes + +- The client now checks the `X-ClickHouse-Exception-Code` response header to detect server errors even when the HTTP status code indicates success. In some scenarios (for example, when an exception occurs while streaming the response progress in headers, or with certain proxy setups), ClickHouse responds with HTTP 200 but sets the `X-ClickHouse-Exception-Code` header. Previously, such responses were treated as successful, and the exception text could surface as malformed response data; now the request is rejected with a parsed `ClickHouseError` (with the proper `code` and `type`), consistent with non-2xx error responses. This applies to both the Node.js and Web clients. ([#554], supersedes [#350], related issue: [#332]) + +[#554]: https://github.com/ClickHouse/clickhouse-js/pull/554 +[#350]: https://github.com/ClickHouse/clickhouse-js/pull/350 +[#332]: https://github.com/ClickHouse/clickhouse-js/issues/332 + +# 1.20.0 + +## New Features + +- Added an optional **tracer API** that the user can pass through the client config (`tracer`) and that gets called around key lifecycle operations (`query`, `command`, `exec`, `insert`, `ping`). The `ClickHouseTracer` interface is a structural subset of the OpenTelemetry `Tracer`/`Span` APIs, so a raw OTEL tracer (`trace.getTracer(...)`) can be passed to the client as-is - but the client itself ships no tracing dependency. Each operation runs inside `tracer.startActiveSpan(...)`, so auto-instrumented child spans nest under the ClickHouse operation spans; for OpenTelemetry, this requires the `AsyncLocalStorageContextManager` to be registered (the default in the OpenTelemetry Node.js SDK). Tracer exceptions are NOT caught, so a broken tracer will break client operations. See [`docs/howto/tracing.md`](../../docs/howto/tracing.md) for the full surface description, and [`examples/node/coding/otel_tracing.ts`](../../examples/node/coding/otel_tracing.ts) for a runnable Node.js example. ([#776]) + +```ts +import { createClient } from "@clickhouse/client"; +import { trace } from "@opentelemetry/api"; + +// a raw OpenTelemetry tracer is structurally compatible - no adapter needed +const client = createClient({ + url: "http://localhost:8123", + tracer: trace.getTracer("@clickhouse/client"), +}); +``` + +## Migration Notes + +- TypeScript: `ClickHouseLogLevel` is now exported as a literal numeric union type (`0 | 1 | 2 | 3 | 4 | 127`) instead of a TypeScript `enum` type. If you were assigning arbitrary `number` values to `ClickHouseLogLevel`, you may need to narrow/cast those values during migration. + +## Improvements + +- Added TypeScript typings for the remaining HTTP-specific ClickHouse settings, so they are now suggested by autocomplete when used in `clickhouse_settings`: `buffer_size`, `compress`, `decompress`, `quota_key`, and `stacktrace` (in addition to the existing `wait_end_of_query`, `default_format`, `session_timeout`, and `session_check`). + +```ts +await client.query({ + query: "SELECT 1", + clickhouse_settings: { + // Buffer the entire response on the server before sending it to the client + wait_end_of_query: 1, + buffer_size: "1048576", + }, +}); +``` + +## Bug Fixes + +- (Node.js only) Fixed a race condition in `ResultSet.json()` and `ResultSet.stream()` on `JSONEachRow` (and other streamable) result sets where calling `json()` on a fast/small response could throw `Stream has been already consumed` if the underlying stream ended between internal `readableEnded` checks. The consumption guard has been hardened: the stream is now shielded through a single `consume()` path that marks the result set as consumed in the appropriate branches, after format validation, so a successful `json()` call no longer races against the stream finishing. ([#603]) + +[#603]: https://github.com/ClickHouse/clickhouse-js/pull/603 + +# 1.19.0 + +## Improvements + +- Re-exported the `ResponseHeaders` type from `@clickhouse/client` and `@clickhouse/client-web`. Previously this type was only available from `@clickhouse/client-common`; it is now part of the public re-export surface of both flavored packages, alongside the other commonly used types. This is part of an ongoing effort to make `@clickhouse/client-common` an internal-only package so downstream consumers can depend solely on `@clickhouse/client` or `@clickhouse/client-web`. ([#758]) + +[#758]: https://github.com/ClickHouse/clickhouse-js/pull/758 +[#776]: https://github.com/ClickHouse/clickhouse-js/pull/776 + +## Bug Fixes + +- **Enum type parsing now correctly unescapes backslash escape sequences in enum names.** Previously, `parseEnumType` returned enum names with raw escape sequences (e.g., `f\'` instead of `f'`). Now it properly decodes escape sequences including `\'` (single quote), `\\` (backslash), `\n` (newline), `\t` (tab), and `\r` (carriage return). This matches the behavior of ClickHouse string literals and ensures consistency with how the client encodes strings when sending data to the server. If you were relying on the previous incorrect behavior where backslash escape sequences were preserved in enum names, you will need to update your code to handle properly unescaped values. + +Example: + +```ts +// Before (incorrect): +parseEnumType({ + columnType: "Enum8('f\\'' = 1)", + sourceType: "Enum8('f\\'' = 1)", +}); +// returned: { values: { 1: "f\\'" } } // with backslash + +// After (correct): +parseEnumType({ + columnType: "Enum8('f\\'' = 1)", + sourceType: "Enum8('f\\'' = 1)", +}); +// returns: { values: { 1: "f'" } } // unescaped +``` + +# 1.18.5 + +## Improvements + +- (Node.js only) Added `max_response_headers_size` client option that forwards the [`maxHeaderSize`](https://nodejs.org/api/http.html#httprequesturl-options-callback) option to the underlying `http(s).request` call. This raises the per-request limit on the total size of HTTP response headers received from the server (Node.js default is ~16 KB). It is most useful when running long-running queries with `send_progress_in_http_headers` enabled — the `X-ClickHouse-Progress` headers accumulate over the lifetime of the request and can exceed the default limit, causing the request to fail with `HPE_HEADER_OVERFLOW`. Setting this option avoids the need to use the global `--max-http-header-size` Node.js CLI flag or the `NODE_OPTIONS` environment variable. Has no effect for the Web client (which uses `fetch`) and no effect when a custom `http_agent` is configured with a request implementation that does not honor the option. + +```ts +const client = createClient({ + request_timeout: 400_000, + max_response_headers_size: 1024 * 1024, // accept up to 1 MiB of response headers + clickhouse_settings: { + send_progress_in_http_headers: 1, + http_headers_progress_interval_ms: "110000", + }, +}); +``` + +- The `@clickhouse/client` npm package now ships embedded AI-agent skills, `clickhouse-js-node-coding` and `clickhouse-js-node-troubleshooting`, under `node_modules/@clickhouse/client/skills/`. These skills are also declared in the `agents.skills` field of the package manifest for discovery tools that scan `node_modules`. This allows agentic coding tools to load focused, Node-client-specific coding and troubleshooting guidance without any additional setup. ([#682]) + +[#682]: https://github.com/ClickHouse/clickhouse-js/pull/682 + +# 1.18.4 + +A release-infrastructure-only version bump (no user-facing changes). See 1.18.5 for the next release with user-facing improvements. + +# 1.18.3 + +## Improvements + +- Added `keep_alive.eagerly_destroy_stale_sockets` option (Node.js only, default: `false`). When enabled, sockets that have been idle for longer than `idle_socket_ttl` are destroyed immediately before each request, rather than waiting for the idle timeout to fire. This helps reclaim stale sockets during event loop delays, where the timeout callback may not run on time. + +```ts +const client = createClient({ + keep_alive: { + enabled: true, + idle_socket_ttl: 2500, + eagerly_destroy_stale_sockets: true, + }, +}); +``` + +- Added auto-detection and warning when `request_timeout` is high (> 60 seconds) but progress headers are not configured. Long-running queries may fail with socket hang-up errors if they exceed the load balancer idle timeout. The client now warns users to enable `send_progress_in_http_headers` and `http_headers_progress_interval_ms` settings to prevent such issues. + +```ts +// This will now trigger a warning +const client = createClient({ + request_timeout: 120_000, // 120 seconds + // send_progress_in_http_headers is not configured +}); + +// ✓ Properly configured to avoid load balancer timeouts +const client = createClient({ + request_timeout: 400_000, + clickhouse_settings: { + send_progress_in_http_headers: 1, + http_headers_progress_interval_ms: "110000", // ~10s below LB timeout + }, +}); +``` + +# 1.18.2 + +## Improvements + +- Added a helping `WARN` level log message with a suggestion to check the `keep_alive` configuration if the client receives an `ECONNRESET` error from the server, which can happen when the server closes idle connections after a certain timeout, and the client tries to reuse such a connection from the pool. This can be especially helpful for new users who might not be aware of this aspect of HTTP connection management. The log message is only emitted if the `keep_alive` option is enabled in the client configuration, and it includes the server's keep-alive timeout value (if available) to assist with troubleshooting. ([#597](https://github.com/ClickHouse/clickhouse-js/pull/597)) + +How to reproduce the issue that triggers the log message: + +```ts +const client = createClient({ + // ... + keep_alive: { + enabled: true, + // ❌ DON'T SET THIS VALUE SO HIGH IN PRODUCTION + idle_socket_ttl: 1_000_000, + }, + log: { + level: ClickHouseLogLevel.WARN, // to see the warning logs + }, +}); + +for (let i = 0; i < 1000; i++) { + await client.ping({ + // To use a regular query instead of the /ping endpoint + // which might be configured differently on the server side + // and have different timeout settings. + select: true, + }); + + // Wait long enough to let the server close the idle connection, + // but not too long to let the client remove it from the pool, + // in other words try to hit the scenario when the race condition + // happens between the server closing the connection and the client + // trying to reuse it. + await sleep(SERVER_KEEP_ALIVE_TIMEOUT_MS - 100); +} +``` + +Example log message: + +```json +{ + "message": "Ping: idle socket TTL is greater than server keep-alive timeout, try setting idle socket TTL to a value lower than the server keep-alive timeout to prevent unexpected connection resets, see https://github.com/ClickHouse/clickhouse-js/blob/main/docs/howto/keep_alive_timeout.md for more details.", + "args": { + "operation": "Ping", + "connection_id": "8dc1c9bd-7895-49b1-8a95-276470151c65", + "query_id": "beee95af-2e83-4dcb-8e1e-045bd61f4985", + "request_id": "8dc1c9bd-7895-49b1-8a95-276470151c65:2", + "socket_id": "8dc1c9bd-7895-49b1-8a95-276470151c65:1", + "server_keep_alive_timeout_ms": 10000, + "idle_socket_ttl": 15000 + }, + "module": "HTTP Adapter" +} +``` + +# 1.18.1 + +## Improvements + +- Setting `log.level` default value to `ClickHouseLogLevel.WARN` instead of `ClickHouseLogLevel.OFF` to provide better visibility into potential issues without overwhelming users with too much information by default. + +```ts +const client = createClient({ + // ... + log: { + level: ClickHouseLogLevel.WARN, // default is now ClickHouseLogLevel.WARN instead of ClickHouseLogLevel.OFF + }, +}); +``` + +- Logging is now lazy, which means that the log messages will only be constructed if the log level is appropriate for the message. This can improve performance in cases where constructing the log message is expensive, and the log level is set to ignore such messages. See `ClickHouseLogLevel` enum for the complete list of log levels. ([#520]) + +```ts +const client = createClient({ + // ... + log: { + level: ClickHouseLogLevel.TRACE, // to log everything available down to the network level events + }, +}); +``` + +- Enhanced the logging of the HTTP request / socket lifecycle with additional trace messages and context such as Connection ID (UUID) and Request ID and Socket ID that embed the connection ID for ease of tracing the logs of a particular request across the connection lifecycle. To enable such logs, set the `log.level` config option to `ClickHouseLogLevel.TRACE`. ([#567]) + +```console +[2026-02-25T09:19:13.511Z][TRACE][@clickhouse/client][Connection] Insert: received 'close' event, 'free' listener removed +Arguments: { + operation: 'Insert', + connection_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c', + query_id: '9dfda627-39a2-41a6-9fc9-8f8716574826', + request_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:3', + socket_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:2', + event: 'close' +} +[2026-02-25T09:19:13.502Z][TRACE][@clickhouse/client][Connection] Query: reusing socket +Arguments: { + operation: 'Query', + connection_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c', + query_id: 'ad0127e8-b1c7-4ed6-9681-c0162f7a0ea9', + request_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:4', + socket_id: 'da3c9796-5dc5-46ef-83b0-ed1f4422094c:2', + usage_count: 1 +} +``` + +- A step towards structured logging: the client now passes rich context to the logger `args` parameter (e.g. `connection_id`, `query_id`, `request_id`, `socket_id`). ([#576]) + +## Deprecated API + +- The `drainStream` utility function is now deprecated, as the client will handle draining the stream internally when needed. Use `client.command()` instead, which will handle draining the stream internally when needed. ([#578]) + +- The `sleep` utility function is now deprecated, as it is not intended to be used outside of the client implementation. Use `setTimeout` directly or a more full-featured utility library if you need additional features like cancellation or timers management. ([#578]) + +[#520]: https://github.com/ClickHouse/clickhouse-js/pull/520 +[#567]: https://github.com/ClickHouse/clickhouse-js/pull/567 +[#576]: https://github.com/ClickHouse/clickhouse-js/pull/576 +[#578]: https://github.com/ClickHouse/clickhouse-js/pull/578 + +# 1.18.0 + +A beta version. See 1.18.1 for the stable release. + +# 1.17.0 + +## New features + +- Added `http_status_code` to query, insert, and exec commands ([#525], [Kinzeng]) +- Fixed `ignore_error_response` not getting passed when using `command` ([#536], [Kinzeng]) + +[#525]: https://github.com/ClickHouse/clickhouse-js/pull/525 +[#536]: https://github.com/ClickHouse/clickhouse-js/pull/536 + +# 1.16.0 + +## New features + +- Added support for the new [Disposable API] (a.k.a the `using` keyword) (#500) + +[Disposable API]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/using + +```ts +async function main() { + using resultSet = await client.query(…); + + // some code that can throw + // but thanks to `using` the resultSet will still get disposed + + // resultSet is also automatically disposed here by calling [Symbol.dispose] +} +``` + +Without the new `using` keyword it is required to wrap the code that might leak expensive resources like sockets and big buffers in ` try / finally` + +```ts +async function main() { + let client + try { + client = await createClient(…); + // some code that can throw + } finally { + if (client) { + await client.close() + } + } +} +``` + +# 1.15.0 + +## New features + +- Added support for [BigInt] values in query parameters. ([#487], @dalechyn) + +[#487]: https://github.com/ClickHouse/clickhouse-js/pull/487 +[BigInt]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/BigInt + +# 1.14.0 + +## New features + +- It is now possible to specify custom `parse` and `stringify` functions that will be used instead of the standard `JSON.parse` and `JSON.stringify` methods for JSON serialization/deserialization when working with `JSON*` family formats. See `ClickHouseClientConfigOptions.json`, and a new [custom_json_handling] example for more details. ([#481], [looskie]) +- (Node.js only) Added an `ignore_error_response` param to `ClickHouseClient.exec`, which allows callers to manually handle request errors on the application side. ([#483], [Kinzeng]) + +[#481]: https://github.com/ClickHouse/clickhouse-js/pull/481 +[#483]: https://github.com/ClickHouse/clickhouse-js/pull/483 +[looskie]: https://github.com/looskie +[Kinzeng]: https://github.com/Kinzeng +[custom_json_handling]: https://github.com/ClickHouse/clickhouse-js/blob/1.14.0/examples/custom_json_handling.ts + +# 1.13.0 + +## New features + +- Server-side exceptions that occur in the middle of the HTTP stream are now handled correctly. This requires [ClickHouse 25.11+](https://github.com/ClickHouse/ClickHouse/pull/88818). Previous ClickHouse versions are unaffected by this change. ([#478]) + +## Improvements + +- `TupleParam` constructor now accepts a readonly array to permit more usages. ([#465], [Malien]) + +## Bug fixes + +- Fixed boolean value formatting in query parameters. Boolean values within `Array`, `Tuple`, and `Map` types are now correctly formatted as `TRUE`/`FALSE` instead of `1`/`0` to ensure proper type compatibility with ClickHouse. ([#475], [baseballyama]) + +[#465]: https://github.com/ClickHouse/clickhouse-js/pull/465 +[#475]: https://github.com/ClickHouse/clickhouse-js/pull/475 +[#478]: https://github.com/ClickHouse/clickhouse-js/pull/478 +[Malien]: https://github.com/Malien +[baseballyama]: https://github.com/baseballyama + +# 1.12.1 + +## Improvements + +- Improved performance of `toSearchParams`. ([#449], [twk]) + +## Other + +- Added Node.js 24.x to the CI matrix. Node.js 18.x was removed from the CI due to [EOL](https://endoflife.date/nodejs). + +[#449]: https://github.com/ClickHouse/clickhouse-js/pull/449 +[twk]: https://github.com/twk + +# 1.12.0 + +## Types + +- Add missing `allow_experimental_join_condition` to `ClickHouseSettings` typing. ([#430], [looskie]) +- Fixed `JSONEachRowWithProgress` TypeScript flow after the breaking changes in [ClickHouse 25.1]. `RowOrProgress` now has an additional variant: `SpecialEventRow`. The library now additionally exports the `parseError` method, and newly added `isRow` / `isException` type guards. See the updated [JSONEachRowWithProgress example] ([#443]) +- Added missing `allow_experimental_variant_type` (24.1+), `allow_experimental_dynamic_type` (24.5+), `allow_experimental_json_type` (24.8+), `enable_json_type` (25.3+), `enable_time_time64_type` (25.6+) to `ClickHouseSettings` typing. ([#445]) + +## Improvements + +- Add a warning on a socket closed without fully consuming the stream (e.g., when using `query` or `exec` method). ([#441]) +- (Node.js only) An option to use a simple SELECT query for ping checks instead of `/ping` endpoint. See the new optional argument to the `ClickHouseClient.ping` method and `PingParams` typings. Note that the Web version always used a SELECT query by default, as the `/ping` endpoint does not support CORS, and that cannot be changed. ([#442]) + +## Other + +- The project now uses [Codecov] instead of SonarCloud for code coverage reports. ([#444]) + +[#430]: https://github.com/ClickHouse/clickhouse-js/pull/430 +[#441]: https://github.com/ClickHouse/clickhouse-js/pull/441 +[#442]: https://github.com/ClickHouse/clickhouse-js/pull/442 +[#443]: https://github.com/ClickHouse/clickhouse-js/pull/443 +[#444]: https://github.com/ClickHouse/clickhouse-js/pull/444 +[#445]: https://github.com/ClickHouse/clickhouse-js/pull/445 +[looskie]: https://github.com/looskie +[ClickHouse 25.1]: https://github.com/ClickHouse/ClickHouse/pull/74181 +[JSONEachRowWithProgress example]: https://github.com/ClickHouse/clickhouse-js/blob/main/examples/node/select_json_each_row_with_progress.ts +[Codecov]: https://codecov.io/gh/ClickHouse/clickhouse-js + +# 1.11.2 (Common, Node.js) + +A minor release to allow further investigation regarding uncaught error issues with [#410]. + +## Types + +- Added missing `lightweight_deletes_sync` typing to `ClickHouseSettings` ([#422], [pratimapatel2008]) + +## Improvements (Node.js) + +- Added a new configuration option: `capture_enhanced_stack_trace`; see the JS doc in the Node.js client package. Note that it is disabled by default due to a possible performance impact. ([#427]) +- Added more try-catch blocks to the Node.js connection layer. ([#427]) + +[#410]: https://github.com/ClickHouse/clickhouse-js/pull/410 +[#422]: https://github.com/ClickHouse/clickhouse-js/pull/422 +[#427]: https://github.com/ClickHouse/clickhouse-js/pull/427 +[pratimapatel2008]: https://github.com/pratimapatel2008 + +# 1.11.1 (Common, Node.js, Web) + +## Bug fixes + +- Fixed an issue with URLEncoded special characters in the URL configuration for username or password. ([#407](https://github.com/ClickHouse/clickhouse-js/issues/407)) + +## Improvements + +- Added support for streaming on 32-bit platforms. ([#403](https://github.com/ClickHouse/clickhouse-js/pull/403), [shevchenkonik](https://github.com/shevchenkonik)) + +# 1.11.0 (Common, Node.js, Web) + +## New features + +- It is now possible to provide custom HTTP headers when calling the `query`/`insert`/`command`/`exec` methods using the `http_headers` option. NB: `http_headers` specified this way will override `http_headers` set on the client instance level. ([#394](https://github.com/ClickHouse/clickhouse-js/issues/374), [@DylanRJohnston](https://github.com/DylanRJohnston)) +- (Web only) It is now possible to provide a custom `fetch` implementation to the client. ([#315](https://github.com/ClickHouse/clickhouse-js/issues/315), [@lucacasonato](https://github.com/lucacasonato)) + +# 1.10.1 (Common, Node.js, Web) + +## Bug fixes + +- Fixed `NULL` parameter binding with `Tuple`, `Array`, and `Map` types. ([#374](https://github.com/ClickHouse/clickhouse-js/issues/374)) + +## Improvements + +- `ClickHouseSettings` typings now include `session_timeout` and `session_check` settings. ([#370](https://github.com/ClickHouse/clickhouse-js/issues/370)) + +# 1.10.0 (Common, Node.js, Web) + +## New features + +- Added support for JWT authentication (ClickHouse Cloud feature) in both Node.js and Web API packages. JWT token can be set via `access_token` client configuration option. + + ```ts + const client = createClient({ + // ... + access_token: "", + }); + ``` + + Access token can also be configured via the URL params, e.g., `https://host:port?access_token=...`. + + It is also possible to override the access token for a particular request (see `BaseQueryParams.auth` for more details). + + NB: do not mix access token and username/password credentials in the configuration; the client will throw an error if both are set. + +# 1.9.1 (Node.js only) + +## Bug fixes + +- Fixed an uncaught exception that could happen in case of malformed ClickHouse response when response compression is enabled ([#363](https://github.com/ClickHouse/clickhouse-js/issues/363)) + +# 1.9.0 (Common, Node.js, Web) + +## New features + +- Added `input_format_json_throw_on_bad_escape_sequence` to the `ClickhouseSettings` type. ([#355](https://github.com/ClickHouse/clickhouse-js/pull/355), [@emmanuel-bonin](https://github.com/emmanuel-bonin)) +- The client now exports `TupleParam` wrapper class, allowing tuples to be properly used as query parameters. Added support for JS Map as a query parameter. ([#359](https://github.com/ClickHouse/clickhouse-js/pull/359)) + +## Improvements + +- The client will throw a more informative error if the buffered response is larger than the max allowed string length in V8, which is `2**29 - 24` bytes. ([#357](https://github.com/ClickHouse/clickhouse-js/pull/357)) + +# 1.8.1 (Node.js) + +## Bug fixes + +- When a custom HTTP agent is used, the HTTP or HTTPS request implementation is now correctly chosen based on the URL protocol. ([#352](https://github.com/ClickHouse/clickhouse-js/issues/352)) + +# 1.8.0 (Common, Node.js, Web) + +## New features + +- Added support for specifying roles via request query parameters. See [this example](examples/role.ts) for more details. ([@pulpdrew](https://github.com/pulpdrew), [#328](https://github.com/ClickHouse/clickhouse-js/pull/328)) + +# 1.7.0 (Common, Node.js, Web) + +## Bug fixes + +- (Web only) Fixed an issue where streaming large datasets could provide corrupted results. See [#333](https://github.com/ClickHouse/clickhouse-js/pull/333) (PR) for more details. + +## New features + +- Added `JSONEachRowWithProgress` format support, `ProgressRow` interface, and `isProgressRow` type guard. See [this Node.js example](../../examples/node/select_json_each_row_with_progress.ts) for more details. It should work similarly with the Web version. +- (Experimental) Exposed the `parseColumnType` function that takes a string representation of a ClickHouse type (e.g., `FixedString(16)`, `Nullable(Int32)`, etc.) and returns an AST-like object that represents the type. For example: + + ```ts + for (const type of [ + "Int32", + "Array(Nullable(String))", + `Map(Int32, DateTime64(9, 'UTC'))`, + ]) { + console.log(`##### Source ClickHouse type: ${type}`); + console.log(parseColumnType(type)); + } + ``` + + The above code will output: + + ``` + ##### Source ClickHouse type: Int32 + { type: 'Simple', columnType: 'Int32', sourceType: 'Int32' } + ##### Source ClickHouse type: Array(Nullable(String)) + { + type: 'Array', + value: { + type: 'Nullable', + sourceType: 'Nullable(String)', + value: { type: 'Simple', columnType: 'String', sourceType: 'String' } + }, + dimensions: 1, + sourceType: 'Array(Nullable(String))' + } + ##### Source ClickHouse type: Map(Int32, DateTime64(9, 'UTC')) + { + type: 'Map', + key: { type: 'Simple', columnType: 'Int32', sourceType: 'Int32' }, + value: { + type: 'DateTime64', + timezone: 'UTC', + precision: 9, + sourceType: "DateTime64(9, 'UTC')" + }, + sourceType: "Map(Int32, DateTime64(9, 'UTC'))" + } + ``` + + While the original intention was to use this function internally for `Native`/`RowBinaryWithNamesAndTypes` data formats headers parsing, it can be useful for other purposes as well (e.g., interfaces generation, or custom JSON serializers). + + NB: currently unsupported source types to parse: + - Geo + - (Simple)AggregateFunction + - Nested + - Old/new experimental JSON + - Dynamic + - Variant + +# 1.6.0 (Common, Node.js, Web) + +## New features + +- Added optional `real_time_microseconds` field to the `ClickHouseSummary` interface (see ) + +## Bug fixes + +- Fixed unhandled exceptions produced when calling `ResultSet.json` if the response data was not in fact a valid JSON. ([#311](https://github.com/ClickHouse/clickhouse-js/pull/311)) + +# 1.5.0 (Node.js) + +## New features + +- It is now possible to disable the automatic decompression of the response stream with the `exec` method. See `ExecParams.decompress_response_stream` for more details. ([#298](https://github.com/ClickHouse/clickhouse-js/issues/298)). + +# 1.4.1 (Node.js, Web) + +## Improvements + +- `ClickHouseClient` is now exported as a value from `@clickhouse/client` and `@clickhouse/client-web` packages, allowing for better integration in dependency injection frameworks that rely on IoC (e.g., [Nest.js](https://github.com/nestjs/nest), [tsyringe](https://github.com/microsoft/tsyringe)) ([@mathieu-bour](https://github.com/mathieu-bour), [#292](https://github.com/ClickHouse/clickhouse-js/issues/292)). + +## Bug fixes + +- Fixed a potential socket hang up issue that could happen under 100% CPU load ([#294](https://github.com/ClickHouse/clickhouse-js/issues/294)). + +# 1.4.0 (Node.js) + +## New features + +- (Node.js only) The `exec` method now accepts an optional `values` parameter, which allows you to pass the request body as a `Stream.Readable`. This can be useful in case of custom insert streaming with arbitrary ClickHouse data formats (which might not be explicitly supported and allowed by the client in the `insert` method yet). NB: in this case, you are expected to serialize the data in the stream in the required input format yourself. + +# 1.3.0 (Common, Node.js, Web) + +## New features + +- It is now possible to get the entire response headers object from the `query`/`insert`/`command`/`exec` methods. With `query`, you can access the `ResultSet.response_headers` property; other methods (`insert`/`command`/`exec`) return it as parts of their response objects as well. + For example: + + ```ts + const rs = await client.query({ + query: "SELECT * FROM system.numbers LIMIT 1", + format: "JSONEachRow", + }); + console.log(rs.response_headers["content-type"]); + ``` + + This will print: `application/x-ndjson; charset=UTF-8`. It can be used in a similar way with the other methods. + +## Improvements + +- Re-exported several constants from the `@clickhouse/client-common` package for convenience: + - `SupportedJSONFormats` + - `SupportedRawFormats` + - `StreamableFormats` + - `StreamableJSONFormats` + - `SingleDocumentJSONFormats` + - `RecordsJSONFormats` + +# 1.2.0 (Node.js) + +## New features + +- (Experimental) Added an option to provide a custom HTTP Agent in the client configuration via the `http_agent` option ([#283](https://github.com/ClickHouse/clickhouse-js/issues/283), related: [#278](https://github.com/ClickHouse/clickhouse-js/issues/278)). The following conditions apply if a custom HTTP Agent is provided: + - The `max_open_connections` and `tls` options will have _no effect_ and will be ignored by the client, as it is a part of the underlying HTTP Agent configuration. + - `keep_alive.enabled` will only regulate the default value of the `Connection` header (`true` -> `Connection: keep-alive`, `false` -> `Connection: close`). + - While the idle socket management will still work, it is now possible to disable it completely by setting the `keep_alive.idle_socket_ttl` value to `0`. +- (Experimental) Added a new client configuration option: `set_basic_auth_header`, which disables the `Authorization` header that is set by the client by default for every outgoing HTTP request. One of the possible scenarios when it is necessary to disable this header is when a custom HTTPS agent is used, and the server requires TLS authorization. For example: + + ```ts + const agent = new https.Agent({ + ca: fs.readFileSync("./ca.crt"), + }); + const client = createClient({ + url: "https://server.clickhouseconnect.test:8443", + http_agent: agent, + // With a custom HTTPS agent, the client won't use the default HTTPS connection implementation; the headers should be provided manually + http_headers: { + "X-ClickHouse-User": "default", + "X-ClickHouse-Key": "", + }, + // Authorization header conflicts with the TLS headers; disable it. + set_basic_auth_header: false, + }); + ``` + +NB: It is currently not possible to set the `set_basic_auth_header` option via the URL params. + +If you have feedback on these experimental features, please let us know by creating [an issue](https://github.com/ClickHouse/clickhouse-js/issues) in the repository. + +# 1.1.0 (Common, Node.js, Web) + +## New features + +- Added an option to override the credentials for a particular `query`/`command`/`exec`/`insert` request via the `BaseQueryParams.auth` setting; when set, the credentials will be taken from there instead of the username/password provided during the client instantiation ([#278](https://github.com/ClickHouse/clickhouse-js/issues/278)). +- Added an option to override the `session_id` for a particular `query`/`command`/`exec`/`insert` request via the `BaseQueryParams.session_id` setting; when set, it will be used instead of the session id provided during the client instantiation ([@holi0317](https://github.com/Holi0317), [#271](https://github.com/ClickHouse/clickhouse-js/issues/271)). + +## Bug fixes + +- Fixed the incorrect `ResponseJSON.totals` TypeScript type. Now it correctly matches the shape of the data (`T`, default = `unknown`) instead of the former `Record` definition ([#274](https://github.com/ClickHouse/clickhouse-js/issues/274)). + +# 1.0.2 (Common, Node.js, Web) + +## Bug fixes + +- The `command` method now drains the response stream properly, as the previous implementation could cause the `Keep-Alive` socket to close after each request. +- Removed an unnecessary error log in the `ResultSet.stream` method if the request was aborted or the result set was closed ([#263](https://github.com/ClickHouse/clickhouse-js/issues/263)). + +## Improvements + +- `ResultSet.stream` logs an error via the `Logger` instance, if the stream emits an error event instead of a simple `console.error` call. +- Minor adjustments to the `DefaultLogger` log messages formatting. +- Added missing `rows_before_limit_at_least` to the ResponseJSON type ([@0237h](https://github.com/0237h), [#267](https://github.com/ClickHouse/clickhouse-js/issues/267)). + +# 1.0.1 (Common, Node.js, Web) + +## Bug fixes + +- Fixed the regression where the default HTTP/HTTPS port numbers (80/443) could not be used with the URL configuration ([#258](https://github.com/ClickHouse/clickhouse-js/issues/258)). + +# 1.0.0 (Common, Node.js, Web) + +Formal stable release milestone with a lot of improvements and some [breaking changes](#breaking-changes-in-100). + +Major new features overview: + +- [Advanced TypeScript support for `query` + `ResultSet`](#advanced-typescript-support-for-query--resultset) +- [URL configuration](#url-configuration) + +From now on, the client will follow the [official semantic versioning](https://docs.npmjs.com/about-semantic-versioning) guidelines. + +## Deprecated API + +The following configuration parameters are marked as deprecated: + +- `host` configuration parameter is deprecated; use `url` instead. +- `additional_headers` configuration parameter is deprecated; use `http_headers` instead. + +The client will log a warning if any of these parameters are used. However, it is still allowed to use `host` instead of `url` and `additional_headers` instead of `http_headers` for now; this deprecation is not supposed to break the existing code. + +These parameters will be removed in the next major release (2.0.0). + +See "New features" section for more details. + +## Breaking changes in 1.0.0 + +- `compression.response` is now disabled by default in the client configuration options, as it cannot be used with readonly=1 users, and it was not clear from the ClickHouse error message what exact client option was causing the failing query in this case. If you'd like to continue using response compression, you should explicitly enable it in the client configuration. +- As the client now supports parsing [URL configuration](#url-configuration), you should specify `pathname` as a separate configuration option (as it would be considered as the `database` otherwise). +- (TypeScript only) `ResultSet` and `Row` are now more strictly typed, according to the format used during the `query` call. See [this section](#advanced-typescript-support-for-query--resultset) for more details. +- (TypeScript only) Both Node.js and Web versions now uniformly export correct `ClickHouseClient` and `ClickHouseClientConfigOptions` types, specific to each implementation. Exported `ClickHouseClient` now does not have a `Stream` type parameter, as it was unintended to expose it there. NB: you should still use `createClient` factory function provided in the package. + +## New features in 1.0.0 + +### Advanced TypeScript support for `query` + `ResultSet` + +Client will now try its best to figure out the shape of the data based on the DataFormat literal specified to the `query` call, as well as which methods are allowed to be called on the `ResultSet`. + +Live demo (see the full description below): + +[Screencast](https://github.com/ClickHouse/clickhouse-js/assets/3175289/b66afcb2-3a10-4411-af59-51d2754c417e) + +Complete reference: + +| Format | `ResultSet.json()` | `ResultSet.stream()` | Stream data | `Row.json()` | +| ------------------------------- | --------------------- | --------------------------- | ----------------- | --------------- | +| JSON | ResponseJSON\ | never | never | never | +| JSONObjectEachRow | Record\ | never | never | never | +| All other `JSON*EachRow` | Array\ | Stream\\>\> | Array\\> | T | +| CSV/TSV/CustomSeparated/Parquet | never | Stream\\>\> | Array\\> | never | + +By default, `T` (which represents `JSONType`) is still `unknown`. However, considering `JSONObjectsEachRow` example: prior to 1.0.0, you had to specify the entire type hint, including the shape of the data, manually: + +```ts +type Data = { foo: string }; + +const resultSet = await client.query({ + query: "SELECT * FROM my_table", + format: "JSONObjectsEachRow", +}); + +// pre-1.0.0, `resultOld` has type Record +const resultOld = resultSet.json>(); +// const resultOld = resultSet.json() // incorrect! The type hint should've been `Record` here. + +// 1.0.0, `resultNew` also has type Record; client inferred that it has to be a Record from the format literal. +const resultNew = resultSet.json(); +``` + +This is even more handy in case of streaming on the Node.js platform: + +```ts +const resultSet = await client.query({ + query: "SELECT * FROM my_table", + format: "JSONEachRow", +}); + +// pre-1.0.0 +// `streamOld` was just a regular Node.js Stream.Readable +const streamOld = resultSet.stream(); +// `rows` were `any`, needed an explicit type hint +streamNew.on("data", (rows: Row[]) => { + rows.forEach((row) => { + // without an explicit type hint to `rows`, calling `forEach` and other array methods resulted in TS compiler errors + const t = row.text; + const j = row.json(); // `j` needed a type hint here, otherwise, it's `unknown` + }); +}); + +// 1.0.0 +// `streamNew` is now StreamReadable (Node.js Stream.Readable with a bit more type hints); +// type hint for the further `json` calls can be added here (and removed from the `json` calls) +const streamNew = resultSet.stream(); +// `rows` are inferred as an Array> instead of `any` +streamNew.on("data", (rows) => { + // `row` is inferred as Row + rows.forEach((row) => { + // no explicit type hints required, you can use `forEach` straight away and TS compiler will be happy + const t = row.text; + const j = row.json(); // `j` will be of type Data + }); +}); + +// async iterator now also has type hints +// similarly to the `on(data)` example above, `rows` are inferred as Array> +for await (const rows of streamNew) { + // `row` is inferred as Row + rows.forEach((row) => { + const t = row.text; + const j = row.json(); // `j` will be of type Data + }); +} +``` + +Calling `ResultSet.stream` is not allowed for certain data formats, such as `JSON` and `JSONObjectsEachRow` (unlike `JSONEachRow` and the rest of `JSON*EachRow`, these formats return a single object). In these cases, the client throws an error. However, it was previously not reflected on the type level; now, calling `stream` on these formats will result in a TS compiler error. For example: + +```ts +const resultSet = await client.query("SELECT * FROM table", { + format: "JSON", +}); +const stream = resultSet.stream(); // `stream` is `never` +``` + +Calling `ResultSet.json` also does not make sense on `CSV` and similar "raw" formats, and the client throws. Again, now, it is typed properly: + +```ts +const resultSet = await client.query("SELECT * FROM table", { + format: "CSV", +}); +// `json` is `never`; same if you stream CSV, and call `Row.json` - it will be `never`, too. +const json = resultSet.json(); +``` + +Currently, there is one known limitation: as the general shape of the data and the methods allowed for calling are inferred from the format literal, there might be situations where it will fail to do so, for example: + +```ts +// assuming that `queryParams` has `JSONObjectsEachRow` format inside +async function runQuery( + queryParams: QueryParams, +): Promise> { + const resultSet = await client.query(queryParams); + // type hint here will provide a union of all known shapes instead of a specific one + // inferred shapes: Data[] | ResponseJSON | Record + return resultSet.json(); +} +``` + +In this case, as it is _likely_ that you already know the desired format in advance (otherwise, returning a specific shape like `Record` would've been incorrect), consider helping the client a bit: + +```ts +async function runQuery( + queryParams: QueryParams, +): Promise> { + const resultSet = await client.query({ + ...queryParams, + format: "JSONObjectsEachRow", + }); + // TS understands that it is a Record now + return resultSet.json(); +} +``` + +If you are interested in more details, see the [related test](../../packages/client-node/__tests__/integration/node_query_format_types.test.ts) (featuring a great ESLint plugin [expect-types](https://github.com/JoshuaKGoldberg/eslint-plugin-expect-type)) in the client package. + +### URL configuration + +- Added `url` configuration parameter. It is intended to replace the deprecated `host`, which was already supposed to be passed as a valid URL. +- It is now possible to configure most of the client instance parameters with a URL. The URL format is `http[s]://[username:password@]hostname:port[/database][?param1=value1¶m2=value2]`. In almost every case, the name of a particular parameter reflects its path in the config options interface, with a few exceptions. The following parameters are supported: + +| Parameter | Type | +| ------------------------------------------- | ----------------------------------------------------------------- | +| `pathname` | an arbitrary string. | +| `application_id` | an arbitrary string. | +| `session_id` | an arbitrary string. | +| `request_timeout` | non-negative number. | +| `max_open_connections` | non-negative number, greater than zero. | +| `compression_request` | boolean. See below [1]. | +| `compression_response` | boolean. | +| `log_level` | allowed values: `OFF`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`. | +| `keep_alive_enabled` | boolean. | +| `clickhouse_setting_*` or `ch_*` | see below [2]. | +| `http_header_*` | see below [3]. | +| (Node.js only) `keep_alive_idle_socket_ttl` | non-negative number. | + +[1] For booleans, valid values will be `true`/`1` and `false`/`0`. + +[2] Any parameter prefixed with `clickhouse_setting_` or `ch_` will have this prefix removed and the rest added to client's `clickhouse_settings`. For example, `?ch_async_insert=1&ch_wait_for_async_insert=1` will be the same as: + +```ts +createClient({ + clickhouse_settings: { + async_insert: 1, + wait_for_async_insert: 1, + }, +}); +``` + +Note: boolean values for `clickhouse_settings` should be passed as `1`/`0` in the URL. + +[3] Similar to [2], but for `http_header` configuration. For example, `?http_header_x-clickhouse-auth=foobar` will be an equivalent of: + +```ts +createClient({ + http_headers: { + "x-clickhouse-auth": "foobar", + }, +}); +``` + +**Important: URL will _always_ overwrite the hardcoded values and a warning will be logged in this case.** + +Currently not supported via URL: + +- `log.LoggerClass` +- (Node.js only) `tls_ca_cert`, `tls_cert`, `tls_key`. + +See also: [URL configuration example](../../examples/url_configuration.ts). + +### Performance + +- (Node.js only) Improved performance when decoding the entire set of rows with _streamable_ JSON formats (such as `JSONEachRow` or `JSONCompactEachRow`) by calling the `ResultSet.json()` method. NB: The actual streaming performance when consuming the `ResultSet.stream()` hasn't changed. Only the `ResultSet.json()` method used a suboptimal stream processing in some instances, and now `ResultSet.json()` just consumes the same stream transformer provided by the `ResultSet.stream()` method (see [#253](https://github.com/ClickHouse/clickhouse-js/pull/253) for more details). + +### Miscellaneous + +- Added `http_headers` configuration parameter as a direct replacement for `additional_headers`. Functionally, it is the same, and the change is purely cosmetic, as we'd like to leave an option to implement TCP connection in the future open. + +## 0.3.1 (Common, Node.js, Web) + +### Bug fixes + +- Fixed an issue where query parameters containing tabs or newline characters were not encoded properly. + +## 0.3.0 (Node.js only) + +This release primarily focuses on improving the Keep-Alive mechanism's reliability on the client side. + +### New features + +- Idle sockets timeout rework; now, the client attaches internal timers to idling sockets, and forcefully removes them from the pool if it considers that a particular socket is idling for too long. The intention of this additional sockets housekeeping is to eliminate "Socket hang-up" errors that could previously still occur on certain configurations. Now, the client does not rely on KeepAlive agent when it comes to removing the idling sockets; in most cases, the server will not close the socket before the client does. +- There is a new `keep_alive.idle_socket_ttl` configuration parameter. The default value is `2500` (milliseconds), which is considered to be safe, as [ClickHouse versions prior to 23.11 had `keep_alive_timeout` set to 3 seconds by default](https://github.com/ClickHouse/ClickHouse/commit/1685cdcb89fe110b45497c7ff27ce73cc03e82d1), and `keep_alive.idle_socket_ttl` is supposed to be slightly less than that to allow the client to remove the sockets that are about to expire before the server does so. +- Logging improvements: more internal logs on failing requests; all client methods except ping will log an error on failure now. A failed ping will log a warning, since the underlying error is returned as a part of its result. Client logging still needs to be enabled explicitly by specifying the desired `log.level` config option, as the log level is `OFF` by default. Currently, the client logs the following events, depending on the selected `log.level` value: + - `TRACE` - low-level information about the Keep-Alive sockets lifecycle. + - `DEBUG` - response information (without authorization headers and host info). + - `INFO` - still mostly unused, will print the current log level when the client is initialized. + - `WARN` - non-fatal errors; failed `ping` request is logged as a warning, as the underlying error is included in the returned result. + - `ERROR` - fatal errors from `query`/`insert`/`exec`/`command` methods, such as a failed request. + +### Breaking changes + +- `keep_alive.retry_on_expired_socket` and `keep_alive.socket_ttl` configuration parameters are removed. +- The `max_open_connections` configuration parameter is now 10 by default, as we should not rely on the KeepAlive agent's defaults. +- Fixed the default `request_timeout` configuration value (now it is correctly set to `30_000`, previously `300_000` (milliseconds)). + +### Bug fixes + +- Fixed a bug with Ping that could lead to an unhandled "Socket hang-up" propagation. +- Ensure proper `Connection` header value considering Keep-Alive settings. If Keep-Alive is disabled, its value is now forced to ["close"](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Connection#close). + +## 0.3.0-beta.1 (Node.js only) + +See [0.3.0](#030-nodejs-only). + +## 0.2.10 (Common, Node.js, Web) + +### New features + +- If `InsertParams.values` is an empty array, no request is sent to the server and `ClickHouseClient.insert` short-circuits itself. In this scenario, the newly added `InsertResult.executed` flag will be `false`, and `InsertResult.query_id` will be an empty string. + +### Bug fixes + +- Client no longer produces `Code: 354. inflate failed: buffer error` exception if request compression is enabled and `InsertParams.values` is an empty array (see above). + +## 0.2.9 (Common, Node.js, Web) + +### New features + +- It is now possible to set additional HTTP headers for outgoing ClickHouse requests. This might be useful if, for example, you use a reverse proxy with authorization. ([@teawithfruit](https://github.com/teawithfruit), [#224](https://github.com/ClickHouse/clickhouse-js/pull/224)) + +```ts +const client = createClient({ + additional_headers: { + "X-ClickHouse-User": "clickhouse_user", + "X-ClickHouse-Key": "clickhouse_password", + }, +}); +``` + +## 0.2.8 (Common, Node.js, Web) + +### New features + +- (Web only) Allow to modify Keep-Alive setting (previously always disabled). + Keep-Alive setting **is now enabled by default** for the Web version. + +```ts +import { createClient } from "@clickhouse/client-web"; +const client = createClient({ keep_alive: { enabled: true } }); +``` + +- (Node.js & Web) It is now possible to either specify a list of columns to insert the data into or a list of excluded columns: + +```ts +// Generated query: INSERT INTO mytable (message) FORMAT JSONEachRow +await client.insert({ + table: "mytable", + format: "JSONEachRow", + values: [{ message: "foo" }], + columns: ["message"], +}); + +// Generated query: INSERT INTO mytable (* EXCEPT (message)) FORMAT JSONEachRow +await client.insert({ + table: "mytable", + format: "JSONEachRow", + values: [{ id: 42 }], + columns: { except: ["message"] }, +}); +``` + +See also the new examples: + +- [Including specific columns](../../examples/insert_specific_columns.ts) or [excluding certain ones instead](../../examples/insert_exclude_columns.ts) +- [Leveraging this feature](../../examples/insert_ephemeral_columns.ts) when working with + [ephemeral columns](https://clickhouse.com/docs/en/sql-reference/statements/create/table#ephemeral) + ([#217](https://github.com/ClickHouse/clickhouse-js/issues/217)) + +## 0.2.7 (Common, Node.js, Web) + +### New features + +- (Node.js only) `X-ClickHouse-Summary` response header is now parsed when working with `insert`/`exec`/`command` methods. + See the [related test](../../packages/client-node/__tests__/integration/node_summary.test.ts) for more details. + NB: it is guaranteed to be correct only for non-streaming scenarios. + Web version does not currently support this due to CORS limitations. ([#210](https://github.com/ClickHouse/clickhouse-js/issues/210)) + +### Bug fixes + +- Drain insert response stream in Web version - required to properly work with `async_insert`, especially in the Cloudflare Workers context. + +## 0.2.6 (Common, Node.js) + +### New features + +- Added [Parquet format](https://clickhouse.com/docs/en/integrations/data-formats/parquet) streaming support. + See the new examples: + [insert from a file](../../examples/node/insert_file_stream_parquet.ts), + [select into a file](../../examples/node/select_parquet_as_file.ts). + +## 0.2.5 (Common, Node.js, Web) + +### Bug fixes + +- `pathname` segment from `host` client configuration parameter is now handled properly when making requests. + See this [comment](https://github.com/ClickHouse/clickhouse-js/issues/164#issuecomment-1785166626) for more details. + +## 0.2.4 (Node.js only) + +No changes in web/common modules. + +### Bug fixes + +- (Node.js only) Fixed an issue where streaming large datasets could provide corrupted results. See [#171](https://github.com/ClickHouse/clickhouse-js/issues/171) (issue) and [#204](https://github.com/ClickHouse/clickhouse-js/pull/204) (PR) for more details. + +## 0.2.3 (Node.js only) + +No changes in web/common modules. + +### Bug fixes + +- (Node.js only) Fixed an issue where the underlying socket was closed every time after using `insert` with a `keep_alive` option enabled, which led to performance limitations. See [#202](https://github.com/ClickHouse/clickhouse-js/issues/202) for more details. ([@varrocs](https://github.com/varrocs)) + +## 0.2.2 (Common, Node.js & Web) + +### New features + +- Added `default_format` setting, which allows to perform `exec` calls without `FORMAT` clause. + +## 0.2.1 (Common, Node.js & Web) + +### Breaking changes + +Date objects in query parameters are now serialized as time-zone-agnostic Unix timestamps (NNNNNNNNNN[.NNN], optionally with millisecond-precision) instead of datetime strings without time zones (YYYY-MM-DD HH:MM:SS[.MMM]). This means the server will receive the same absolute timestamp the client sent even if the client's time zone and the database server's time zone differ. Previously, if the server used one time zone and the client used another, Date objects would be encoded in the client's time zone and decoded in the server's time zone and create a mismatch. + +For instance, if the server used UTC (GMT) and the client used PST (GMT-8), a Date object for "2023-01-01 13:00:00 **PST**" would be encoded as "2023-01-01 13:00:00.000" and decoded as "2023-01-01 13:00:00 **UTC**" (which is 2023-01-01 **05**:00:00 PST). Now, "2023-01-01 13:00:00 PST" is encoded as "1672606800000" and decoded as "2023-01-01 **21**:00:00 UTC", the same time the client sent. + +## 0.2.0 (web platform support) + +Introduces web client (using native [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) +and [WebStream](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API) APIs) +without Node.js modules in the common interfaces. No polyfills are required. + +Web client is confirmed to work with Chrome/Firefox/CloudFlare workers. + +It is now possible to implement new custom connections on top of `@clickhouse/client-common`. + +The client was refactored into three packages: + +- `@clickhouse/client-common`: all possible platform-independent code, types and interfaces +- `@clickhouse/client-web`: new web (or non-Node.js env) connection, uses native fetch. +- `@clickhouse/client`: Node.js connection as it was before. + +### Node.js client breaking changes + +- Changed `ping` method behavior: it will not throw now. + Instead, either `{ success: true }` or `{ success: false, error: Error }` is returned. +- Log level configuration parameter is now explicit instead of `CLICKHOUSE_LOG_LEVEL` environment variable. + Default is `OFF`. +- `query` return type signature changed to is `BaseResultSet` (no functional changes) +- `exec` return type signature changed to `ExecResult` (no functional changes) +- `insert` params argument type changed to `InsertParams` (no functional changes) +- Experimental `schema` module is removed + +### Web client known limitations + +- Streaming for select queries works, but it is disabled for inserts (on the type level as well). +- KeepAlive is disabled and not configurable yet. +- Request compression is disabled and configuration is ignored. Response compression works. +- No logging support yet. + +## 0.1.1 + +## New features + +- Expired socket detection on the client side when using Keep-Alive. If a potentially expired socket is detected, + and retry is enabled in the configuration, both socket and request will be immediately destroyed (before sending the data), + and the client will recreate the request. See `ClickHouseClientConfigOptions.keep_alive` for more details. Disabled by default. +- Allow disabling Keep-Alive feature entirely. +- `TRACE` log level. + +## Examples + +#### Disable Keep-Alive feature + +```ts +const client = createClient({ + keep_alive: { + enabled: false, + }, +}); +``` + +#### Retry on expired socket + +```ts +const client = createClient({ + keep_alive: { + enabled: true, + // should be slightly less than the `keep_alive_timeout` setting in server's `config.xml` + // default is 3s there, so 2500 milliseconds seems to be a safe client value in this scenario + // another example: if your configuration has `keep_alive_timeout` set to 60s, you could put 59_000 here + socket_ttl: 2500, + retry_on_expired_socket: true, + }, +}); +``` + +## 0.1.0 + +## Breaking changes + +- `connect_timeout` client setting is removed, as it was unused in the code. + +## New features + +- `command` method is introduced as an alternative to `exec`. + `command` does not expect user to consume the response stream, and it is destroyed immediately. + Essentially, this is a shortcut to `exec` that destroys the stream under the hood. + Consider using `command` instead of `exec` for DDLs and other custom commands which do not provide any valuable output. + +Example: + +```ts +// incorrect: stream is not consumed and not destroyed, request will be timed out eventually +await client.exec("CREATE TABLE foo (id String) ENGINE Memory"); + +// correct: stream does not contain any information and just destroyed +const { stream } = await client.exec( + "CREATE TABLE foo (id String) ENGINE Memory", +); +stream.destroy(); + +// correct: same as exec + stream.destroy() +await client.command("CREATE TABLE foo (id String) ENGINE Memory"); +``` + +### Bug fixes + +- Fixed delays on subsequent requests after calling `insert` that happened due to unclosed stream instance when using low number of `max_open_connections`. See [#161](https://github.com/ClickHouse/clickhouse-js/issues/161) for more details. +- Request timeouts internal logic rework (see [#168](https://github.com/ClickHouse/clickhouse-js/pull/168)) + +## 0.0.16 + +- Fix NULL parameter binding. + As HTTP interface expects `\N` instead of `'NULL'` string, it is now correctly handled for both `null` + and _explicitly_ `undefined` parameters. See the [test scenarios](https://github.com/ClickHouse/clickhouse-js/blob/f1500e188600d85ddd5ee7d2a80846071c8cf23e/__tests__/integration/select_query_binding.test.ts#L273-L303) for more details. + +## 0.0.15 + +### Bug fixes + +- Fix Node.JS 19.x/20.x timeout error (@olexiyb) + +## 0.0.14 + +### New features + +- Added support for `JSONStrings`, `JSONCompact`, `JSONCompactStrings`, `JSONColumnsWithMetadata` formats (@andrewzolotukhin). + +## 0.0.13 + +### New features + +- `query_id` can be now overridden for all main client's methods: `query`, `exec`, `insert`. + +## 0.0.12 + +### New features + +- `ResultSet.query_id` contains a unique query identifier that might be useful for retrieving query metrics from `system.query_log` +- `User-Agent` HTTP header is set according to the [language client spec](https://docs.google.com/document/d/1924Dvy79KXIhfqKpi1EBVY3133pIdoMwgCQtZ-uhEKs/edit#heading=h.ah33hoz5xei2). + For example, for client version 0.0.12 and Node.js runtime v19.0.4 on Linux platform, it will be `clickhouse-js/0.0.12 (lv:nodejs/19.0.4; os:linux)`. + If `ClickHouseClientConfigOptions.application` is set, it will be prepended to the generated `User-Agent`. + +### Breaking changes + +- `client.insert` now returns `{ query_id: string }` instead of `void` +- `client.exec` now returns `{ stream: Stream.Readable, query_id: string }` instead of just `Stream.Readable` + +## 0.0.11, 2022-12-08 + +### Breaking changes + +- `log.enabled` flag was removed from the client configuration. +- Use `CLICKHOUSE_LOG_LEVEL` environment variable instead. Possible values: `OFF`, `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`. + Currently, there are only debug messages, but we will log more in the future. + +For more details, see PR [#110](https://github.com/ClickHouse/clickhouse-js/pull/110) + +## 0.0.10, 2022-11-14 + +### New features + +- Remove request listeners synchronously. + [#123](https://github.com/ClickHouse/clickhouse-js/issues/123) + +## 0.0.9, 2022-10-25 + +### New features + +- Added ClickHouse session_id support. + [#121](https://github.com/ClickHouse/clickhouse-js/pull/121) + +## 0.0.8, 2022-10-18 + +### New features + +- Added SSL/TLS support (basic and mutual). + [#52](https://github.com/ClickHouse/clickhouse-js/issues/52) + +## 0.0.7, 2022-10-18 + +### Bug fixes + +- Allow semicolons in select clause. + [#116](https://github.com/ClickHouse/clickhouse-js/issues/116) + +## 0.0.6, 2022-10-07 + +### New features + +- Add JSONObjectEachRow input/output and JSON input formats. + [#113](https://github.com/ClickHouse/clickhouse-js/pull/113) + +## 0.0.5, 2022-10-04 + +### Breaking changes + +- Rows abstraction was renamed to ResultSet. +- now, every iteration over `ResultSet.stream()` yields `Row[]` instead of a single `Row`. + Please check out [an example](https://github.com/ClickHouse/clickhouse-js/blob/c86c31dada8f4845cd4e6843645177c99bc53a9d/examples/select_streaming_on_data.ts) + and [this PR](https://github.com/ClickHouse/clickhouse-js/pull/109) for more details. + These changes allowed us to significantly reduce overhead on select result set streaming. + +### New features + +- [split2](https://www.npmjs.com/package/split2) is no longer a package dependency. diff --git a/packages/client-web/package.json b/packages/client-web/package.json index ce37e87e3..dec875f8f 100644 --- a/packages/client-web/package.json +++ b/packages/client-web/package.json @@ -15,7 +15,8 @@ }, "private": false, "files": [ - "dist" + "dist", + "CHANGELOG.md" ], "exports": { "types": "./dist/index.d.ts", diff --git a/packages/datatype-parser/CHANGELOG.md b/packages/datatype-parser/CHANGELOG.md new file mode 100644 index 000000000..2806b7b6a --- /dev/null +++ b/packages/datatype-parser/CHANGELOG.md @@ -0,0 +1,20 @@ +# Changelog — `@clickhouse/datatype-parser` + +> This file tracks the standalone `@clickhouse/datatype-parser` package. Entries +> relevant to it (through `@clickhouse/client` 1.23.0) were previously recorded +> in the now-frozen repository-wide [`CHANGELOG.md`](../../CHANGELOG.md). + +# 0.1.3 + +## Migration Notes + +- Node.js 18.x is no longer supported. The `engines.node` floor was raised from `>=18.0.0` to `>=20`. Node.js 20.x, 22.x, 24.x, and 26.x are supported and exercised in CI. ([#906]) + +# 0.1.2 + +## New features + +- Initial published release of `@clickhouse/datatype-parser`: a small, dependency-free standalone parser for ClickHouse data-type strings (the kind sent in the types row of `RowBinaryWithNamesAndTypes`, e.g. `Array(Nullable(UInt64))`, `Tuple(a UInt8, b String)`, `Enum8('a' = 1)`). It is a faithful port of the server's `ParserDataType` and emits a JSON AST that is byte-identical to the server's `EXPLAIN AST json = 1` data-type subtree (`parseDataType` plus its `Node` AST). It supersedes the deprecated `parseColumnType` exported from `@clickhouse/client`, `@clickhouse/client-web`, and `@clickhouse/client-common`. ([#893]) + +[#893]: https://github.com/ClickHouse/clickhouse-js/pull/893 +[#906]: https://github.com/ClickHouse/clickhouse-js/pull/906 diff --git a/packages/datatype-parser/package.json b/packages/datatype-parser/package.json index d410cacc7..7dd90846e 100644 --- a/packages/datatype-parser/package.json +++ b/packages/datatype-parser/package.json @@ -15,11 +15,12 @@ "files": [ "dist", "README.md", - "LICENSE" + "LICENSE", + "CHANGELOG.md" ], "sideEffects": false, "engines": { - "node": ">=18.0.0" + "node": ">=20" }, "repository": { "type": "git", diff --git a/skills/AGENTS.md b/skills/AGENTS.md new file mode 100644 index 000000000..29e8bf10b --- /dev/null +++ b/skills/AGENTS.md @@ -0,0 +1,8 @@ +# Recommendations for AI agents — `skills/` + +Guidance for the shipped agent skills. See the [repo-root `AGENTS.md`](../AGENTS.md) for cross-cutting guidance. The `@clickhouse/rowbinary` skill has its own [`clickhouse-js-node-rowbinary/AGENTS.md`](clickhouse-js-node-rowbinary/AGENTS.md). + +- Each shipped skill must also be listed in the `agents.skills` array of + [`packages/client-node/package.json`](../packages/client-node/package.json) so downstream tooling can + discover it. The [`Skills E2E`](../.github/workflows/e2e-skills.yml) workflow + (`tests/e2e/skills/check.js`) asserts that the packaged tarball contains the declared skills. diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/enums.ts b/skills/clickhouse-js-node-rowbinary-parser/src/enums.ts deleted file mode 100644 index cc2513520..000000000 --- a/skills/clickhouse-js-node-rowbinary-parser/src/enums.ts +++ /dev/null @@ -1,28 +0,0 @@ -import { Cursor, advance } from "./core.js"; - -/** - * Read an `Enum8`: the value's underlying signed `Int8`. The name<->value map - * lives in the column's type, not the bytes. Two strategies, both better than one - * shared name-resolving reader: - * - * - Keep the number: carry the raw Int8 and map to a name only where needed — - * most hot loops never need it. - * - Or generate a per-enum reader with a baked-in constant map, so the JIT can - * monomorphize each enum's decode: - * - * const STATUS = { 1: "active", 2: "closed" } as const; - * const readStatusEnum = (s) => STATUS[readInt8(s) as keyof typeof STATUS]; - */ -export function readEnum8(state: Cursor): number { - return state.view.getInt8(advance(state, 1)); -} - -/** - * Read an `Enum16`: the value's underlying signed `Int16` (2 bytes). The - * name<->value map lives in the column's type definition, not the bytes. Prefer - * keeping the number, or a generated per-enum reader with a baked-in constant - * map so the JIT can optimize each enum's decode independently. - */ -export function readEnum16(state: Cursor): number { - return state.view.getInt16(advance(state, 2), true); -} diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Enum16.test.ts b/skills/clickhouse-js-node-rowbinary-parser/tests/Enum16.test.ts deleted file mode 100644 index 932b3aada..000000000 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Enum16.test.ts +++ /dev/null @@ -1,44 +0,0 @@ -import { describe, expect, it } from "vitest"; -import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readEnum16 } from "../src/enums.js"; - -async function reader(expr: string): Promise { - return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); -} - -describe("readEnum16", () => { - it("decodes a 16-bit underlying value", async () => { - const r = await reader("CAST('big' AS Enum16('small' = 1, 'big' = 300))"); - const value = readEnum16(r); - expect(value).toBe(300); - expect(r.pos).toBe(2); - }); - - it("decodes a negative enum value", async () => { - const value = readEnum16( - await reader("CAST('lo' AS Enum16('lo' = -1000, 'hi' = 1000))"), - ); - expect(value).toBe(-1000); - }); - - describe("advance() edge cases", () => { - it("throws NeedMoreData for every incomplete prefix (0 .. full.length-1)", async () => { - const full = await query( - "SELECT CAST('big' AS Enum16('small' = 1, 'big' = 300)) FORMAT RowBinary", - ); - for (let len = 0; len < full.length; len++) { - const r = new Cursor(full.subarray(0, len)); - let thrown: unknown; - try { - readEnum16(r); - } catch (e) { - thrown = e; - } - expect(thrown, `prefix length ${len} of ${full.length}`).toBe( - NeedMoreData, - ); - } - }); - }); -}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/.gitignore b/skills/clickhouse-js-node-rowbinary/.gitignore similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/.gitignore rename to skills/clickhouse-js-node-rowbinary/.gitignore diff --git a/skills/clickhouse-js-node-rowbinary/AGENTS.md b/skills/clickhouse-js-node-rowbinary/AGENTS.md new file mode 100644 index 000000000..ee5b61e1f --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/AGENTS.md @@ -0,0 +1,44 @@ +# Recommendations for AI agents — `@clickhouse/rowbinary` + +Guidance for the [`clickhouse-js-node-rowbinary`](.) package (the RowBinary codec library and agent skill). See the [repo-root `AGENTS.md`](../../AGENTS.md) for cross-cutting guidance. + +This package has a symmetric reader/writer codebase. + +## Tests + +The tests follow a few conventions worth preserving: + +- **Reader and writer tests are separate files.** Readers are tested in `tests/*.test.ts`; writers in + `tests/*.write.test.ts`. Keep the two independent: a writer test must **never** decode its bytes back + through a reader (and vice versa), so a bug on one side cannot mask a bug on the other. Writer tests + assert the encoded bytes against **live ClickHouse output** as the source of truth. +- **Each case is an isolated `it()` with a fully-inline body.** Write the assertion out per case, e.g. + `expect(encode(writer, value)).toEqual(await query("SELECT … FORMAT RowBinary"))`. Do **not** hide the + assertion behind a thunk-factory helper (`it("name", expectFoo(...))`), and do **not** wrap the query + in a per-file helper — embed the literal SQL inline, including any `SETTINGS` clause, so the full + query is visible in the test. The only shared helpers are the generic `query()` (`tests/clickhouse.ts`, + runs SQL → bytes) and `encode()` (`tests/encode.ts`, value → bytes). Repeating SQL across cases is + fine; reviewability beats DRY here. See [`tests/Integers.write.test.ts`](tests/Integers.write.test.ts) + as the canonical example. + +## No defensive validation in readers/writers + +These are hot-path codecs. **Do not add runtime validation of input values** (`isFinite`, +range/`NaN` checks, type guards, etc.) to the `readX`/`writeX` functions. The data at this level is +expected to be correct, and an invalid value is a programming error — document the precondition in the +JSDoc instead (see `writeUVarint` and the `writeDate*`/`writeDateTime` writers as the canonical +examples). A `Math.round`-style transform that silently shifts a _valid_ value to the wrong encoding is +a correctness bug and must be fixed; rejecting an _invalid_ value is not our job here. + +Two narrow exceptions where a check **is** warranted: + +1. **It keeps the protocol in sync.** A check belongs in only when skipping it would desync the wire + stream — e.g. `writeIPv6` requires exactly 16 bytes because a wrong length shifts every subsequent + field, and `parseIPv6` rejects malformed groups because it can't otherwise produce 16 well-defined + bytes. These guard the _framing_, not the user's data semantics. +2. **The cost is genuinely zero or it can't reach the server.** Pure parse-time helpers (string → + bytes, before anything is on the wire) may validate, since there's no hot loop and no server to fall + back on. + +Otherwise, prefer letting the ClickHouse server reject bad bytes server-side over guarding client-side: +it already validates, and duplicating that on the encode path costs throughput for no real safety. diff --git a/skills/clickhouse-js-node-rowbinary/CHANGELOG.md b/skills/clickhouse-js-node-rowbinary/CHANGELOG.md new file mode 100644 index 000000000..c79cad000 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/CHANGELOG.md @@ -0,0 +1,49 @@ +# Changelog — `@clickhouse/rowbinary` + +> This file tracks the standalone `@clickhouse/rowbinary` package (the RowBinary +> parser library and agent skill). Entries relevant to it (through +> `@clickhouse/client` 1.23.0) were previously recorded in the now-frozen +> repository-wide [`CHANGELOG.md`](../../CHANGELOG.md). + +# 0.2.0 + +## New features + +- Added the RowBinary **writer** — the encode mirror of the reader: type-specific `writeX(sink, value)` building blocks and combinators that compose with no per-element closures, exported from `@clickhouse/rowbinary/writer` (or the per-type modules). ([#911]) + + `writeRows(writeRow)` drives an `Iterable` of rows into a plain `RowBinary` payload. Note its shape: it is a **streaming generator**, not a one-shot `Writer`. It writes into a fixed-size buffer (`bufferSize`, default 64 KiB), yields each batch of whole rows when the buffer fills — rewinding so a half-written row never leaks — and starts a fresh buffer for the rest, so a result larger than the buffer (or an unbounded row source) streams out chunk by chunk. An oversized row grows the buffer (doubling) rather than failing, and per-buffer fill is published on the `@clickhouse/rowbinary:writeRows.flush` `node:diagnostics_channel` for buffer-utilization metrics. ([#915]) + + ```ts + import { + writeRows, + writeTupleNamed, + writeUInt64, + writeString, + } from "@clickhouse/rowbinary/writer"; + + const writeRow = writeTupleNamed({ id: writeUInt64, name: writeString }); + for (const chunk of writeRows(writeRow)(rows, 64 * 1024)) send(chunk); + ``` + + Writer edge-case hardening: `writeDate`/`writeDate32`/`writeDateTime` floor to the calendar day / whole second instead of rounding (so a non-midnight `Date` or sub-second `DateTime` no longer rounds up); `parseIPv6` rejects malformed hex groups instead of silently encoding `0`; and `writeGeometry` validates the discriminant before writing its byte, so an out-of-range value can't leave a partial payload. ([#916]) + +[#911]: https://github.com/ClickHouse/clickhouse-js/pull/911 +[#915]: https://github.com/ClickHouse/clickhouse-js/pull/915 +[#916]: https://github.com/ClickHouse/clickhouse-js/pull/916 + +# 0.1.2 + +## Improvements + +- Bumped the bundled `@clickhouse/datatype-parser` dependency to `0.1.2`. ([#895]) +- Patch release. ([#903]) + +# 0.1.1 + +## New features + +- Initial release of `@clickhouse/rowbinary`: a RowBinary reader library (and the companion agent skill) shipping type-specific, monomorphizable building blocks for decoding `RowBinary` / `RowBinaryWithNames` / `RowBinaryWithNamesAndTypes` streams (full-buffer and chunked), plus a skill that guides an agent to generate bespoke high-performance parsers from a query's column types. The same library is also bundled into `@clickhouse/client` (registered in `agents.skills`). Requires Node.js `>=20`. A matching RowBinary writer is planned. ([#864]) + +[#864]: https://github.com/ClickHouse/clickhouse-js/pull/864 +[#895]: https://github.com/ClickHouse/clickhouse-js/pull/895 +[#903]: https://github.com/ClickHouse/clickhouse-js/pull/903 diff --git a/skills/clickhouse-js-node-rowbinary-parser/EXAMPLES.md b/skills/clickhouse-js-node-rowbinary/EXAMPLES.md similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/EXAMPLES.md rename to skills/clickhouse-js-node-rowbinary/EXAMPLES.md diff --git a/skills/clickhouse-js-node-rowbinary-parser/README.md b/skills/clickhouse-js-node-rowbinary/README.md similarity index 78% rename from skills/clickhouse-js-node-rowbinary-parser/README.md rename to skills/clickhouse-js-node-rowbinary/README.md index 6910f48c2..bb4b74674 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/README.md +++ b/skills/clickhouse-js-node-rowbinary/README.md @@ -1,10 +1,10 @@ -# ClickHouse Node.js RowBinary Parser Generator +# ClickHouse Node.js RowBinary Codec Generator -**If JS had a -O3 compiler flag, this skill would be it.** (for RowBinary parsing) +**If JS had a -O3 compiler flag, this skill would be it.** (for RowBinary read & write) -A skill and a library that lets a coding agent generate bespoke RowBinary parsers on the first pass from the column type definitions of a ClickHouse response. The [spirit](#the-spirit) behind the approach. +A skill and a library that lets a coding agent generate bespoke RowBinary codecs on the first pass from the column type definitions of a ClickHouse response. The [spirit](#the-spirit) behind the approach. -**Reader only** for now. Today this covers reading (decoding) RowBinary streams. A matching RowBinary writer (encoding) is planned. +**Reads and writes.** Both directions are covered: readers (decode bytes → values) and writers (encode values → bytes), split under `src/readers/` and `src/writers/`. The reader path is the more mature one — the writers mirror it type-for-type, with a few decode-only paths (`Dynamic`, `JSON`, the runtime header/compile path, and the columnar typed-array path) not yet mirrored. ## Status @@ -12,7 +12,7 @@ A skill and a library that lets a coding agent generate bespoke RowBinary parser - ✅ Opus 4.8: 71% -> 94.7% pass rate - ✅ Haiku 4.5: 52% -> 86.0% pass rate - ✅ Composer 2.5 Fast: 3x parser performance -- ✅ 469/469 tests +- ✅ 724/724 tests (readers + writers) - ✅ type-checked - ✅ benchmarked @@ -35,7 +35,7 @@ const readOrderRow: Reader = (s) => ({ id: readUInt8(s), uid: formatUUID(readUUID(s)), price: readDecimal64(2)(s), - status: readEnum8(s), + status: readInt8(s), // raw enum int; `readEnum8(map)` resolves it to the name }); ``` @@ -70,14 +70,74 @@ npx skills-npm setup As a skill only: ```bash -npx skills add ClickHouse/clickhouse-js/skills/clickhouse-js-node-rowbinary-parser +npx skills add ClickHouse/clickhouse-js/skills/clickhouse-js-node-rowbinary ``` ```console -> Hey, Claude, tell me what the rowbinary parser skill can do for me. -> A lot! It generates custom, high-performance RowBinary parsers… -> Super, generate a parser for the queries in app/src/model.ts. -< Reading skill clickhouse-js-node-rowbinary-parser… +> Hey, Claude, tell me what the rowbinary skill can do for me. +> A lot! It generates custom, high-performance RowBinary readers and writers… +> Super, generate a reader for the queries in app/src/model.ts. +< Reading skill clickhouse-js-node-rowbinary… +``` + +## Using it with the ClickHouse JS client + +This library only **decodes** the bytes — it doesn't open connections. Pair it +with the official client to fetch a `RowBinary` response and feed the byte chunks +into `streamRowBatches(chunks, readRow)`. + +`RowBinary` isn't one of the formats the client decodes itself, so don't use +`client.query({ format: ... })` for it. Instead use `client.exec({ query })` with +the `FORMAT RowBinary` clause written into the SQL yourself — `exec` hands back the +**raw, undecoded byte stream** of the response, which is exactly what this library +consumes. (Use plain `RowBinary`, not `RowBinaryWithNamesAndTypes`, unless your +reader also skips the leading names/types header.) + +The row reader below is the `orders` example from [EXAMPLES.md](EXAMPLES.md); swap +in the reader the skill generates for your own columns. + +```ts +import { + type Reader, + readUInt8, + readInt8, + readUUID, + formatUUID, + readDecimal64, + type DecimalValue, + streamRowBatches, +} from "@clickhouse/rowbinary"; +import { createClient } from "@clickhouse/client"; + +type OrderRow = { + id: number; + uid: string; + price: DecimalValue; + status: number; +}; + +const readOrderRow: Reader = (s) => ({ + id: readUInt8(s), + uid: formatUUID(readUUID(s)), + price: readDecimal64(2)(s), + status: readInt8(s), // raw enum int; `readEnum8(map)` resolves it to the name +}); + +// `exec` resolves to a Node `Stream.Readable`. It is already an +// `AsyncIterable` (chunks are `Buffer`/`Uint8Array`, which +// `streamRowBatches` normalizes), so pass `stream` straight in: + +const client = createClient(); + +const { stream } = await client.exec({ + query: "SELECT id, uid, price, status FROM orders FORMAT RowBinary", +}); + +for await (const rows of streamRowBatches(stream, readOrderRow)) { + for (const row of rows) console.log(row); // { id, uid, price: [unscaled, scale], status } +} + +await client.close(); ``` ## Why it's worth it @@ -208,18 +268,22 @@ Measure, don't assume. ## Scope -- **In scope:** `RowBinary`, `RowBinaryWithNames`, and +- **In scope (reading):** `RowBinary`, `RowBinaryWithNames`, and `RowBinaryWithNamesAndTypes` decoding for Node.js — full-buffer and streaming (chunked) via `advance()`/`NeedMoreData`, `readRows()`, and the async `streamRowBatches()` (with a built-in small-chunk warning and the optional `coalesceChunks()` debounce filter). -- **Planned:** RowBinary **writing / encoding** (the inverse of everything above) +- **In scope (writing):** the inverse encode path — a `writeX` mirroring every + `readX`, appending bytes to a `Sink`, plus `writeRows()`. Imported from + `@clickhouse/rowbinary/writer`. A handful of decode-only paths are not yet + mirrored: `Dynamic`, `JSON`, the runtime header/compile path, and the columnar + typed-array path. - **Out of scope (for now):** browsers and Edge runtimes, non-RowBinary formats (JSON / CSV / TSV / Parquet), and big-endian hosts. ## The spirit -A RowBinary parser generator is a narrow thing. But it's built as an instance of +A RowBinary codec generator is a narrow thing. But it's built as an instance of a broader bet about what libraries become once a capable LLM is part of the toolchain. Three shifts, each already visible in this repo: diff --git a/skills/clickhouse-js-node-rowbinary/SKILL.md b/skills/clickhouse-js-node-rowbinary/SKILL.md new file mode 100644 index 000000000..d8cee6a2f --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/SKILL.md @@ -0,0 +1,111 @@ +--- +name: clickhouse-js-node-rowbinary +description: > + Generate TypeScript/JavaScript code that reads/decodes AND writes/encodes + ClickHouse RowBinary streams for the ClickHouse HTTP server. + Use this skill whenever a user wants to parse or produce `RowBinary`, + `RowBinaryWithNames`, or `RowBinaryWithNamesAndTypes`. + Node.js only, doesn't cover browsers. +--- + +# ClickHouse JS RowBinary Codec Generator for Node.js + +This skill generates both directions of the wire format: **readers** (decode +bytes → values) and **writers** (encode values → bytes, the mirror). A given +task normally needs only one side. This file is the shared entry point — the +format gate plus the principles common to both directions; the per-direction +decisions, guidance, and the per-type reference tables live in two sibling files. + +**Pick your side — read only the one you need:** + +- **Decoding a `RowBinary*` response** from ClickHouse into JS values → + **[reader.md](reader.md)**. Streaming vs whole-buffer, row-objects vs columnar, + fixed vs runtime schema, and the per-type reader reference. +- **Encoding JS values into a `RowBinary` payload** to send to ClickHouse → + **[writer.md](writer.md)**. The `Sink`/`writeX` building blocks, `writeRows` + streaming, and the per-type writer reference. + +The per-type code is real, split by direction under `src/readers/` and +`src/writers/`. + +## First: is RowBinary even the right format? + +RowBinary exists for throughput, but it is **not automatically the fastest +path** — match the format to the shape of the data before committing to a +bespoke parser. + +**Prefer a `JSON*` format (e.g. `JSONEachRow`) when** the result is mostly +strings / JSON-like values that you consume wholesale — randomly accessing +essentially every field, running string/regexp methods on them, treating values +as text. V8's native `JSON.parse` is heavily optimized C++ and builds JS strings +and objects faster than a JS-level RowBinary decoder can; pair it with HTTP +response compression (`gzip` / `zstd`, which crushes JSON's repetitive keys) and +the wire cost shrinks too. + +**RowBinary clearly wins when** the result is dominated by: + +- **Wide numerics** — `Int128`/`Int256`/`UInt128`/`UInt256`, + `Decimal128`/`Decimal256`. +- **Binary / fixed-width blobs** — `IPv4`, `IPv6`, `UUID`, `FixedString`. +- **High-volume fixed-width numeric columns** generally, where each value is a + single `DataView` read. + +**Prefer the `Native` format when** columnar load and client-side analytics are +the main goal (fold/scan/filter columns, feed typed arrays to a Worker or WASM). +`Native` is column-major, so it loads straight into one typed array per column +with no transpose. + +For help choosing and consuming a `JSON*` format (or CSV / TSV) instead, use the +**`clickhouse-js-node-coding`** skill. + +## Core guidance (both directions) + +These principles apply whether you are generating a reader or a writer; the +side-specific operational guidance is in [reader.md](reader.md) / +[writer.md](writer.md). + +- **Little-endian only.** RowBinary is little-endian; target x86/ARM. Read and + write every multi-byte number with `DataView` accessors passing a **literal** + `true` for the `littleEndian` flag. + +- **Correct first, then optimize.** First emit a correct codec built from the + plain per-type API. Only after it's correct (and tested) specialize it. Don't + bake performance assumptions in before correctness. + +- **Monomorphize generic/composite types.** Emit specialized, inlined code per + type combination instead of passing functions as arguments where the type is + known ahead of time. + +- **Inline the leaf ops.** The per-type `readX`/`writeX` functions are the + correct, composable reference; the generated codec should INLINE their bodies, + not call them, so the row loop is straight-line with no per-field indirection + (and so the fixed-width coalescing can fold the offset arithmetic together). + +- **Annotate the type per column.** Inlining erases the type structure, so put a + short comment above each column's encode/decode block naming the ClickHouse + type it handles. + +- **Shared scratch is not reentrant.** Some hot methods reuse a module-level + scratch buffer as a write-then-read pair — correct only because the access is + fully synchronous. An `async`/`yield` boundary between populating and reading + it corrupts the value. + +- **TypeScript by default.** Generate TypeScript code and helpers unless the user + explicitly asks for plain JavaScript. + +## Worked examples + +Six end-to-end examples with real speedup are catalogued in [EXAMPLES.md](EXAMPLES.md). + +## Out of scope + +- **JSON / CSV / TSV / Parquet parsing** → use `clickhouse-js-node-coding`. +- **Connection errors, hangs, type mismatches** → use + `clickhouse-js-node-troubleshooting`. +- **Browser / Web Worker / Edge** → `@clickhouse/client-web`. + +## Still Stuck? + +- [ClickHouse RowBinary format](https://clickhouse.com/docs/interfaces/formats#rowbinary) +- [ClickHouse data types](https://clickhouse.com/docs/sql-reference/data-types) +- [ClickHouse JS client docs](https://clickhouse.com/docs/integrations/javascript) diff --git a/skills/clickhouse-js-node-rowbinary-parser/case-studies/iot-rowbinary-vs-json.md b/skills/clickhouse-js-node-rowbinary/case-studies/iot-rowbinary-vs-json.md similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/case-studies/iot-rowbinary-vs-json.md rename to skills/clickhouse-js-node-rowbinary/case-studies/iot-rowbinary-vs-json.md diff --git a/skills/clickhouse-js-node-rowbinary-parser/case-studies/ledger-rowbinary-vs-json.md b/skills/clickhouse-js-node-rowbinary/case-studies/ledger-rowbinary-vs-json.md similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/case-studies/ledger-rowbinary-vs-json.md rename to skills/clickhouse-js-node-rowbinary/case-studies/ledger-rowbinary-vs-json.md diff --git a/skills/clickhouse-js-node-rowbinary-parser/case-studies/logs-json-wins.md b/skills/clickhouse-js-node-rowbinary/case-studies/logs-json-wins.md similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/case-studies/logs-json-wins.md rename to skills/clickhouse-js-node-rowbinary/case-studies/logs-json-wins.md diff --git a/skills/clickhouse-js-node-rowbinary-parser/case-studies/wasm-vs-js.md b/skills/clickhouse-js-node-rowbinary/case-studies/wasm-vs-js.md similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/case-studies/wasm-vs-js.md rename to skills/clickhouse-js-node-rowbinary/case-studies/wasm-vs-js.md diff --git a/skills/clickhouse-js-node-rowbinary-parser/eval_result.md b/skills/clickhouse-js-node-rowbinary/eval_result.md similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/eval_result.md rename to skills/clickhouse-js-node-rowbinary/eval_result.md diff --git a/skills/clickhouse-js-node-rowbinary-parser/eval_result_composer.md b/skills/clickhouse-js-node-rowbinary/eval_result_composer.md similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/eval_result_composer.md rename to skills/clickhouse-js-node-rowbinary/eval_result_composer.md diff --git a/skills/clickhouse-js-node-rowbinary-parser/eval_result_haiku.md b/skills/clickhouse-js-node-rowbinary/eval_result_haiku.md similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/eval_result_haiku.md rename to skills/clickhouse-js-node-rowbinary/eval_result_haiku.md diff --git a/skills/clickhouse-js-node-rowbinary-parser/eval_result_sonnet.md b/skills/clickhouse-js-node-rowbinary/eval_result_sonnet.md similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/eval_result_sonnet.md rename to skills/clickhouse-js-node-rowbinary/eval_result_sonnet.md diff --git a/skills/clickhouse-js-node-rowbinary-parser/package-lock.json b/skills/clickhouse-js-node-rowbinary/package-lock.json similarity index 99% rename from skills/clickhouse-js-node-rowbinary-parser/package-lock.json rename to skills/clickhouse-js-node-rowbinary/package-lock.json index ecaa39cde..36ad3788a 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/package-lock.json +++ b/skills/clickhouse-js-node-rowbinary/package-lock.json @@ -1,12 +1,12 @@ { "name": "@clickhouse/rowbinary", - "version": "0.1.2", + "version": "0.2.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@clickhouse/rowbinary", - "version": "0.1.2", + "version": "0.2.0", "license": "Apache-2.0", "dependencies": { "@clickhouse/datatype-parser": "^0.1.2" diff --git a/skills/clickhouse-js-node-rowbinary-parser/package.json b/skills/clickhouse-js-node-rowbinary/package.json similarity index 62% rename from skills/clickhouse-js-node-rowbinary-parser/package.json rename to skills/clickhouse-js-node-rowbinary/package.json index fc10f2efe..efe5ae522 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/package.json +++ b/skills/clickhouse-js-node-rowbinary/package.json @@ -1,21 +1,23 @@ { "name": "@clickhouse/rowbinary", - "version": "0.1.2", - "description": "RowBinary building blocks for Node.js — read/decode ClickHouse RowBinary / RowBinaryWithNames(AndTypes) streams (a matching writer is planned). Ships with the clickhouse-js-node-rowbinary-parser agent skill.", - "homepage": "https://github.com/ClickHouse/clickhouse-js/tree/main/skills/clickhouse-js-node-rowbinary-parser", + "version": "0.2.0", + "description": "RowBinary building blocks for Node.js — read and write (decode/encode) ClickHouse RowBinary / RowBinaryWithNames(AndTypes) streams. Ships with the clickhouse-js-node-rowbinary agent skill.", + "homepage": "https://github.com/ClickHouse/clickhouse-js/tree/main/skills/clickhouse-js-node-rowbinary", "license": "Apache-2.0", "keywords": [ "clickhouse", "rowbinary", "parser", "decoder", + "encoder", + "writer", "streaming", "skill" ], "repository": { "type": "git", "url": "git+https://github.com/ClickHouse/clickhouse-js.git", - "directory": "skills/clickhouse-js-node-rowbinary-parser" + "directory": "skills/clickhouse-js-node-rowbinary" }, "type": "module", "sideEffects": false, @@ -25,13 +27,17 @@ "publishConfig": { "access": "public" }, - "main": "./dist/reader.js", - "module": "./dist/reader.js", - "types": "./dist/reader.d.ts", + "main": "./dist/readers/reader.js", + "module": "./dist/readers/reader.js", + "types": "./dist/readers/reader.d.ts", "exports": { ".": { - "types": "./dist/reader.d.ts", - "import": "./dist/reader.js" + "types": "./dist/readers/reader.d.ts", + "import": "./dist/readers/reader.js" + }, + "./writer": { + "types": "./dist/writers/writer.d.ts", + "import": "./dist/writers/writer.js" }, "./*": { "types": "./dist/*.d.ts", @@ -42,13 +48,16 @@ "dist", "src", "SKILL.md", + "reader.md", + "writer.md", "README.md", - "EXAMPLES.md" + "EXAMPLES.md", + "CHANGELOG.md" ], "agents": { "skills": [ { - "name": "clickhouse-js-node-rowbinary-parser", + "name": "clickhouse-js-node-rowbinary", "path": "." } ] diff --git a/skills/clickhouse-js-node-rowbinary-parser/SKILL.md b/skills/clickhouse-js-node-rowbinary/reader.md similarity index 54% rename from skills/clickhouse-js-node-rowbinary-parser/SKILL.md rename to skills/clickhouse-js-node-rowbinary/reader.md index 1fb41fb38..02dd1e6f1 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/SKILL.md +++ b/skills/clickhouse-js-node-rowbinary/reader.md @@ -1,46 +1,12 @@ ---- -name: clickhouse-js-node-rowbinary-parser -description: > - Generate TypeScript/JavaScript code that reads and decodes ClickHouse - RowBinary streams from the ClickHouse HTTP server. - Use this skill whenever a user wants to parse `RowBinary`, - `RowBinaryWithNames`, or `RowBinaryWithNamesAndTypes`. - Node.js only, doesn't cover browsers. ---- - -# ClickHouse JS RowBinary Parser Generator for Node.js - -## First: is RowBinary even the right format? - -RowBinary exists for throughput, but it is **not automatically the fastest -path** — match the format to the shape of the data before committing to a -bespoke parser. - -**Prefer a `JSON*` format (e.g. `JSONEachRow`) when** the result is mostly -strings / JSON-like values that you consume wholesale — randomly accessing -essentially every field, running string/regexp methods on them, treating values -as text. V8's native `JSON.parse` is heavily optimized C++ and builds JS strings -and objects faster than a JS-level RowBinary decoder can; pair it with HTTP -response compression (`gzip` / `zstd`, which crushes JSON's repetitive keys) and -the wire cost shrinks too. - -**RowBinary clearly wins when** the result is dominated by: - -- **Wide numerics** — `Int128`/`Int256`/`UInt128`/`UInt256`, - `Decimal128`/`Decimal256`. -- **Binary / fixed-width blobs** — `IPv4`, `IPv6`, `UUID`, `FixedString`. -- **High-volume fixed-width numeric columns** generally, where each value is a - single `DataView` read. - -**Prefer the `Native` format when** columnar load and client-side analytics are -the main goal (fold/scan/filter columns, feed typed arrays to a Worker or WASM). -`Native` is column-major, so it loads straight into one typed array per column -with no transpose. - -For help choosing and consuming a `JSON*` format (or CSV / TSV) instead, use the -**`clickhouse-js-node-coding`** skill. - -## Second: complete buffer, or incremental stream? +# RowBinary reader (decode) for Node.js + +Decoding a `RowBinary` / `RowBinaryWithNames` / `RowBinaryWithNamesAndTypes` +response from ClickHouse into JS values. Read [SKILL.md](SKILL.md) first for the +format gate ("is RowBinary even the right format?") and the principles that +apply to **both** directions; this file covers the decisions and the per-type +reference specific to **reading**. Writing? See [writer.md](writer.md). + +## First: complete buffer, or incremental stream? Decide this before writing the reader — it changes the shape of the code and is a real performance fork. @@ -56,7 +22,7 @@ a real performance fork. The exposed API is streaming by default and requires an optimisation pass. -## Third: row objects, or columnar (typed arrays)? +## Second: row objects, or columnar (typed arrays)? The default output is one object per row (array-of-structs). For a **numeric, fixed-width result that the consumer reads column-wise**, decode instead into one @@ -77,56 +43,39 @@ bandwidth). Measured in `tests/iot.columnar.bench.ts`; rationale in complete-row count is `(chunk.length / stride) | 0`, and the leftover bytes carry to the next chunk. Yield one typed-array batch per chunk, each owning a fresh transferable `ArrayBuffer` (see `streamSensorColumns` in - `src/columnar.ts`). + `src/readers/columnar.ts`). - **Stay row-oriented when** downstream code is row-shaped, the row is string-dominated (columnar's win is numeric — a JS string allocates either way), or the schema is nested/heterogeneous (`Array`/`Map`/`Tuple`). - **Hybrid:** store columnar, expose a lazy `rowAt(i)` accessor that builds an object only for rows actually touched (see `iotRowAt` in `src/examples/iot.ts`). -## Fourth: are the column types known ahead of time? +## Third: are the column types known ahead of time? - **Known (the default).** Generate a straight-line reader specialized to those types — everything below. - **Only at runtime** (the schema varies, or you just want to decode an arbitrary `RowBinaryWithNamesAndTypes` stream). Call - `compileRowBinaryWithNamesAndTypes(cursor)` (`src/rowBinaryWithNamesAndTypes.ts`): + `compileRowBinaryWithNamesAndTypes(cursor)` (`src/readers/rowBinaryWithNamesAndTypes.ts`): it reads the header, folds each column type's AST into a `Reader` - (`astToReader`, `src/compile.ts`; type strings parsed by + (`astToReader`, `src/readers/compile.ts`; type strings parsed by `@clickhouse/datatype-parser`), and returns a `readRows` driver for the rest of the stream. Generic and unoptimized (no codegen), so prefer the specialized path whenever the types are fixed. -## Core guidance - -When generating a parser, follow these: - -- **Little-endian only.** RowBinary is little-endian; target x86/ARM. Read every - multi-byte number with `DataView` accessors passing a **literal** `true` for - the `littleEndian` flag. +## Reader guidance -- **Correct first, then optimize.** First emit a correct reader built from the - plain per-type API. Only after it's correct (and tested) specialize it. Don't - bake performance assumptions in before correctness. - -- **Monomorphize generic/composite types.** Emit specialized, inlined code per - type combination instead of passing functions as arguments where the type - is known ahead of time. +On top of the shared principles in [SKILL.md](SKILL.md), the read path has its own: - **Streaming: throw + restart, not generators.** To signal "need more bytes", a synchronous reader that throws a sentinel (`NeedMoreData`) and restarts the - row beats generators for realistic chunk sizes; + row beats generators for realistic chunk sizes. - **Keep an eye on chunk sizes.** Partial trailing rows, small chunks are a silent throughput killer: `streamRowBatches` warns once when rows-per-chunk falls too low, and `coalesceChunks(source, { minSize, timeoutMs })` merges small chunks in front of it when the source size isn't yours to raise. -- **Shared scratch is not reentrant.** Some hot methods reuse a module-level - scratch buffer as a write-then-read pair — correct only because reads are fully - synchronous. An `async`/`yield` boundary between populating and reading it - corrupts the value. - - **Hoist the cursor into locals.** Prefer the working buffer and view declared once at the top of the generated reader, and keep the read offset in a **local variable**, operating on it directly instead of re-reading from an object. @@ -135,72 +84,43 @@ When generating a parser, follow these: neighbouring fixed-width columns has a known combined size, so bounds-check it ONCE. -- **Inline the leaf reads.** The per-type `readX` functions are the correct, - composable reference; the generated parser should INLINE their bodies, not call - them, so the row reader is straight-line with no per-field indirection (and so - the two points above can fold the offset arithmetic together). - -- **Annotate the decoded type per column.** Inlining erases the type structure, - so put a short comment above each column's decode block naming the ClickHouse - type it reads. - - **Pre-allocate small result arrays.** RowBinary gives every array/map its element count up front (the LEB128 prefix), so DEFAULT is to `new Array(n)`. NOTE: for **large** arrays the application will iterate or compute over repeatedly, prefer `[]` + `push` (faster to traverse in V8) — or a typed array (`Float64Array`…) for numeric elements. -- **TypeScript by default.** Generate TypeScript parsers and helpers unless the - user explicitly asks for plain JavaScript. - -## Type family references - -The readers live as real code under `src/`, split by type family. - -| Result contains (trigger) | Open | -| ---------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Always** — cursor state, `advance()`, `NeedMoreData`, `Reader` | `src/core.ts` | -| LEB128 length/count prefixes for `String`/`Array`/`Map` (`readUVarint`) | `src/varint.ts` | -| `Int8`–`Int256`, `UInt8`–`UInt256` | `src/integers.ts` | -| `Bool` | `src/bool.ts` | -| `Enum8`, `Enum16` | `src/enums.ts` | -| `Float32`, `Float64`, `BFloat16` | `src/floats.ts` | -| `Decimal32/64/128/256`, `Decimal(P, S)` | `src/decimals.ts` | -| `String`, `FixedString(N)` | `src/strings.ts` | -| `UUID` | `src/uuid.ts` | -| `IPv4`, `IPv6` | `src/ip.ts` | -| `Date`, `Date32`, `DateTime`, `DateTime(tz)`, `DateTime64(P[, tz])` | `src/datetime.ts` | -| `Time`, `Time64(P)` | `src/time.ts` | -| `IntervalNanosecond` … `IntervalYear` | `src/interval.ts` | -| `Array(T)`, `Map(K, V)`, `Tuple(...)`, `Nullable(T)`, `Variant(...)`, `QBit(...)` | `src/composite.ts` | -| `Point`, `Ring`, `LineString`, `MultiLineString`, `Polygon`, `MultiPolygon`, `Geometry` | `src/geo.ts` | -| `Dynamic` (and `Variant`/`Interval`/`Nested`/`Dynamic` nested inside it) | `src/dynamic.ts` | -| `JSON` | `src/json.ts` | -| The whole result — loop rows to EOF (`readRows`) | `src/rows.ts` | -| A chunked HTTP response — `streamRowBatches`, `coalesceChunks` | `src/stream.ts` | -| The `RowBinaryWithNamesAndTypes` header — column names + type strings (`readHeader`) | `src/header.ts` | -| Fold one parsed type AST into a `Reader` (`astToReader`) — AST in, reader out | `src/compile.ts` | -| **Types known only at runtime** — compile a whole header into a row reader (`compileRowBinaryWithNamesAndTypes`, `typeStringToReader`) | `src/rowBinaryWithNamesAndTypes.ts` | -| **Numeric/fixed-width result read column-wise** (aggregate/scan/plot, hand to a Worker/WASM) → decode into typed arrays, not row objects (~4x) | `src/columnar.ts` (`streamSensorColumns` — streaming, yields transferable typed-array batches); `decodeIotColumnar` in `src/examples/iot.ts` is the whole-buffer form | -| `LowCardinality(T)` — transparent, decode as `T` | `src/lowCardinality.ts` | -| `SimpleAggregateFunction(f, T)` — transparent, decode as `T` | `src/simpleAggregateFunction.ts` | -| `Nested(...)` — no wire of its own; `Array(Tuple(...))` | `src/nested.ts` | -| `Nothing` — zero-width, never decoded (only wrapped) | `src/nothing.ts` | -| `AggregateFunction(...)` — opaque state; finalize server-side | `src/aggregateFunction.ts` | - -## Worked examples - -Six end-to-end examples with real speedup are catalogued in [EXAMPLES.md](EXAMPLES.md). - -## Out of scope - -- **JSON / CSV / TSV / Parquet parsing** → use `clickhouse-js-node-coding`. -- **Connection errors, hangs, type mismatches** → use - `clickhouse-js-node-troubleshooting`. -- **Browser / Web Worker / Edge** → `@clickhouse/client-web`. - -## Still Stuck? - -- [ClickHouse RowBinary format](https://clickhouse.com/docs/interfaces/formats#rowbinary) -- [ClickHouse data types](https://clickhouse.com/docs/sql-reference/data-types) -- [ClickHouse JS client docs](https://clickhouse.com/docs/integrations/javascript) +## Reader type family references + +The readers live as real code under `src/readers/`, split by type family. + +| Result contains (trigger) | Open | +| ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Always** — cursor state, `advance()`, `NeedMoreData`, `Reader` | `src/readers/core.ts` | +| LEB128 length/count prefixes for `String`/`Array`/`Map` (`readUVarint`) | `src/readers/varint.ts` | +| `Int8`–`Int256`, `UInt8`–`UInt256` | `src/readers/integers.ts` | +| `Bool` | `src/readers/bool.ts` | +| `Enum8`, `Enum16` (resolve to the value's name; `readInt8`/`readInt16` for the raw int) | `src/readers/enums.ts` | +| `Float32`, `Float64`, `BFloat16` | `src/readers/floats.ts` | +| `Decimal32/64/128/256`, `Decimal(P, S)` | `src/readers/decimals.ts` | +| `String`, `FixedString(N)` | `src/readers/strings.ts` | +| `UUID` | `src/readers/uuid.ts` | +| `IPv4`, `IPv6` | `src/readers/ip.ts` | +| `Date`, `Date32`, `DateTime`, `DateTime(tz)`, `DateTime64(P[, tz])` | `src/readers/datetime.ts` | +| `Time`, `Time64(P)` | `src/readers/time.ts` | +| `IntervalNanosecond` … `IntervalYear` | `src/readers/interval.ts` | +| `Array(T)`, `Map(K, V)`, `Tuple(...)`, `Nullable(T)`, `Variant(...)`, `QBit(...)` | `src/readers/composite.ts` | +| `Point`, `Ring`, `LineString`, `MultiLineString`, `Polygon`, `MultiPolygon`, `Geometry` | `src/readers/geo.ts` | +| `Dynamic` (and `Variant`/`Interval`/`Nested`/`Dynamic` nested inside it) | `src/readers/dynamic.ts` | +| `JSON` | `src/readers/json.ts` | +| The whole result — loop rows to EOF (`readRows`) | `src/readers/rows.ts` | +| A chunked HTTP response — `streamRowBatches`, `coalesceChunks` | `src/readers/stream.ts` | +| The `RowBinaryWithNamesAndTypes` header — column names + type strings (`readHeader`) | `src/readers/header.ts` | +| Fold one parsed type AST into a `Reader` (`astToReader`) — AST in, reader out | `src/readers/compile.ts` | +| **Types known only at runtime** — compile a whole header into a row reader (`compileRowBinaryWithNamesAndTypes`, `typeStringToReader`) | `src/readers/rowBinaryWithNamesAndTypes.ts` | +| **Numeric/fixed-width result read column-wise** (aggregate/scan/plot, hand to a Worker/WASM) → decode into typed arrays, not row objects (~4x) | `src/readers/columnar.ts` (`streamSensorColumns` — streaming, yields transferable typed-array batches); `decodeIotColumnar` in `src/examples/iot.ts` is the whole-buffer form | +| `LowCardinality(T)` — transparent, decode as `T` | `src/readers/lowCardinality.ts` | +| `SimpleAggregateFunction(f, T)` — transparent, decode as `T` | `src/readers/simpleAggregateFunction.ts` | +| `Nested(...)` — no wire of its own; `Array(Tuple(...))` | `src/readers/nested.ts` | +| `Nothing` — zero-width, never decoded (only wrapped) | `src/readers/nothing.ts` | +| `AggregateFunction(...)` — opaque state; finalize server-side | `src/readers/aggregateFunction.ts` | diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/examples/carts.ts b/skills/clickhouse-js-node-rowbinary/src/examples/carts.ts similarity index 88% rename from skills/clickhouse-js-node-rowbinary-parser/src/examples/carts.ts rename to skills/clickhouse-js-node-rowbinary/src/examples/carts.ts index 902412be5..faa7352c0 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/examples/carts.ts +++ b/skills/clickhouse-js-node-rowbinary/src/examples/carts.ts @@ -1,8 +1,12 @@ -import { readArray, readNullable, readTupleNamed } from "../composite.js"; -import { type Reader, advance } from "../core.js"; -import { readInt32, readUInt16, readUInt32 } from "../integers.js"; -import { readString } from "../strings.js"; -import { readUVarint } from "../varint.js"; +import { + readArray, + readNullable, + readTupleNamed, +} from "../readers/composite.js"; +import { type Reader, advance } from "../readers/core.js"; +import { readInt32, readUInt16, readUInt32 } from "../readers/integers.js"; +import { readString } from "../readers/strings.js"; +import { readUVarint } from "../readers/varint.js"; /** * Example: a carts table — nested generics. diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/examples/events.ts b/skills/clickhouse-js-node-rowbinary/src/examples/events.ts similarity index 87% rename from skills/clickhouse-js-node-rowbinary-parser/src/examples/events.ts rename to skills/clickhouse-js-node-rowbinary/src/examples/events.ts index b07614523..15d4e7bb5 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/examples/events.ts +++ b/skills/clickhouse-js-node-rowbinary/src/examples/events.ts @@ -1,8 +1,8 @@ -import { type Reader, advance } from "../core.js"; -import { readDateTime } from "../datetime.js"; -import { readUInt64 } from "../integers.js"; -import { readString } from "../strings.js"; -import { readUVarint } from "../varint.js"; +import { type Reader, advance } from "../readers/core.js"; +import { readDateTime } from "../readers/datetime.js"; +import { readUInt64 } from "../readers/integers.js"; +import { readString } from "../readers/strings.js"; +import { readUVarint } from "../readers/varint.js"; /** * Example: a plain events table — the scalar baseline. diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/examples/iot.ts b/skills/clickhouse-js-node-rowbinary/src/examples/iot.ts similarity index 96% rename from skills/clickhouse-js-node-rowbinary-parser/src/examples/iot.ts rename to skills/clickhouse-js-node-rowbinary/src/examples/iot.ts index 24da52536..222ecb315 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/examples/iot.ts +++ b/skills/clickhouse-js-node-rowbinary/src/examples/iot.ts @@ -1,7 +1,7 @@ -import { type Reader, advance } from "../core.js"; -import { readDateTime64P3 } from "../datetime.js"; -import { readFloat32, readFloat64 } from "../floats.js"; -import { readUInt8, readUInt32 } from "../integers.js"; +import { type Reader, advance } from "../readers/core.js"; +import { readDateTime64P3 } from "../readers/datetime.js"; +import { readFloat32, readFloat64 } from "../readers/floats.js"; +import { readUInt8, readUInt32 } from "../readers/integers.js"; /** * Example: a table of IoT sensor readings — the dense, fixed-width NUMERIC case diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/examples/ledger.ts b/skills/clickhouse-js-node-rowbinary/src/examples/ledger.ts similarity index 95% rename from skills/clickhouse-js-node-rowbinary-parser/src/examples/ledger.ts rename to skills/clickhouse-js-node-rowbinary/src/examples/ledger.ts index 3e0dbc63c..15091957c 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/examples/ledger.ts +++ b/skills/clickhouse-js-node-rowbinary/src/examples/ledger.ts @@ -1,10 +1,10 @@ -import { type Reader, advance } from "../core.js"; +import { type Reader, advance } from "../readers/core.js"; import { type DecimalValue, readDecimal64, readDecimal128, -} from "../decimals.js"; -import { readInt64, readUInt128, readUInt256 } from "../integers.js"; +} from "../readers/decimals.js"; +import { readInt64, readUInt128, readUInt256 } from "../readers/integers.js"; /** * Example: a financial ledger — the WIDE-NUMERIC case where RowBinary wins on diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/examples/logs.ts b/skills/clickhouse-js-node-rowbinary/src/examples/logs.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/src/examples/logs.ts rename to skills/clickhouse-js-node-rowbinary/src/examples/logs.ts index 4fd3b233f..0b7a72038 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/examples/logs.ts +++ b/skills/clickhouse-js-node-rowbinary/src/examples/logs.ts @@ -1,7 +1,7 @@ -import { type Reader, advance } from "../core.js"; -import { readDateTime } from "../datetime.js"; -import { readString } from "../strings.js"; -import { readUVarint } from "../varint.js"; +import { type Reader, advance } from "../readers/core.js"; +import { readDateTime } from "../readers/datetime.js"; +import { readString } from "../readers/strings.js"; +import { readUVarint } from "../readers/varint.js"; /** * Example: an application log table — the STRING-HEAVY case where the skill diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/examples/observability.ts b/skills/clickhouse-js-node-rowbinary/src/examples/observability.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/src/examples/observability.ts rename to skills/clickhouse-js-node-rowbinary/src/examples/observability.ts index 7897af584..131a1ac72 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/examples/observability.ts +++ b/skills/clickhouse-js-node-rowbinary/src/examples/observability.ts @@ -4,15 +4,14 @@ import { readNullable, readTupleNamed, readVariant, -} from "../composite.js"; -import { type Reader, advance } from "../core.js"; -import { readDateTime64P3 } from "../datetime.js"; -import { readEnum8 } from "../enums.js"; -import { readFloat64 } from "../floats.js"; -import { readInt64, readUInt64 } from "../integers.js"; -import { readString } from "../strings.js"; -import { formatUUID, formatUUIDTable, readUUID } from "../uuid.js"; -import { readUVarint } from "../varint.js"; +} from "../readers/composite.js"; +import { type Reader, advance } from "../readers/core.js"; +import { readDateTime64P3 } from "../readers/datetime.js"; +import { readFloat64 } from "../readers/floats.js"; +import { readInt8, readInt64, readUInt64 } from "../readers/integers.js"; +import { readString } from "../readers/strings.js"; +import { formatUUID, formatUUIDTable, readUUID } from "../readers/uuid.js"; +import { readUVarint } from "../readers/varint.js"; /** * Example: an observability/events table — the gotcha-heavy one. It packs the @@ -60,7 +59,7 @@ export type ObsRow = { export const readObsRow: Reader = (s) => ({ id: readUInt64(s), ts: readDateTime64P3(s).toISOString(), - level: readEnum8(s), + level: readInt8(s), traceId: formatUUID(readUUID(s)), payload: readVariant([readFloat64, readInt64, readString])(s), tags: readMap(readString, readString)(s), diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/examples/orders.ts b/skills/clickhouse-js-node-rowbinary/src/examples/orders.ts similarity index 77% rename from skills/clickhouse-js-node-rowbinary-parser/src/examples/orders.ts rename to skills/clickhouse-js-node-rowbinary/src/examples/orders.ts index ddf862457..a6169e900 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/examples/orders.ts +++ b/skills/clickhouse-js-node-rowbinary/src/examples/orders.ts @@ -1,8 +1,7 @@ -import { type Reader, advance } from "../core.js"; -import { type DecimalValue, readDecimal64 } from "../decimals.js"; -import { readEnum8 } from "../enums.js"; -import { readUInt8 } from "../integers.js"; -import { formatUUID, formatUUIDTable, readUUID } from "../uuid.js"; +import { type Reader, advance } from "../readers/core.js"; +import { type DecimalValue, readDecimal64 } from "../readers/decimals.js"; +import { readInt8, readUInt8 } from "../readers/integers.js"; +import { formatUUID, formatUUIDTable, readUUID } from "../readers/uuid.js"; /** * Example: an orders table — UUID, Decimal, and Enum (awkward-as-JSON types). @@ -15,9 +14,11 @@ import { formatUUID, formatUUIDTable, readUUID } from "../uuid.js"; * * Shows the parse/format split and faithful values: `uid` is read as raw bytes * then formatted with `formatUUID`; `price` stays the exact `[unscaled, scale]` - * pair (`[1234n, 2]` == 12.34), not a lossy float; `status` decodes to the - * underlying `Int8` value (1/2/3), the name<->value map being type metadata, not - * on the wire. The declared scale `2` is baked into `readDecimal64(2)`. + * pair (`[1234n, 2]` == 12.34), not a lossy float; `status` is read as the raw + * underlying `Int8` (1/2/3) with `readInt8` — a hand-written reader that bakes + * in the schema can skip name resolution, whereas the generic `readEnum8(map)` + * (used by the dynamic/header path) resolves the value to its name. The declared + * scale `2` is baked into `readDecimal64(2)`. */ export type OrderRow = { id: number; @@ -30,7 +31,7 @@ export const readOrderRow: Reader = (s) => ({ id: readUInt8(s), uid: formatUUID(readUUID(s)), price: readDecimal64(2)(s), - status: readEnum8(s), + status: readInt8(s), }); /** diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/examples/profiles.ts b/skills/clickhouse-js-node-rowbinary/src/examples/profiles.ts similarity index 86% rename from skills/clickhouse-js-node-rowbinary-parser/src/examples/profiles.ts rename to skills/clickhouse-js-node-rowbinary/src/examples/profiles.ts index 7a09c3148..a68b4a2c4 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/examples/profiles.ts +++ b/skills/clickhouse-js-node-rowbinary/src/examples/profiles.ts @@ -1,8 +1,8 @@ -import { readArray, readNullable } from "../composite.js"; -import { type Reader, advance } from "../core.js"; -import { readInt32, readUInt32 } from "../integers.js"; -import { readString } from "../strings.js"; -import { readUVarint } from "../varint.js"; +import { readArray, readNullable } from "../readers/composite.js"; +import { type Reader, advance } from "../readers/core.js"; +import { readInt32, readUInt32 } from "../readers/integers.js"; +import { readString } from "../readers/strings.js"; +import { readUVarint } from "../readers/varint.js"; /** * Example: a profiles table — Array and Nullable wrappers. diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/examples/telemetry.ts b/skills/clickhouse-js-node-rowbinary/src/examples/telemetry.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/src/examples/telemetry.ts rename to skills/clickhouse-js-node-rowbinary/src/examples/telemetry.ts index 3cc38a5af..4f6c08fdf 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/examples/telemetry.ts +++ b/skills/clickhouse-js-node-rowbinary/src/examples/telemetry.ts @@ -3,12 +3,12 @@ import { readMap, readNullable, readTupleNamed, -} from "../composite.js"; -import { type Reader, advance } from "../core.js"; -import { readFloat64 } from "../floats.js"; -import { readUInt16, readUInt32 } from "../integers.js"; -import { readString } from "../strings.js"; -import { readUVarint } from "../varint.js"; +} from "../readers/composite.js"; +import { type Reader, advance } from "../readers/core.js"; +import { readFloat64 } from "../readers/floats.js"; +import { readUInt16, readUInt32 } from "../readers/integers.js"; +import { readString } from "../readers/strings.js"; +import { readUVarint } from "../readers/varint.js"; /** * Example: a telemetry table — composite readers that nest. diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/aggregateFunction.ts b/skills/clickhouse-js-node-rowbinary/src/readers/aggregateFunction.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/aggregateFunction.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/aggregateFunction.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/bool.ts b/skills/clickhouse-js-node-rowbinary/src/readers/bool.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/bool.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/bool.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/columnar.ts b/skills/clickhouse-js-node-rowbinary/src/readers/columnar.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/columnar.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/columnar.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/compile.ts b/skills/clickhouse-js-node-rowbinary/src/readers/compile.ts similarity index 93% rename from skills/clickhouse-js-node-rowbinary-parser/src/compile.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/compile.ts index 8e647a896..9f3861be1 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/compile.ts +++ b/skills/clickhouse-js-node-rowbinary/src/readers/compile.ts @@ -115,10 +115,12 @@ export class RowBinaryTypeError extends Error { */ export function astToReader(node: Node): Reader { switch (node.kind) { - case NodeKind.EnumDataType: - // Explicit-value enum: the wire value is the underlying int; the - // name<->value map is metadata we don't need to decode. - return node.name === "Enum16" ? readEnum16 : readEnum8; + case NodeKind.EnumDataType: { + // Explicit-value enum: the wire value is the underlying int, which we + // resolve to its NAME via the `'name' = value` pairs the type carries. + const map = enumNameMap(node); + return node.name === "Enum16" ? readEnum16(map) : readEnum8(map); + } case NodeKind.TupleDataType: return tupleReader(node); case NodeKind.DataType: @@ -185,12 +187,13 @@ function dataTypeReader(node: Node): Reader { return readDecimal128(literalInt(requireArg(node, 0))); case "Decimal256": return readDecimal256(literalInt(requireArg(node, 0))); - // Auto-assigned enums arrive as a plain DataType (no explicit values); the - // wire value is still the underlying int. + // A bare `Enum8`/`Enum16` (no explicit values — only reachable from a + // hand-written type string) has no names to resolve, so the reader falls + // back to the stringified underlying int. case "Enum8": - return readEnum8; + return readEnum8(new Map()); case "Enum16": - return readEnum16; + return readEnum16(new Map()); default: { // Interval (IntervalSecond, IntervalDay, …): all decode to the @@ -247,6 +250,13 @@ function nestedReader(node: Node): Reader { return readNested(fields); } +/** Builds an enum's underlying-int -> name lookup from its `'name' = value` pairs. */ +function enumNameMap(node: Node): ReadonlyMap { + const map = new Map(); + for (const ev of node.values) map.set(Number(ev.value), ev.name); + return map; +} + /** Folds `Decimal(P, S)` to the right width by precision P; scale S drives decoding. */ function decimalReader(node: Node): Reader { const precision = literalInt(requireArg(node, 0)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/composite.ts b/skills/clickhouse-js-node-rowbinary/src/readers/composite.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/composite.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/composite.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/core.ts b/skills/clickhouse-js-node-rowbinary/src/readers/core.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/core.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/core.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/datetime.ts b/skills/clickhouse-js-node-rowbinary/src/readers/datetime.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/datetime.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/datetime.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/decimals.ts b/skills/clickhouse-js-node-rowbinary/src/readers/decimals.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/decimals.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/decimals.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/dynamic.ts b/skills/clickhouse-js-node-rowbinary/src/readers/dynamic.ts similarity index 95% rename from skills/clickhouse-js-node-rowbinary-parser/src/dynamic.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/dynamic.ts index 336ea8e7e..93d42fbd4 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/src/dynamic.ts +++ b/skills/clickhouse-js-node-rowbinary/src/readers/dynamic.ts @@ -15,6 +15,7 @@ import { readUInt256, } from "./integers.js"; import { readBool } from "./bool.js"; +import { readEnum8, readEnum16 } from "./enums.js"; import { readFloat32, readFloat64 } from "./floats.js"; import { readString, readFixedString } from "./strings.js"; import { readUUID } from "./uuid.js"; @@ -143,23 +144,26 @@ export function readDynamicType(state: Cursor): Reader { return readString; case 0x16: return readFixedString(readUVarint(state)); - // Enum8 / Enum16: a count then (name String, value Int8/Int16) pairs. The - // name<->value map is metadata; the stored value is the underlying int. + // Enum8 / Enum16: a count then (name String, value Int8/Int16) pairs. We + // collect them into the name<->value map so the value resolves to its name, + // matching the textual-type path in compile.ts. case 0x17: { const n = readUVarint(state); + const map = new Map(); for (let i = 0; i < n; i++) { - readString(state); - readInt8(state); + const name = readString(state); + map.set(readInt8(state), name); } - return readInt8; + return readEnum8(map); } case 0x18: { const n = readUVarint(state); + const map = new Map(); for (let i = 0; i < n; i++) { - readString(state); - readInt16(state); + const name = readString(state); + map.set(readInt16(state), name); } - return readInt16; + return readEnum16(map); } // Decimals: header carries precision P then scale S (both varint). Only S // matters for decoding; P is consumed and dropped. Returns [unscaled, S]. diff --git a/skills/clickhouse-js-node-rowbinary/src/readers/enums.ts b/skills/clickhouse-js-node-rowbinary/src/readers/enums.ts new file mode 100644 index 000000000..6fa9a05c2 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/readers/enums.ts @@ -0,0 +1,40 @@ +import { type Reader, advance } from "./core.js"; + +/** + * Maps an enum's underlying integer to its name. Built from the type's + * `'name' = value` pairs (which live in the column type, not the wire) by the + * compile step / dynamic decoder and handed to the readers below. + */ +export type EnumNameMap = ReadonlyMap; + +/** + * Read an `Enum8` and resolve it to its NAME — the ergonomic default, since the + * point of an enum is usually the label, not the raw `Int8`. + * + * This is a factory: the name<->value map is metadata carried by the type, not + * the bytes, so it is supplied once (per column) and closed over. A value with + * no matching name falls back to its stringified integer rather than throwing, + * so a decode never blows up on an unexpected wire value. + * + * Need the raw underlying integer instead (e.g. a monomorphized hot path that + * has the schema baked in)? Read it directly with {@link readInt8} — that is the + * fast, allocation-free path the enum's name resolution sits on top of. + */ +export function readEnum8(valueToName: EnumNameMap): Reader { + return (state) => + resolveName(valueToName, state.view.getInt8(advance(state, 1))); +} + +/** + * Read an `Enum16` (2-byte underlying `Int16`) and resolve it to its NAME. See + * {@link readEnum8}; use {@link readInt16} for the raw underlying integer. + */ +export function readEnum16(valueToName: EnumNameMap): Reader { + return (state) => + resolveName(valueToName, state.view.getInt16(advance(state, 2), true)); +} + +function resolveName(valueToName: EnumNameMap, value: number): string { + const name = valueToName.get(value); + return name !== undefined ? name : String(value); +} diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/floats.ts b/skills/clickhouse-js-node-rowbinary/src/readers/floats.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/floats.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/floats.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/geo.ts b/skills/clickhouse-js-node-rowbinary/src/readers/geo.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/geo.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/geo.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/header.ts b/skills/clickhouse-js-node-rowbinary/src/readers/header.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/header.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/header.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/integers.ts b/skills/clickhouse-js-node-rowbinary/src/readers/integers.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/integers.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/integers.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/interval.ts b/skills/clickhouse-js-node-rowbinary/src/readers/interval.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/interval.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/interval.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/ip.ts b/skills/clickhouse-js-node-rowbinary/src/readers/ip.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/ip.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/ip.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/json.ts b/skills/clickhouse-js-node-rowbinary/src/readers/json.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/json.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/json.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/lowCardinality.ts b/skills/clickhouse-js-node-rowbinary/src/readers/lowCardinality.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/lowCardinality.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/lowCardinality.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/nested.ts b/skills/clickhouse-js-node-rowbinary/src/readers/nested.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/nested.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/nested.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/nothing.ts b/skills/clickhouse-js-node-rowbinary/src/readers/nothing.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/nothing.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/nothing.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/reader.ts b/skills/clickhouse-js-node-rowbinary/src/readers/reader.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/reader.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/reader.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/rowBinaryWithNamesAndTypes.ts b/skills/clickhouse-js-node-rowbinary/src/readers/rowBinaryWithNamesAndTypes.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/rowBinaryWithNamesAndTypes.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/rowBinaryWithNamesAndTypes.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/rows.ts b/skills/clickhouse-js-node-rowbinary/src/readers/rows.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/rows.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/rows.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/simpleAggregateFunction.ts b/skills/clickhouse-js-node-rowbinary/src/readers/simpleAggregateFunction.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/simpleAggregateFunction.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/simpleAggregateFunction.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/stream.ts b/skills/clickhouse-js-node-rowbinary/src/readers/stream.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/stream.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/stream.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/strings.ts b/skills/clickhouse-js-node-rowbinary/src/readers/strings.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/strings.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/strings.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/time.ts b/skills/clickhouse-js-node-rowbinary/src/readers/time.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/time.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/time.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/uuid.ts b/skills/clickhouse-js-node-rowbinary/src/readers/uuid.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/uuid.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/uuid.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/src/varint.ts b/skills/clickhouse-js-node-rowbinary/src/readers/varint.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/src/varint.ts rename to skills/clickhouse-js-node-rowbinary/src/readers/varint.ts diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/aggregateFunction.ts b/skills/clickhouse-js-node-rowbinary/src/writers/aggregateFunction.ts new file mode 100644 index 000000000..0d9ce29b5 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/aggregateFunction.ts @@ -0,0 +1,18 @@ +import { type Writer } from "./core.js"; + +/** + * Inverse of `readAggregateFunction`: an `AggregateFunction(func, T…)` column is + * OPAQUE, unframed aggregation state with a layout specific to `func` and the + * server version, so it cannot be produced generically from a value. Build the + * state server-side (the `-State` combinators) rather than encoding it on the + * client. + * + * This writer throws to stop a generic encoder from emitting a misaligned row. + */ +export const writeAggregateFunction: Writer = () => { + throw new Error( + "RowBinary: AggregateFunction is opaque, unframed aggregation state with no " + + "length prefix — not generically encodable. Produce the state server-side " + + "(the -State combinators) instead of encoding it on the client.", + ); +}; diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/bool.ts b/skills/clickhouse-js-node-rowbinary/src/writers/bool.ts new file mode 100644 index 000000000..689847f29 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/bool.ts @@ -0,0 +1,10 @@ +import { Sink } from "./core.js"; +import { writeUInt8 } from "./integers.js"; + +/** + * Write a `Bool`: 1 byte, stored as `UInt8` (`false` -> 0, `true` -> 1). Mirror + * of `readBool`. + */ +export function writeBool(sink: Sink, value: boolean): void { + writeUInt8(sink, value ? 1 : 0); +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/composite.ts b/skills/clickhouse-js-node-rowbinary/src/writers/composite.ts new file mode 100644 index 000000000..ed14451ab --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/composite.ts @@ -0,0 +1,140 @@ +import { type Writer } from "./core.js"; +import { writeUInt8 } from "./integers.js"; +import { writeUVarint } from "./varint.js"; + +// --- Writers: the encode mirror of the combinators in `composite.ts`. Each takes +// sub-WRITERS (instead of sub-readers) and returns a Writer; MONOMORPHIZE when +// generating code, exactly as noted for the readers. + +/** + * Write a `Nullable(T)`: a 1-byte null flag (0 = present, 1 = NULL), then the + * inner value ONLY when present. The inverse of `readNullable`; curried — pass the + * inner writer, get a `Writer`. + */ +export function writeNullable(writeValue: Writer): Writer { + return (sink, value) => { + if (value === null) { + writeUInt8(sink, 1); + } else { + writeUInt8(sink, 0); + writeValue(sink, value); + } + }; +} + +/** + * Write an `Array(T)`: a LEB128 element count, then each element back-to-back. The + * inverse of `readArray`; curried — pass the element writer, get a `Writer`. + */ +export function writeArray(writeElement: Writer): Writer { + return (sink, values) => { + writeUVarint(sink, values.length); + // C-style loop, not for-of: this is a hot path and we don't want the + // iterator protocol overhead on a plain array. + for (let i = 0; i < values.length; i++) writeElement(sink, values[i]!); + }; +} + +/** + * Write a `QBit(element_type, dimension)`: in RowBinary it is byte-for-byte an + * `Array(element_type)`, so this is just {@link writeArray}. The inverse of + * `readQBit`. + */ +export function writeQBit(writeElement: Writer): Writer { + return writeArray(writeElement); +} + +/** + * Write a `Tuple(...)` from a positional array: each element's value + * back-to-back, with NO count and NO delimiter. The inverse of `readTuple`; + * curried — pass one writer per element (in order), get a `Writer` of the tuple. + */ +export function writeTuple(writers: { + [K in keyof T]: Writer; +}): Writer { + const fns = writers as ReadonlyArray>; + return (sink, value) => { + for (let i = 0; i < fns.length; i++) fns[i]!(sink, value[i]); + }; +} + +/** + * Write a named `Tuple(name1 T1, ...)` from an object. The wire is identical to an + * unnamed tuple — values back-to-back, no count or delimiter — so the writers run + * in the `writers` object's key order, which MUST match the tuple's declared field + * order. The inverse of `readTupleNamed`; curried. + */ +export function writeTupleNamed>(writers: { + [K in keyof T]: Writer; +}): Writer { + const fns = writers as Record>; + const keys = Object.keys(fns); + return (sink, value) => { + // C-style loop, not for-of: hot path, plain array of keys. + for (let i = 0; i < keys.length; i++) { + const key = keys[i]!; + fns[key]!(sink, value[key]); + } + }; +} + +/** + * Write a `Map(K, V)`: a LEB128 pair count, then key/value interleaved + * (k, v, k, v, ...). The inverse of `readMap`; curried — pass the key and value + * writers, get a `Writer>` (`Map` iteration order is preserved). + */ +export function writeMap( + writeKey: Writer, + writeValue: Writer, +): Writer> { + return (sink, map) => { + writeUVarint(sink, map.size); + // for-of is intentional here: it is the fastest way to iterate a `Map` + // (unlike plain arrays, where a C-style index loop wins). + for (const [key, value] of map) { + writeKey(sink, key); + writeValue(sink, value); + } + }; +} + +/** + * A tagged `Variant` value for {@link writeVariant}: the active alternative's + * `discriminant` (its index in the sorted-type-name order) paired with its value, + * or `null` for a NULL. + * + * WHY TAGGED: `readVariant` returns only the decoded VALUE — the discriminant is + * consumed from the wire and not surfaced — so encode cannot recover which + * alternative a bare value belongs to (e.g. is `5` the `UInt8` or the `Int32` + * alternative?). The discriminant must therefore be supplied explicitly, the + * encode-side analog of the `readGeometry` switch. + */ +export type VariantValue = + | readonly [discriminant: number, value: unknown] + | null; + +/** + * Write a `Variant(T1, ..., Tn)`: a 1-byte discriminant then the chosen + * alternative's value (discriminant `0xFF` = NULL, no value). The inverse of + * `readVariant`; curried — pass the alternative writers in sorted-type-name order + * (same order the reader expects), get a `Writer`. + */ +export function writeVariant( + writers: ReadonlyArray>, +): Writer { + return (sink, value) => { + if (value === null) { + writeUInt8(sink, 0xff); + return; + } + const [discriminant, inner] = value; + const fn = writers[discriminant] as Writer | undefined; + if (fn === undefined) { + throw new RangeError( + `RowBinary Variant: discriminant ${discriminant} out of range (${writers.length} alternatives)`, + ); + } + writeUInt8(sink, discriminant); + fn(sink, inner); + }; +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/core.ts b/skills/clickhouse-js-node-rowbinary/src/writers/core.ts new file mode 100644 index 000000000..2004d684a --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/core.ts @@ -0,0 +1,92 @@ +/** + * Thrown by {@link reserve} when the sink's buffer is full — i.e. lacks the bytes + * a write needs. The encode-side mirror of the reader's `NeedMoreData`. Like the + * reader, the `Sink` treats its buffer as a FIXED-length window: when a write + * would overflow it, `reserve` throws this sentinel WITHOUT moving the position, + * so a driver can flush the bytes written so far down the connection (a transport + * filter can glue successive buffers back together) and continue into a fresh + * buffer. + * + * A bare sentinel, NOT an `Error` subclass, on purpose — exactly as `NeedMoreData` + * on the read side: constructing an `Error` captures a stack trace (the expensive + * part of throwing), pure waste on a path that fires once per buffer boundary. + */ +export const BufferFull = Symbol("RowBinary.BufferFull"); + +/** + * The write-side mirror of the reader's `Cursor`: the cursor every writer threads + * through. A `Buffer` to write into, the current write position, and a + * `DataView` over the same bytes. The encode counterpart of decode's `Cursor`. + * + * Deliberately STATE only — no write methods. Encoding lives in the free + * `writeX(sink, value)` functions in the sibling modules, so a generated encoder + * pulls in only the per-type writers a result needs (exactly like the reader + * side). `view`/`buf` are public so those free functions can reach them. + * + * Like a `Cursor`, a `Sink` wraps a FIXED-length buffer (supplied by the caller): + * it never reallocates. {@link reserve} throws {@link BufferFull} when the next + * write would overflow, the encode mirror of the reader's `advance` throwing + * `NeedMoreData` on underflow. Size the buffer to a chunk you intend to flush, and + * pull the written bytes with {@link Sink.bytes}. + */ +export class Sink { + pos = 0; + + /** + * The buffer being written into. Only `buf.subarray(0, pos)` (see + * {@link Sink.bytes}) holds written bytes; the tail is unwritten headroom. + * Built with the buffer's own `byteOffset`/`byteLength` view in + * {@link Sink.view}, exactly like the reader's `Cursor`. + */ + readonly buf: Buffer; + + /** + * `DataView` over {@link Sink.buf}, for fixed-width integer/float writes. Built + * with the buffer's own `byteOffset`/`byteLength`: a `Buffer` is often a window + * into a larger pooled `ArrayBuffer`, so `new DataView(buf.buffer)` alone would + * point at the wrong bytes. + */ + readonly view: DataView; + + constructor(buf: Buffer) { + this.buf = buf; + this.view = new DataView(buf.buffer, buf.byteOffset, buf.byteLength); + } + + /** + * The written bytes — `buf.subarray(0, pos)`. A zero-copy VIEW into the sink's + * buffer, so use `Buffer.from(sink.bytes())` if you need an independent copy. + */ + bytes(): Buffer { + return this.buf.subarray(0, this.pos); + } +} + +/** + * A `Writer` encodes one value of type `T` into the sink, advancing it — the + * mirror of the reader's `Reader`. Leaf writers (e.g. `writeUInt32`) are + * `Writer`s directly; combinators (e.g. `writeArray`) take sub-`Writer`s and + * return a `Writer`, so types compose with no per-element closures. + */ +export type Writer = (sink: Sink, value: T) => void; + +/** + * Reserve `n` bytes for the next write: bounds-check them, advance the position + * past them, and return the offset the write starts at (the value BEFORE + * advancing). The write-side mirror of the reader's `advance`: every fixed-width + * write goes through it, so the capacity check and position bookkeeping live in + * one place: + * + * function writeInt32(s, v) { s.view.setInt32(reserve(s, 4), v, true); } + * + * Throws {@link BufferFull} when fewer than `n` bytes remain, WITHOUT moving + * the position — the buffer is fixed-length, exactly as the reader's input is, so + * a driver flushes what is written and retries the row into a fresh buffer. + */ +export function reserve(sink: Sink, n: number): number { + const start = sink.pos; + const next = start + n; + if (next > sink.buf.length) throw BufferFull; + sink.pos = next; + return start; +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/datetime.ts b/skills/clickhouse-js-node-rowbinary/src/writers/datetime.ts new file mode 100644 index 000000000..4c8a6fa45 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/datetime.ts @@ -0,0 +1,123 @@ +import { type Writer, Sink, reserve } from "./core.js"; +import { type Microseconds, type Nanoseconds } from "../readers/datetime.js"; + +const MS_PER_DAY = 86_400_000; + +/** + * Write a `Date`: 2-byte `UInt16` count of days since 1970-01-01 (UTC). The + * inverse of `readDate` — the `Date` it produced is at UTC midnight, so + * `getTime() / 86_400_000` recovers the whole-day count exactly. A non-midnight + * input is floored to its calendar day (matching ClickHouse's truncation), + * never rounded up into the next day. + * + * PRECONDITION: a valid `Date` whose day count fits the `UInt16` range + * (1970-01-01 … ~2149-06-06). Like every leaf writer (see `writeUVarint`) this + * is not range-checked — an invalid or out-of-range `Date` (e.g. a pre-1970 one, + * which belongs in {@link writeDate32}) is a programming error; the resulting + * bytes are rejected server-side. + */ +export function writeDate(sink: Sink, value: Date): void { + sink.view.setUint16( + reserve(sink, 2), + Math.floor(value.getTime() / MS_PER_DAY), + true, + ); +} + +/** + * Write a `Date32`: 4-byte signed `Int32` count of days since 1970-01-01 (UTC), + * negative for pre-1970 dates. The inverse of `readDate32`. A non-midnight input + * is floored toward -inf to its calendar day, so pre-1970 instants land on the + * correct (more negative) day rather than rounding toward the epoch. + * + * PRECONDITION: a valid `Date` whose day count fits `Int32`. Not range-checked + * (as elsewhere) — an invalid `Date` is a programming error, rejected server-side. + */ +export function writeDate32(sink: Sink, value: Date): void { + sink.view.setInt32( + reserve(sink, 4), + Math.floor(value.getTime() / MS_PER_DAY), + true, + ); +} + +/** + * Write a `DateTime`: 4-byte `UInt32` Unix seconds. The inverse of `readDateTime`; + * the column timezone is metadata, not in the bytes. Sub-second components are + * floored away (matching the reader and `writeDateTime64`'s `Math.floor`), never + * rounded up to the next second. + * + * PRECONDITION: a valid `Date` whose Unix-seconds fit the `UInt32` range + * (1970-01-01 … 2106-02-07). Not range-checked (as elsewhere) — an invalid or + * out-of-range `Date` is a programming error, rejected server-side. + */ +export function writeDateTime(sink: Sink, value: Date): void { + sink.view.setUint32( + reserve(sink, 4), + Math.floor(value.getTime() / 1000), + true, + ); +} + +/** + * Write a `DateTime64(P)`: 8-byte signed `Int64` count of `10^-P`-second ticks. + * Curried: `writeDateTime64(P)` returns the writer. The inverse of + * `readDateTime64`, which returns `[date, nanoseconds]` (date truncated to whole + * seconds, nanoseconds the sub-second remainder regardless of P). This recombines + * them: `ticks = seconds * 10^P + nanoseconds / 10^(9 - P)`. The reader floors + * seconds toward -inf with a non-negative remainder, so the seconds are computed + * with `Math.floor` on the cheap JS-number millisecond value (not bigint division, + * which truncates toward zero) — exact for negative instants too. + */ +export function writeDateTime64( + precision: number, +): Writer<[Date, Nanoseconds]> { + const scale = 10n ** BigInt(precision); + const nsPerTick = 10n ** BigInt(9 - precision); + return (sink, [date, nanoseconds]) => { + const seconds = BigInt(Math.floor(date.getTime() / 1000)); + sink.buf.writeBigInt64LE( + seconds * scale + BigInt(nanoseconds) / nsPerTick, + reserve(sink, 8), + ); + }; +} + +/** + * Write a `DateTime64(3)` (milliseconds) from a plain `Date` — the inverse of + * `readDateTime64P3`. P=3 is a `Date`'s own resolution, so the tick count is + * exactly `getTime()` in milliseconds. + */ +export function writeDateTime64P3(sink: Sink, value: Date): void { + sink.buf.writeBigInt64LE(BigInt(value.getTime()), reserve(sink, 8)); +} + +/** + * Write a `DateTime64(6)` (microseconds) from `[date, microseconds]` — the + * inverse of `readDateTime64P6`. `ticks = seconds * 1_000_000 + micros`. + */ +export function writeDateTime64P6( + sink: Sink, + [date, microseconds]: [Date, Microseconds], +): void { + const seconds = BigInt(Math.floor(date.getTime() / 1000)); + sink.buf.writeBigInt64LE( + seconds * 1_000_000n + BigInt(microseconds), + reserve(sink, 8), + ); +} + +/** + * Write a `DateTime64(9)` (nanoseconds) from `[date, nanoseconds]` — the inverse + * of `readDateTime64P9`. `ticks = seconds * 1_000_000_000 + nanos`. + */ +export function writeDateTime64P9( + sink: Sink, + [date, nanoseconds]: [Date, Nanoseconds], +): void { + const seconds = BigInt(Math.floor(date.getTime() / 1000)); + sink.buf.writeBigInt64LE( + seconds * 1_000_000_000n + BigInt(nanoseconds), + reserve(sink, 8), + ); +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/decimals.ts b/skills/clickhouse-js-node-rowbinary/src/writers/decimals.ts new file mode 100644 index 000000000..21ed031f0 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/decimals.ts @@ -0,0 +1,51 @@ +import { type Writer } from "./core.js"; +import { type DecimalValue } from "../readers/decimals.js"; +import { + writeInt32, + writeInt64, + writeInt128, + writeInt256, +} from "./integers.js"; + +/** + * Parse a fixed-point decimal string into a {@link DecimalValue} at the given + * `scale` — the inverse of `formatDecimal`. `"1.5000"` with scale 4 -> + * `[15000n, 4]`. A shorter fraction is right-padded with zeros to `scale`; a + * longer one is truncated (not rounded). Plug in only when you start from a + * string; if you already have the unscaled bigint, build the pair directly. + */ +export function parseDecimal(text: string, scale: number): DecimalValue { + const neg = text.startsWith("-"); + const body = neg ? text.slice(1) : text; + const dot = body.indexOf("."); + const intPart = dot < 0 ? body : body.slice(0, dot); + const fracPart = dot < 0 ? "" : body.slice(dot + 1); + const frac = (fracPart + "0".repeat(scale)).slice(0, scale); + let unscaled = BigInt((intPart || "0") + frac); + if (neg) unscaled = -unscaled; + return [unscaled, scale]; +} + +/** + * Write a `Decimal32(P, S)`: the `unscaled` part of a {@link DecimalValue} as a + * 4-byte little-endian signed integer (same wire as `Int32`). The inverse of + * `readDecimal32`; the `scale` lives in the type, so only `unscaled` is written + * (it must fit in `Int32`). + * + * `Decimal(P, S)` picks the width by precision P, exactly as the readers: P<=9 -> + * Decimal32, <=18 -> Decimal64, <=38 -> Decimal128, <=76 -> Decimal256. + */ +export const writeDecimal32: Writer = (sink, [unscaled]) => + writeInt32(sink, Number(unscaled)); + +/** Write a `Decimal64(P, S)`: 8-byte LE signed integer. Inverse of `readDecimal64`. */ +export const writeDecimal64: Writer = (sink, [unscaled]) => + writeInt64(sink, unscaled); + +/** Write a `Decimal128(P, S)`: 16-byte LE signed integer. Inverse of `readDecimal128`. */ +export const writeDecimal128: Writer = (sink, [unscaled]) => + writeInt128(sink, unscaled); + +/** Write a `Decimal256(P, S)`: 32-byte LE signed integer. Inverse of `readDecimal256`. */ +export const writeDecimal256: Writer = (sink, [unscaled]) => + writeInt256(sink, unscaled); diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/enums.ts b/skills/clickhouse-js-node-rowbinary/src/writers/enums.ts new file mode 100644 index 000000000..473e2ce12 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/enums.ts @@ -0,0 +1,18 @@ +import { Sink, reserve } from "./core.js"; + +/** + * Write an `Enum8`: the value's underlying signed `Int8` (1 byte). Mirror of + * `readEnum8` — the name<->value map lives in the column type, so take the raw + * numeric value. + */ +export function writeEnum8(sink: Sink, value: number): void { + sink.view.setInt8(reserve(sink, 1), value); +} + +/** + * Write an `Enum16`: the value's underlying signed `Int16` (2 bytes, + * little-endian). Mirror of `readEnum16`. + */ +export function writeEnum16(sink: Sink, value: number): void { + sink.view.setInt16(reserve(sink, 2), value, true); +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/floats.ts b/skills/clickhouse-js-node-rowbinary/src/writers/floats.ts new file mode 100644 index 000000000..29e5bff0b --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/floats.ts @@ -0,0 +1,40 @@ +import { Sink, reserve } from "./core.js"; + +/** Write a `Float32`: 4 bytes, little-endian IEEE 754. Mirror of `readFloat32`. */ +export function writeFloat32(sink: Sink, value: number): void { + sink.view.setFloat32(reserve(sink, 4), value, true); +} + +/** Write a `Float64`: 8 bytes, little-endian IEEE 754. Mirror of `readFloat64`. */ +export function writeFloat64(sink: Sink, value: number): void { + sink.view.setFloat64(reserve(sink, 8), value, true); +} + +/** + * Scratch view for narrowing a float32 to a `BFloat16`: BFloat16's 16 bits are + * the top half of an IEEE 754 float32, so we stage the float32 and take its high + * 16 bits back out. + */ +const bf16Scratch = new DataView(new ArrayBuffer(4)); + +/** + * Write a `BFloat16`: 2 bytes, little-endian — the high 16 bits of `value`'s + * float32 representation (same 8-bit exponent, 7-bit mantissa). Mirror of + * `readBFloat16`: it widens a BFloat16 to a float32 by placing the bits in the + * top half, so here we stage the float32 and take that top half back. + * + * NOTE: this TRUNCATES the float32 mantissa to BFloat16's 7 bits (no rounding), + * matching the reader's exact inverse for values that originated as BFloat16. An + * arbitrary float32 loses precision, exactly as ClickHouse's own BFloat16 cast. + * + * NOTE: `bf16Scratch` is module-level shared state written-then-read; safe + * because the body is synchronous (do NOT introduce an `await`/`yield` between + * the `setFloat32` and the `getUint16`). + */ +export function writeBFloat16(sink: Sink, value: number): void { + bf16Scratch.setFloat32(0, value, true); + // The float32's little-endian bytes are [lo16, hi16]; the high 16 bits at byte + // offset 2 are the BFloat16 payload. + const bits = bf16Scratch.getUint16(2, true); + sink.view.setUint16(reserve(sink, 2), bits, true); +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/geo.ts b/skills/clickhouse-js-node-rowbinary/src/writers/geo.ts new file mode 100644 index 000000000..f95aae145 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/geo.ts @@ -0,0 +1,125 @@ +import { type Writer, Sink, reserve } from "./core.js"; +import { type Point } from "../readers/geo.js"; +import { writeUInt8 } from "./integers.js"; +import { writeUVarint } from "./varint.js"; + +/** + * Write a `Point`: `Tuple(Float64, Float64)` -> `[x, y]`. Inverse of `readPoint`. + * A single `reserve(16)` then two inlined `setFloat64`s — no per-coordinate + * `writeFloat64` call and only one bounds check. + */ +export function writePoint(sink: Sink, [x, y]: Point): void { + const o = reserve(sink, 16); + sink.view.setFloat64(o, x, true); + sink.view.setFloat64(o + 8, y, true); +} + +/** + * Write a `Ring`: `Array(Point)` — a LEB128 point count, then each point. The + * inverse of `readRing`; a SINGLE `reserve(16 * length)` covers the whole point + * block (one bounds check per ring, not per point), then the coordinates are + * written into it inline, mirroring the reader. + */ +export function writeRing(sink: Sink, points: readonly Point[]): void { + writeUVarint(sink, points.length); + let p = reserve(sink, points.length * 16); + for (let i = 0; i < points.length; i++) { + const [x, y] = points[i]!; + sink.view.setFloat64(p, x, true); + sink.view.setFloat64(p + 8, y, true); + p += 16; + } +} + +/** Write a `LineString`: `Array(Point)` (identical wire to a `Ring`). Inverse of `readLineString`. */ +export function writeLineString(sink: Sink, points: readonly Point[]): void { + writeUVarint(sink, points.length); + let p = reserve(sink, points.length * 16); + for (let i = 0; i < points.length; i++) { + const [x, y] = points[i]!; + sink.view.setFloat64(p, x, true); + sink.view.setFloat64(p + 8, y, true); + p += 16; + } +} + +/** Write a `Polygon`: `Array(Ring)` — outer ring first, then holes. Inverse of `readPolygon`. */ +export function writePolygon(sink: Sink, rings: readonly Point[][]): void { + writeUVarint(sink, rings.length); + for (let i = 0; i < rings.length; i++) writeRing(sink, rings[i]!); +} + +/** Write a `MultiLineString`: `Array(LineString)`. Inverse of `readMultiLineString`. */ +export function writeMultiLineString( + sink: Sink, + lines: readonly Point[][], +): void { + writeUVarint(sink, lines.length); + for (let i = 0; i < lines.length; i++) writeLineString(sink, lines[i]!); +} + +/** Write a `MultiPolygon`: `Array(Polygon)`. Inverse of `readMultiPolygon`. */ +export function writeMultiPolygon( + sink: Sink, + polygons: readonly Point[][][], +): void { + writeUVarint(sink, polygons.length); + for (let i = 0; i < polygons.length; i++) writePolygon(sink, polygons[i]!); +} + +/** + * A tagged `Geometry` value for {@link writeGeometry}: the alternative's + * `discriminant` paired with its value, or `null` for NULL. Like + * `readVariant`/`writeVariant`, `readGeometry` returns only the value — and the + * geo value shapes overlap (LineString and Ring are both `Point[]`, + * MultiLineString and Polygon both `Point[][]`) — so encode must be told which geo + * type it is via the discriminant. + * + * Discriminants (sorted by type name): LineString(0), MultiLineString(1), + * MultiPolygon(2), Point(3), Polygon(4), Ring(5); `0xFF` = NULL. + */ +export type GeometryValue = + | readonly [discriminant: number, value: unknown] + | null; + +/** + * Write a `Geometry`: a 1-byte discriminant then the chosen geo type's value. The + * inverse of `readGeometry` (a switch over the discriminant with each branch + * inlined). Takes a tagged {@link GeometryValue} because the value shapes are + * ambiguous on their own. + */ +export const writeGeometry: Writer = (sink, value) => { + if (value === null) { + writeUInt8(sink, 0xff); + return; + } + // The discriminant byte is written inside each case, only after the switch + // has accepted it — an out-of-range value throws from `default` before the + // sink is advanced, so it never leaves a partially-written payload behind + // (mirrors `writeVariant`). + const [discriminant, geo] = value; + switch (discriminant) { + case 0: + writeUInt8(sink, discriminant); + return writeLineString(sink, geo as Point[]); + case 1: + writeUInt8(sink, discriminant); + return writeMultiLineString(sink, geo as Point[][]); + case 2: + writeUInt8(sink, discriminant); + return writeMultiPolygon(sink, geo as Point[][][]); + case 3: + writeUInt8(sink, discriminant); + return writePoint(sink, geo as Point); + case 4: + writeUInt8(sink, discriminant); + return writePolygon(sink, geo as Point[][]); + case 5: + writeUInt8(sink, discriminant); + return writeRing(sink, geo as Point[]); + default: + throw new RangeError( + `RowBinary: unknown Geometry discriminant ${discriminant}`, + ); + } +}; diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/integers.ts b/skills/clickhouse-js-node-rowbinary/src/writers/integers.ts new file mode 100644 index 000000000..d03116a95 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/integers.ts @@ -0,0 +1,90 @@ +import { Sink, reserve } from "./core.js"; + +// --- Writers: the encode mirror of the readers in `integers.ts`. Each writeX is +// the inverse of the matching readX. + +/** Write a `UInt8`: 1 byte (0 .. 255). Mirror of `readUInt8`. */ +export function writeUInt8(sink: Sink, value: number): void { + sink.buf[reserve(sink, 1)] = value; +} + +/** Write an `Int8`: 1 byte, two's-complement signed (-128 .. 127). Mirror of `readInt8`. */ +export function writeInt8(sink: Sink, value: number): void { + sink.view.setInt8(reserve(sink, 1), value); +} + +/** Write a `UInt16`: 2 bytes, little-endian. Mirror of `readUInt16`. */ +export function writeUInt16(sink: Sink, value: number): void { + sink.view.setUint16(reserve(sink, 2), value, true); +} + +/** Write an `Int16`: 2 bytes, little-endian, two's-complement. Mirror of `readInt16`. */ +export function writeInt16(sink: Sink, value: number): void { + sink.view.setInt16(reserve(sink, 2), value, true); +} + +/** Write a `UInt32`: 4 bytes, little-endian. Mirror of `readUInt32`. */ +export function writeUInt32(sink: Sink, value: number): void { + sink.view.setUint32(reserve(sink, 4), value, true); +} + +/** Write an `Int32`: 4 bytes, little-endian, two's-complement. Mirror of `readInt32`. */ +export function writeInt32(sink: Sink, value: number): void { + sink.view.setInt32(reserve(sink, 4), value, true); +} + +/** + * Write a `UInt64`: 8 bytes, little-endian. Takes a `bigint` (mirror of + * `readUInt64`). Uses Node's `Buffer.writeBigUInt64LE`, which writes the 64-bit + * value straight from the bigint — no narrowing to a JS number. The value must be + * in `[0, 2^64)`. + */ +export function writeUInt64(sink: Sink, value: bigint): void { + sink.buf.writeBigUInt64LE(value, reserve(sink, 8)); +} + +/** + * Write an `Int64`: 8 bytes, little-endian, two's-complement. Takes a `bigint` + * (mirror of `readInt64`). Uses Node's `Buffer.writeBigInt64LE`, writing the + * signed 64-bit value straight from the bigint (range `[-2^63, 2^63)`). + */ +export function writeInt64(sink: Sink, value: bigint): void { + sink.buf.writeBigInt64LE(value, reserve(sink, 8)); +} + +/** Mask reducing a bigint word to its unsigned 64-bit (two's-complement) value. */ +const MASK64 = (1n << 64n) - 1n; + +/** + * Write a `UInt128`/`Int128`: 16 bytes, little-endian, as two 64-bit words (low + * then high). Each word is masked to 64 bits with `& MASK64` — a pure bigint + * operation that yields the correct unsigned (two's-complement) representation for + * negatives too — and written with Node's `Buffer.writeBigUInt64LE`, so this one + * function serves both `readUInt128` and `readInt128` without narrowing to a JS + * number. + */ +export function writeUInt128(sink: Sink, value: bigint): void { + const o = reserve(sink, 16); + sink.buf.writeBigUInt64LE(value & MASK64, o); + sink.buf.writeBigUInt64LE((value >> 64n) & MASK64, o + 8); +} + +/** Write an `Int128`: 16 bytes LE two's-complement. Same word layout as {@link writeUInt128}. */ +export const writeInt128 = writeUInt128; + +/** + * Write a `UInt256`/`Int256`: 32 bytes, little-endian, as four 64-bit words + * (least-significant first). Like {@link writeUInt128}, each word is masked with + * `& MASK64` and written via `Buffer.writeBigUInt64LE`, handling both unsigned and + * signed (two's complement) values straight from the bigint. + */ +export function writeUInt256(sink: Sink, value: bigint): void { + const o = reserve(sink, 32); + sink.buf.writeBigUInt64LE(value & MASK64, o); + sink.buf.writeBigUInt64LE((value >> 64n) & MASK64, o + 8); + sink.buf.writeBigUInt64LE((value >> 128n) & MASK64, o + 16); + sink.buf.writeBigUInt64LE((value >> 192n) & MASK64, o + 24); +} + +/** Write an `Int256`: 32 bytes LE two's-complement. Same word layout as {@link writeUInt256}. */ +export const writeInt256 = writeUInt256; diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/interval.ts b/skills/clickhouse-js-node-rowbinary/src/writers/interval.ts new file mode 100644 index 000000000..ede88abe2 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/interval.ts @@ -0,0 +1,11 @@ +import { writeInt64 } from "./integers.js"; + +/** + * Write an `Interval` — any of `IntervalNanosecond` ... `IntervalYear`: a signed + * `Int64` count of the unit. The inverse of `readInterval`; the unit lives in the + * column type, not the bytes, so all 11 interval types share this writer. + * + * It IS `writeInt64` — assigned directly rather than wrapped, so there is no extra + * call frame on the wire-write path. + */ +export const writeInterval = writeInt64; diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/ip.ts b/skills/clickhouse-js-node-rowbinary/src/writers/ip.ts new file mode 100644 index 000000000..24a78ac0f --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/ip.ts @@ -0,0 +1,121 @@ +import { Sink, reserve } from "./core.js"; + +/** + * Write an `IPv4`: the raw 32-bit value (as produced by `readIPv4`) as a 4-byte + * little-endian `UInt32`. The inverse of `readIPv4`; pair with {@link parseIPv4} + * to start from a dotted-quad string. + */ +export function writeIPv4(sink: Sink, value: number): void { + sink.view.setUint32(reserve(sink, 4), value, true); +} + +/** + * Write an `IPv6`: the raw 16 bytes (network order, as produced by `readIPv6`) + * copied verbatim. The inverse of `readIPv6`; pair with {@link parseIPv6} to + * start from a string. Throws unless exactly 16 bytes. + */ +export function writeIPv6(sink: Sink, value: Uint8Array): void { + if (value.length !== 16) { + throw new RangeError( + `RowBinary: IPv6 must be 16 bytes, got ${value.length}`, + ); + } + const o = reserve(sink, 16); + sink.buf.set(value, o); +} + +/** Parse one dotted-quad field into a 0..255 octet, throwing on anything else. */ +function parseOctet(part: string): number { + const octet = Number(part); + if (!Number.isInteger(octet) || octet < 0 || octet > 255) { + throw new RangeError( + `RowBinary: invalid IPv4 octet ${JSON.stringify(part)}`, + ); + } + return octet; +} + +/** + * Parse a dotted-quad IPv4 string into the raw 32-bit value — the inverse of + * `formatIPv4`. `"1.2.3.4"` -> `0x01020304`. Pair with {@link writeIPv4}. + * + * The four octets are read explicitly rather than in a loop (only ever four); + * `>>> 0` coerces the assembled value back to an unsigned 32-bit number. + */ +export function parseIPv4(text: string): number { + const parts = text.split("."); + if (parts.length !== 4) { + throw new RangeError( + `RowBinary: invalid IPv4 string ${JSON.stringify(text)}`, + ); + } + const a = parseOctet(parts[0]!); + const b = parseOctet(parts[1]!); + const c = parseOctet(parts[2]!); + const d = parseOctet(parts[3]!); + return ((a << 24) | (b << 16) | (c << 8) | d) >>> 0; +} + +/** + * Parse an IPv6 string into its raw 16 bytes (network order) — the inverse of + * `formatIPv6`, accepting the canonical forms it emits (`::` zero-run compression + * and the `::ffff:a.b.c.d` IPv4-mapped form) as well as the fully expanded form. + * Pair with {@link writeIPv6}. + * + * Rejects malformed input (throws): a parse-time helper validates because it + * must produce exactly 16 well-defined bytes and there is no hot loop or server + * to fall back on — see the "No defensive validation" exceptions in AGENTS.md. + */ +export function parseIPv6(text: string): Buffer { + const halves = text.split("::"); + if (halves.length > 2) { + throw new RangeError( + `RowBinary: invalid IPv6 string ${JSON.stringify(text)}`, + ); + } + + // Expand a colon-separated side into 16-bit groups, splitting a trailing + // embedded IPv4 (a.b.c.d) into its two groups. + const toGroups = (side: string): number[] => { + if (side === "") return []; + const groups: number[] = []; + for (const part of side.split(":")) { + if (part.includes(".")) { + const v4 = parseIPv4(part); + groups.push((v4 >>> 16) & 0xffff, v4 & 0xffff); + } else { + // 1–4 hex digits only. Parsing strictly rather than `parseInt(...) & + // 0xffff` rejects malformed groups instead of silently turning them + // into 0 (`NaN & 0xffff`) or wrapping negatives like "-1" to 0xffff. + if (!/^[0-9a-fA-F]{1,4}$/.test(part)) { + throw new RangeError( + `RowBinary: invalid IPv6 group ${JSON.stringify(part)}`, + ); + } + groups.push(parseInt(part, 16)); + } + } + return groups; + }; + + const head = toGroups(halves[0]!); + const tail = halves.length === 2 ? toGroups(halves[1]!) : []; + const groups = + halves.length === 2 + ? [...head, ...new Array(8 - head.length - tail.length).fill(0), ...tail] + : head; + if (groups.length !== 8) { + throw new RangeError( + `RowBinary: invalid IPv6 string ${JSON.stringify(text)}`, + ); + } + // SAFE: allocUnsafe — the loop below writes all 16 bytes (out[0..15] for the + // 8 groups), and `out` is only allocated here, past every throw, so an + // uninitialized buffer is never returned. + const out = Buffer.allocUnsafe(16); + for (let i = 0; i < 8; i++) { + out[2 * i] = (groups[i]! >>> 8) & 0xff; + out[2 * i + 1] = groups[i]! & 0xff; + } + return out; +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/lowCardinality.ts b/skills/clickhouse-js-node-rowbinary/src/writers/lowCardinality.ts new file mode 100644 index 000000000..41b1d9fe7 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/lowCardinality.ts @@ -0,0 +1,12 @@ +import { type Writer } from "./core.js"; + +/** + * `LowCardinality(T)` is TRANSPARENT in RowBinary (no dictionary layer on the + * wire), so there is nothing extra to encode: use `T`'s own writer directly. + * This identity combinator mirrors `readLowCardinality` and returns the inner + * writer unchanged: + * + * writeLowCardinality(writeString) === writeString + */ +export const writeLowCardinality = (writeValue: Writer): Writer => + writeValue; diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/nested.ts b/skills/clickhouse-js-node-rowbinary/src/writers/nested.ts new file mode 100644 index 000000000..2a58c6567 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/nested.ts @@ -0,0 +1,17 @@ +import { type Writer } from "./core.js"; +import { writeArray, writeTupleNamed } from "./composite.js"; + +/** + * Inverse of `readNested`: `Nested(...)` has no wire format of its own, so for the + * `flatten_nested = 0` shape it is simply `Array(Tuple(a T1, b T2, …))`. This thin + * alias composes the existing array + named-tuple writers, mirroring the reader: + * + * writeNested({ a: writeUInt8, b: writeString }) + * === writeArray(writeTupleNamed({ a: writeUInt8, b: writeString })) + * + * When generating code, prefer inlining (monomorphize the array + tuple) over this + * generic composition. + */ +export const writeNested = >(writers: { + [K in keyof T]: Writer; +}): Writer => writeArray(writeTupleNamed(writers)); diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/nothing.ts b/skills/clickhouse-js-node-rowbinary/src/writers/nothing.ts new file mode 100644 index 000000000..e45aae927 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/nothing.ts @@ -0,0 +1,21 @@ +import { type Writer } from "./core.js"; + +/** + * Inverse of `readNothing`: a `Nothing` value is NEVER written either. It only + * appears wrapped, where the wrapper short-circuits before reaching it: + * + * writeArray(writeNothing) // only the empty array [] (length 0x00) + * writeNullable(writeNothing) // only null (lone flag byte 0x01) + * + * In both cases the element/inner writer is not invoked. This writer throws if it + * is ever actually called, which would mean a `Nothing` writer was placed where a + * real element/inner type was expected. + */ +export const writeNothing: Writer = () => { + throw new Error( + "RowBinary: Nothing is zero-width and is never encoded — it only appears as " + + "an empty Array(Nothing) or a NULL Nullable(Nothing), where the inner writer " + + "is not called. Reaching here means a Nothing writer was wired where a real " + + "element/inner type was expected.", + ); +}; diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/rows.ts b/skills/clickhouse-js-node-rowbinary/src/writers/rows.ts new file mode 100644 index 000000000..5f868b719 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/rows.ts @@ -0,0 +1,144 @@ +import { channel } from "node:diagnostics_channel"; +import { BufferFull, Sink, type Writer } from "./core.js"; + +/** Default sink size when `writeRows` isn't given one — a typical flush chunk. */ +const DEFAULT_BUFFER_SIZE = 64 * 1024; + +/** + * Payload of {@link FLUSH_CHANNEL_NAME}, published once per buffer `writeRows` + * flushes — the hook for buffer-capacity-utilization metrics (e.g. an OTEL + * `used_bytes` / `capacity_bytes` counter pair, divided in the backend for a + * byte-weighted average fill). Identity (table, `query_id`, …) is deliberately + * NOT here: carry it in `AsyncLocalStorage` and read it in the subscriber, which + * runs synchronously on the publisher's call stack, so its async context is live. + */ +export interface WriteRowsFlush { + /** Bytes actually written into the buffer just flushed (`<= capacityBytes`). */ + usedBytes: number; + /** That buffer's capacity; doubles from `bufferSize` to fit an oversized row. */ + capacityBytes: number; + /** + * The configured initial buffer size for this run. `capacityBytes > bufferSize` + * means the buffer had to grow to fit an oversized row, and `usedBytes / + * bufferSize` is the overflow magnitude — the signal that `bufferSize` is too + * small. (Growth is sticky: once grown, every later flush in the run reports the + * larger `capacityBytes`, so compare against `bufferSize`, not a prior capacity.) + */ + bufferSize: number; + /** Why it flushed: `"full"` mid-stream (next row overflowed) or `"end"` (rows ran out). */ + reason: "full" | "end"; +} + +/** + * `node:diagnostics_channel` name {@link writeRows} publishes a {@link WriteRowsFlush} + * once per flushed buffer. Subscribe to observe buffer-capacity utilization; with no + * subscriber `writeRows` skips the publish entirely (a single `hasSubscribers` + * check per buffer, off the per-row path), so it's free when unused. + */ +export const FLUSH_CHANNEL_NAME = "@clickhouse/rowbinary:writeRows.flush"; + +/** Created once — `channel()` is idempotent (same name → same object). */ +const flushChannel = channel(FLUSH_CHANNEL_NAME); + +/** + * Drive `writeRow` over every row of an iterable into a plain `RowBinary` payload + * — the encode mirror of `readRows`. Rows are concatenated with NO count, length + * prefix, or delimiter (just as the reader expects), so `writeRow` must emit + * EXACTLY one row's bytes. Curried: `writeRows(writeRow)` returns the driver. + * + * Returns a GENERATOR rather than a `Writer` on purpose. A `Sink` + * wraps a FIXED-length buffer, so a large (or unbounded) result won't fit in one + * pass — when a row would overflow, `writeRow` throws {@link BufferFull} from + * `reserve`, and a plain `(sink, rows) => void` would let that escape mid-row, + * leaving the caller unable to tell how many rows actually made it in. Instead + * this owns the sink: it catches `BufferFull`, rewinds to the last COMPLETE row + * boundary (never a half-written row), `yield`s that batch, and starts a FRESH + * `Sink` for the rows that didn't fit. Because each flush gets its own buffer, + * every yielded `Buffer` stays valid after the generator resumes — safe to retain + * or hand to an async sink, no copy needed. The caller supplies a `bufferSize`, + * not a sink, and `rows` is an `Iterable` (not a fixed array), so the same + * driver handles a future infinite/streaming row source unchanged. + * + * The driver loop is just a `for...of` — the generator yields each batch of whole + * rows whenever it stops accumulating (on overflow, or when the rows run out), so + * the final batch comes through the same channel and there's nothing to flush + * afterwards: + * + * const drive = writeRows(writeRow); + * for (const chunk of drive(rows, 64 * 1024)) send(chunk); + * + * OVERSIZED ROWS: when a single row won't fit even an empty buffer, the buffer is + * GROWN (doubled) and the row retried — never dropped, never thrown. The first + * time this happens `writeRows` `console.warn`s ONCE (the buffer may keep doubling + * after that) so an under-sized `bufferSize` or a pathologically large row doesn't + * pass unnoticed. + * + * METRICS: each flushed buffer is published as a {@link WriteRowsFlush} on the + * {@link FLUSH_CHANNEL_NAME} diagnostics channel — wire it to a utilization metric. + * No subscriber means no publish (one `hasSubscribers` check per buffer). + * + * When generating code, inline the per-column writes into the loop body, + * mirroring the reader. + */ +export function writeRows( + writeRow: Writer, +): (rows: Iterable, bufferSize?: number) => Generator { + return function* (rows, bufferSize = DEFAULT_BUFFER_SIZE) { + if (!Number.isSafeInteger(bufferSize) || bufferSize <= 0) + // Guard the growth loop: a 0 / NaN / negative size makes the first row + // overflow forever (`size *= 2` never escapes 0/NaN), so fail fast instead. + throw new RangeError( + `RowBinary writeRows: bufferSize must be a positive integer, got ${bufferSize}`, + ); + let size = bufferSize; + let warned = false; + let sink = new Sink(Buffer.allocUnsafe(size)); + for (const row of rows) { + while (true) { + const committed = sink.pos; // start of this row — the last clean boundary + try { + writeRow(sink, row); + break; // row written cleanly — on to the next + } catch (e) { + if (e !== BufferFull) throw e; + if (committed === 0) { + // An empty buffer couldn't hold even this one row: double it and + // retry the SAME row — never drop it. Nothing was written, so the + // discarded sink had no bytes to flush. + size *= 2; + if (!warned) { + warned = true; + console.warn( + `RowBinary writeRows: a row didn't fit bufferSize=${bufferSize}; ` + + `growing the buffer beyond it (possibly more than once). Raise bufferSize.`, + ); + } + sink = new Sink(Buffer.allocUnsafe(size)); + continue; + } + sink.pos = committed; // drop the partially written row, then flush + if (flushChannel.hasSubscribers) + flushChannel.publish({ + usedBytes: committed, + capacityBytes: size, + bufferSize, + reason: "full", + } satisfies WriteRowsFlush); + yield sink.bytes(); + sink = new Sink(Buffer.allocUnsafe(size)); // fresh buffer; retry the row + } + } + } + if (sink.pos > 0) { + // the final batch + if (flushChannel.hasSubscribers) + flushChannel.publish({ + usedBytes: sink.pos, + capacityBytes: size, + bufferSize, + reason: "end", + } satisfies WriteRowsFlush); + yield sink.bytes(); + } + }; +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/simpleAggregateFunction.ts b/skills/clickhouse-js-node-rowbinary/src/writers/simpleAggregateFunction.ts new file mode 100644 index 000000000..a31044ddb --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/simpleAggregateFunction.ts @@ -0,0 +1,12 @@ +import { type Writer } from "./core.js"; + +/** + * `SimpleAggregateFunction(func, T)` is TRANSPARENT in RowBinary — the column + * holds a finished value of `T` — so encode the inner `T` directly. Identity + * combinator mirroring `readSimpleAggregateFunction`: + * + * writeSimpleAggregateFunction(writeUInt64) === writeUInt64 + */ +export const writeSimpleAggregateFunction = ( + writeValue: Writer, +): Writer => writeValue; diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/strings.ts b/skills/clickhouse-js-node-rowbinary/src/writers/strings.ts new file mode 100644 index 000000000..f31f3587c --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/strings.ts @@ -0,0 +1,77 @@ +import { type Writer, Sink, reserve } from "./core.js"; +import { writeUVarint } from "./varint.js"; + +/** + * Write a `String` from a JS string: a varint byte-length prefix followed by the + * UTF-8 bytes. The inverse of `readString`. + * + * Split from {@link writeStringBytes} (rather than one function branching on the + * argument type) so each is monomorphic — V8 keeps a single shape per call site + * instead of going megamorphic on a `string | Uint8Array` parameter. + */ +export function writeString(sink: Sink, value: string): void { + const len = Buffer.byteLength(value, "utf8"); + writeUVarint(sink, len); + const o = reserve(sink, len); + sink.buf.write(value, o, len, "utf8"); +} + +/** + * Write a `String` from raw bytes: a varint byte-length prefix followed by the + * bytes verbatim. Use this for ClickHouse `String` columns holding arbitrary + * (non-UTF-8) bytes, mirroring the reader's note that `String` is not guaranteed + * UTF-8. The bytes counterpart of {@link writeString} (see the note there on why + * they are kept as two monomorphic functions). + */ +export function writeStringBytes(sink: Sink, value: Uint8Array): void { + writeUVarint(sink, value.length); + const o = reserve(sink, value.length); + sink.buf.set(value, o); +} + +/** + * Write a `FixedString(N)` from a string: exactly `size` bytes, UTF-8 encoded and + * right-padded with NUL bytes (`\x00`) to `size`. Curried: `writeFixedString(N)` + * returns the writer. The inverse of `readFixedString` — which preserves the + * trailing NULs, so re-encoding a value it produced is byte-exact. + * + * Throws if the UTF-8 encoding exceeds `size` bytes (it would not fit the column). + */ +export function writeFixedString(size: number): Writer { + return (sink, value) => { + const len = Buffer.byteLength(value, "utf8"); + if (len > size) { + throw new RangeError( + `RowBinary: FixedString value is ${len} bytes, exceeds FixedString(${size})`, + ); + } + const o = reserve(sink, size); + sink.buf.write(value, o, len, "utf8"); + // The sink's buffer may be uninitialized (allocUnsafe); zero the padding. + // POTENTIAL OPTIMIZATION: drop this fill when the buffer is known to be + // zero-initialized. Kept by default — relying on a zeroed buffer is a footgun + // (a pooled/reused sink would leak stale bytes into the column). + sink.buf.fill(0, o + len, o + size); + }; +} + +/** + * Write a `FixedString(N)` from raw bytes: exactly `size` bytes, the value copied + * verbatim and right-padded with NUL bytes if shorter. Curried: + * `writeFixedStringBytes(N)` returns the writer. The inverse of + * `readFixedStringBytes` (binary columns). + * + * Throws if the value is longer than `size`. + */ +export function writeFixedStringBytes(size: number): Writer { + return (sink, value) => { + if (value.length > size) { + throw new RangeError( + `RowBinary: FixedString value is ${value.length} bytes, exceeds FixedString(${size})`, + ); + } + const o = reserve(sink, size); + sink.buf.set(value, o); + sink.buf.fill(0, o + value.length, o + size); + }; +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/time.ts b/skills/clickhouse-js-node-rowbinary/src/writers/time.ts new file mode 100644 index 000000000..193fea5fd --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/time.ts @@ -0,0 +1,54 @@ +import { type Writer, Sink } from "./core.js"; +import { type ScaledTicks, type Seconds } from "../readers/time.js"; +import { writeInt32, writeInt64 } from "./integers.js"; + +/** + * Write a `Time`: 4-byte signed `Int32` seconds-of-day. The inverse of `readTime`; + * pair with {@link parseTime} to start from an "[-]HH:MM:SS" string. + */ +export function writeTime(sink: Sink, value: Seconds): void { + writeInt32(sink, value); +} + +/** + * Write a `Time64(P)`: 8-byte signed `Int64` count of `10^-P`-second ticks, from + * a {@link ScaledTicks} `[ticks, precision]`. The inverse of `readTime64`; the + * precision lives in the type, so only `ticks` is written. Pair with + * {@link parseTime64} to start from a string. + */ +export const writeTime64: Writer = (sink, [ticks]) => + writeInt64(sink, ticks); + +/** + * Parse an "[-]HH:MM:SS" string into signed seconds-of-day — the inverse of + * `formatTime`. The hour field may exceed two digits (range ±999:59:59). + */ +export function parseTime(text: string): Seconds { + const neg = text.startsWith("-"); + const body = neg ? text.slice(1) : text; + const [hh, mm, ss] = body.split(":"); + const seconds = Number(hh) * 3600 + Number(mm) * 60 + Number(ss); + return neg ? -seconds : seconds; +} + +/** + * Parse an "[-]HH:MM:SS[.fff]" string into a {@link ScaledTicks} at the given + * `precision` — the inverse of `formatTime64`. A shorter fraction is right-padded + * with zeros to `precision`; a longer one is truncated. + */ +export function parseTime64(text: string, precision: number): ScaledTicks { + const neg = text.startsWith("-"); + const body = neg ? text.slice(1) : text; + const dot = body.indexOf("."); + const timePart = dot < 0 ? body : body.slice(0, dot); + const fracPart = dot < 0 ? "" : body.slice(dot + 1); + const [hh, mm, ss] = timePart.split(":"); + const scale = 10n ** BigInt(precision); + const wholeSeconds = + BigInt(Number(hh) * 3600 + Number(mm) * 60 + Number(ss)) * scale; + const frac = BigInt( + (fracPart + "0".repeat(precision)).slice(0, precision) || "0", + ); + const ticks = wholeSeconds + frac; + return [neg ? -ticks : ticks, precision]; +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/uuid.ts b/skills/clickhouse-js-node-rowbinary/src/writers/uuid.ts new file mode 100644 index 000000000..2e3514870 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/uuid.ts @@ -0,0 +1,60 @@ +import { Sink, reserve } from "./core.js"; + +/** + * Parse a canonical `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` UUID string into the + * raw 16 wire bytes — the inverse of `formatUUID`. ClickHouse stores a UUID as + * two little-endian `UInt64` halves (high then low), so the 32 hex digits are + * split at the midpoint and each half written little-endian. Pair with + * {@link writeUUID}. + */ +export function parseUUID(text: string): Buffer { + const hex = text.replace(/-/g, ""); + if (hex.length !== 32) { + throw new RangeError( + `RowBinary: invalid UUID string ${JSON.stringify(text)}`, + ); + } + const v = BigInt("0x" + hex); + // SAFE: allocUnsafe — the two writeBigUInt64LE calls below overwrite all 16 + // bytes (offsets 0..7 and 8..15), so no uninitialized pool memory survives. + const b = Buffer.allocUnsafe(16); + b.writeBigUInt64LE(v >> 64n, 0); // high half -> first 8 bytes + b.writeBigUInt64LE(v & 0xffffffffffffffffn, 8); // low half -> last 8 bytes + return b; +} + +/** + * Write a `UUID` from its raw 16 wire bytes (as produced by `readUUID` or + * {@link parseUUID}): copied verbatim. The inverse of `readUUID`. Throws unless + * exactly 16 bytes are given. + */ +export function writeUUID(sink: Sink, value: Uint8Array): void { + if (value.length !== 16) { + throw new RangeError( + `RowBinary: UUID must be 16 bytes, got ${value.length}`, + ); + } + const o = reserve(sink, 16); + sink.buf.set(value, o); +} + +/** + * Write a `UUID` from a single 128-bit `bigint` (`hi << 64 | lo`) — the inverse + * of `readUUIDBigInt`. The high 64 bits go to the first little-endian `UInt64` + * half, the low 64 bits to the second. + */ +export function writeUUIDBigInt(sink: Sink, value: bigint): void { + const o = reserve(sink, 16); + sink.buf.writeBigUInt64LE((value >> 64n) & 0xffffffffffffffffn, o); + sink.buf.writeBigUInt64LE(value & 0xffffffffffffffffn, o + 8); +} + +/** + * Write a `UUID` from its two raw little-endian `UInt64` halves `[hi, lo]` — the + * inverse of `readUUIDHiLo`, the faithful wire split with no combining work. + */ +export function writeUUIDHiLo(sink: Sink, [hi, lo]: [bigint, bigint]): void { + const o = reserve(sink, 16); + sink.buf.writeBigUInt64LE(hi, o); + sink.buf.writeBigUInt64LE(lo, o + 8); +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/varint.ts b/skills/clickhouse-js-node-rowbinary/src/writers/varint.ts new file mode 100644 index 000000000..53986e81a --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/varint.ts @@ -0,0 +1,64 @@ +import { Sink, reserve } from "./core.js"; + +/** + * Write a LEB128 unsigned varint — the encode mirror of `readUVarint` (used for + * string/array/map lengths). + * + * Takes a JS `number`, so it is NOT bigint-friendly: the value MUST be a + * non-negative integer no larger than `Number.MAX_SAFE_INTEGER` (2^53 - 1). That + * precondition is NOT checked here — at this level the data is expected to be + * correct (a length the encoder itself produced), and an out-of-range value is a + * programming error the server will reject. If you genuinely need lengths beyond + * 2^53, write a bigint version with a bigint accumulator instead of widening this + * one. + * + * UNROLLED, mirroring `readUVarint`: branch on magnitude so the exact byte count + * is known up front for a single {@link reserve}, with no length-counting loop. + * Each byte carries 7 payload bits low-first, with the continuation bit (`+ 0x80`) + * set while more bits remain. `/` and `%` (never `>>>`/`&`): JS bitwise operators + * are 32-bit and would corrupt values past bit 31. The overwhelmingly common + * 1–2 byte case costs one or two compares. + */ +export function writeUVarint(sink: Sink, value: number): void { + if (value < 0x80) { + sink.buf[reserve(sink, 1)] = value; + return; + } + if (value < 0x4000) { + const o = reserve(sink, 2); + sink.buf[o] = (value % 128) + 0x80; + sink.buf[o + 1] = Math.floor(value / 128); + return; + } + if (value < 0x200000) { + const o = reserve(sink, 3); + sink.buf[o] = (value % 128) + 0x80; + sink.buf[o + 1] = (Math.floor(value / 128) % 128) + 0x80; + sink.buf[o + 2] = Math.floor(value / 16384); + return; + } + if (value < 0x10000000) { + const o = reserve(sink, 4); + sink.buf[o] = (value % 128) + 0x80; + sink.buf[o + 1] = (Math.floor(value / 128) % 128) + 0x80; + sink.buf[o + 2] = (Math.floor(value / 16384) % 128) + 0x80; + sink.buf[o + 3] = Math.floor(value / 2097152); + return; + } + // >= 2^28 — rare for RowBinary lengths. Fall back to a short loop writing into + // a single span sized by a leading magnitude count (still one reserve()). + let size = 5; + for ( + let v = Math.floor(value / 268435456); + v >= 0x80; + v = Math.floor(v / 128) + ) + size++; + const o = reserve(sink, size); + let v = value; + for (let i = 0; i < size - 1; i++) { + sink.buf[o + i] = (v % 128) + 0x80; + v = Math.floor(v / 128); + } + sink.buf[o + size - 1] = v; +} diff --git a/skills/clickhouse-js-node-rowbinary/src/writers/writer.ts b/skills/clickhouse-js-node-rowbinary/src/writers/writer.ts new file mode 100644 index 000000000..8765e3c91 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/src/writers/writer.ts @@ -0,0 +1,101 @@ +/** + * Barrel re-export of the RowBinary WRITER — the encode mirror of `reader.ts`, + * split by type family across parallel `*.ts` modules (the readers stay in + * their own files untouched). Import from here for everything in one place, or + * from a specific module (e.g. `./integers.js`, `./strings.js`) to + * pull in only the sub-writers a given result needs — the latter is what a + * generated encoder should do, copying just the modules its column types require. + * + * Each `writeX` is the inverse of the matching `readX`: it appends the value's + * RowBinary bytes to a {@link Sink} (the write-side mirror of the reader's + * `Cursor`). Leaf writers are `Writer`s directly; combinators (e.g. + * `writeArray`) take sub-writers and return a `Writer`, so types compose with no + * per-element closures — exactly like the reader combinators. + * + * const sink = new Sink(Buffer.allocUnsafe(64)); + * writeUInt8(sink, 255); + * sink.bytes(); // the encoded RowBinary + * + * - core — Sink, Writer, reserve (mirror of Cursor, Reader, advance) + * - varint — writeUVarint + * + * The dynamic AST-based encode path (the inverse of `compile.ts` / + * `rowBinaryWithNamesAndTypes.ts` / `dynamic.ts`) is intentionally NOT part of + * this barrel yet. + */ +export { Sink, reserve, BufferFull, type Writer } from "./core.js"; +export { writeUVarint } from "./varint.js"; +export { + writeUInt8, + writeInt8, + writeUInt16, + writeInt16, + writeUInt32, + writeInt32, + writeUInt64, + writeInt64, + writeUInt128, + writeInt128, + writeUInt256, + writeInt256, +} from "./integers.js"; +export { writeBool } from "./bool.js"; +export { writeEnum8, writeEnum16 } from "./enums.js"; +export { writeFloat32, writeFloat64, writeBFloat16 } from "./floats.js"; +export { + writeDecimal32, + writeDecimal64, + writeDecimal128, + writeDecimal256, + parseDecimal, +} from "./decimals.js"; +export { + writeString, + writeStringBytes, + writeFixedString, + writeFixedStringBytes, +} from "./strings.js"; +export { + writeUUID, + writeUUIDBigInt, + writeUUIDHiLo, + parseUUID, +} from "./uuid.js"; +export { writeIPv4, writeIPv6, parseIPv4, parseIPv6 } from "./ip.js"; +export { + writeDate, + writeDate32, + writeDateTime, + writeDateTime64, + writeDateTime64P3, + writeDateTime64P6, + writeDateTime64P9, +} from "./datetime.js"; +export { writeTime, writeTime64, parseTime, parseTime64 } from "./time.js"; +export { writeInterval } from "./interval.js"; +export { + writeNullable, + writeArray, + writeQBit, + writeTuple, + writeTupleNamed, + writeMap, + writeVariant, + type VariantValue, +} from "./composite.js"; +export { writeRows, FLUSH_CHANNEL_NAME, type WriteRowsFlush } from "./rows.js"; +export { + writePoint, + writeRing, + writeLineString, + writePolygon, + writeMultiLineString, + writeMultiPolygon, + writeGeometry, + type GeometryValue, +} from "./geo.js"; +export { writeLowCardinality } from "./lowCardinality.js"; +export { writeSimpleAggregateFunction } from "./simpleAggregateFunction.js"; +export { writeNested } from "./nested.js"; +export { writeNothing } from "./nothing.js"; +export { writeAggregateFunction } from "./aggregateFunction.js"; diff --git a/skills/clickhouse-js-node-rowbinary/tests/AggregateFunction.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/AggregateFunction.write.test.ts new file mode 100644 index 000000000..6900f0f6a --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/AggregateFunction.write.test.ts @@ -0,0 +1,10 @@ +import { describe, expect, it } from "vitest"; +import { encode } from "./encode.js"; +import { writeAggregateFunction } from "../src/writers/aggregateFunction.js"; + +describe("writeAggregateFunction", () => { + it("throws — opaque, unframed state is not generically encodable", () => + expect(() => encode(writeAggregateFunction, undefined as never)).toThrow( + /AggregateFunction is opaque/, + )); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Array.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Array.test.ts similarity index 88% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Array.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Array.test.ts index e76647f78..c94dc632c 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Array.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Array.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readArray, readNullable } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt32, readUInt8 } from "../src/integers.js"; -import { readString } from "../src/strings.js"; +import { readArray, readNullable } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt32, readUInt8 } from "../src/readers/integers.js"; +import { readString } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/BFloat16.test.ts b/skills/clickhouse-js-node-rowbinary/tests/BFloat16.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/BFloat16.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/BFloat16.test.ts index b375dae05..bc05cba33 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/BFloat16.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/BFloat16.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readBFloat16 } from "../src/floats.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readBFloat16 } from "../src/readers/floats.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Bool.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Bool.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Bool.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Bool.test.ts index afedd3eed..d1fece000 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Bool.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Bool.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readBool } from "../src/bool.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; +import { readBool } from "../src/readers/bool.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary/tests/BoolEnumFloat.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/BoolEnumFloat.write.test.ts new file mode 100644 index 000000000..2d526cbbb --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/BoolEnumFloat.write.test.ts @@ -0,0 +1,121 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { encode } from "./encode.js"; +import { writeBool } from "../src/writers/bool.js"; +import { writeEnum8, writeEnum16 } from "../src/writers/enums.js"; +import { + writeFloat32, + writeFloat64, + writeBFloat16, +} from "../src/writers/floats.js"; + +describe("writeBool", () => { + it("encodes true as 0x01", async () => + expect(encode(writeBool, true)).toEqual( + await query("SELECT true::Bool FORMAT RowBinary"), + )); + it("encodes false as 0x00", async () => + expect(encode(writeBool, false)).toEqual( + await query("SELECT false::Bool FORMAT RowBinary"), + )); +}); + +describe("writeEnum8 / writeEnum16", () => { + it("encodes an Enum8 value as its underlying Int8", async () => + expect(encode(writeEnum8, 2)).toEqual( + await query( + "SELECT CAST('b', 'Enum8(\\'a\\' = -1, \\'b\\' = 2)') FORMAT RowBinary", + ), + )); + it("encodes an Enum16 value as its underlying Int16", async () => + expect(encode(writeEnum16, 30000)).toEqual( + await query( + "SELECT CAST('y', 'Enum16(\\'x\\' = -300, \\'y\\' = 30000)') FORMAT RowBinary", + ), + )); +}); + +describe("writeFloat32", () => { + it("encodes 0", async () => + expect(encode(writeFloat32, 0)).toEqual( + await query("SELECT toFloat32(0) FORMAT RowBinary"), + )); + it("encodes 1.5", async () => + expect(encode(writeFloat32, 1.5)).toEqual( + await query("SELECT toFloat32(1.5) FORMAT RowBinary"), + )); + it("encodes -3.25", async () => + expect(encode(writeFloat32, -3.25)).toEqual( + await query("SELECT toFloat32(-3.25) FORMAT RowBinary"), + )); + it("encodes the max finite float32", async () => + expect(encode(writeFloat32, 3.4028234663852886e38)).toEqual( + await query("SELECT toFloat32(3.4028234663852886e38) FORMAT RowBinary"), + )); + it("encodes Infinity", async () => + expect(encode(writeFloat32, Infinity)).toEqual( + await query("SELECT toFloat32(inf) FORMAT RowBinary"), + )); +}); + +describe("writeFloat64", () => { + it("encodes 0", async () => + expect(encode(writeFloat64, 0)).toEqual( + await query("SELECT toFloat64(0) FORMAT RowBinary"), + )); + it("encodes 1.5", async () => + expect(encode(writeFloat64, 1.5)).toEqual( + await query("SELECT toFloat64(1.5) FORMAT RowBinary"), + )); + it("encodes -3.25", async () => + expect(encode(writeFloat64, -3.25)).toEqual( + await query("SELECT toFloat64(-3.25) FORMAT RowBinary"), + )); + it("encodes the max finite float64", async () => + expect(encode(writeFloat64, 1.7976931348623157e308)).toEqual( + await query("SELECT toFloat64(1.7976931348623157e308) FORMAT RowBinary"), + )); +}); + +describe("writeBFloat16", () => { + it("encodes 0", async () => + expect(encode(writeBFloat16, 0)).toEqual( + await query("SELECT toBFloat16(0) FORMAT RowBinary"), + )); + it("encodes 1.5", async () => + expect(encode(writeBFloat16, 1.5)).toEqual( + await query("SELECT toBFloat16(1.5) FORMAT RowBinary"), + )); + it("encodes -3.25", async () => + expect(encode(writeBFloat16, -3.25)).toEqual( + await query("SELECT toBFloat16(-3.25) FORMAT RowBinary"), + )); + it("encodes 100", async () => + expect(encode(writeBFloat16, 100)).toEqual( + await query("SELECT toBFloat16(100) FORMAT RowBinary"), + )); + + // Values whose float32 has nonzero low-16 mantissa bits — truncating to the + // high 16 bits (what ClickHouse does) gives a different result than rounding + // to nearest, so these pin down that the writer truncates exactly like CH. + it("encodes 1.1 (truncates, not rounds, the mantissa)", async () => + expect(encode(writeBFloat16, 1.1)).toEqual( + await query("SELECT toBFloat16(1.1) FORMAT RowBinary"), + )); + it("encodes 1.3", async () => + expect(encode(writeBFloat16, 1.3)).toEqual( + await query("SELECT toBFloat16(1.3) FORMAT RowBinary"), + )); + it("encodes 2.6", async () => + expect(encode(writeBFloat16, 2.6)).toEqual( + await query("SELECT toBFloat16(2.6) FORMAT RowBinary"), + )); + it("encodes 0.1", async () => + expect(encode(writeBFloat16, 0.1)).toEqual( + await query("SELECT toBFloat16(0.1) FORMAT RowBinary"), + )); + it("encodes a negative -1.1 (sign bit preserved)", async () => + expect(encode(writeBFloat16, -1.1)).toEqual( + await query("SELECT toBFloat16(-1.1) FORMAT RowBinary"), + )); +}); diff --git a/skills/clickhouse-js-node-rowbinary/tests/Composite.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Composite.write.test.ts new file mode 100644 index 000000000..bae1f892b --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/Composite.write.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { encode } from "./encode.js"; +import { + writeNullable, + writeArray, + writeTuple, + writeTupleNamed, + writeMap, + writeVariant, +} from "../src/writers/composite.js"; +import { writeInt32, writeUInt32 } from "../src/writers/integers.js"; +import { writeString } from "../src/writers/strings.js"; + +describe("writeNullable", () => { + it("encodes a present value (flag 0 then the value)", async () => + expect(encode(writeNullable(writeUInt32), 42)).toEqual( + await query("SELECT toNullable(toUInt32(42)) FORMAT RowBinary"), + )); + it("encodes NULL as the lone flag byte", async () => + expect(encode(writeNullable(writeUInt32), null)).toEqual( + await query("SELECT CAST(NULL AS Nullable(UInt32)) FORMAT RowBinary"), + )); +}); + +describe("writeArray", () => { + it("encodes Array(UInt32)", async () => + expect(encode(writeArray(writeUInt32), [1, 2, 3])).toEqual( + await query("SELECT [1, 2, 3]::Array(UInt32) FORMAT RowBinary"), + )); + it("encodes an empty array as a single 0x00 length", async () => + expect(encode(writeArray(writeUInt32), [])).toEqual( + await query("SELECT []::Array(UInt32) FORMAT RowBinary"), + )); +}); + +describe("writeTuple / writeTupleNamed", () => { + it("encodes a positional Tuple(UInt32, String)", async () => + expect( + encode(writeTuple<[number, string]>([writeUInt32, writeString]), [ + 7, + "hi", + ]), + ).toEqual(await query("SELECT (toUInt32(7), 'hi') FORMAT RowBinary"))); + + it("encodes a named Tuple in field order", async () => + expect( + encode( + writeTupleNamed<{ id: number; name: string }>({ + id: writeUInt32, + name: writeString, + }), + { id: 7, name: "hi" }, + ), + ).toEqual( + await query( + "SELECT CAST((toUInt32(7), 'hi'), 'Tuple(id UInt32, name String)') FORMAT RowBinary", + ), + )); +}); + +describe("writeMap", () => { + it("encodes Map(String, UInt32) in insertion order", async () => + expect( + encode( + writeMap(writeString, writeUInt32), + new Map([ + ["a", 1], + ["b", 2], + ]), + ), + ).toEqual( + await query( + "SELECT map('a', toUInt32(1), 'b', toUInt32(2)) FORMAT RowBinary", + ), + )); +}); + +describe("writeVariant", () => { + // Variant(Int32, String) sorts by type name: ["Int32", "String"]. + const writeV = writeVariant([writeInt32, writeString]); + + it("encodes the Int32 alternative (discriminant 0)", async () => + expect(encode(writeV, [0, -5])).toEqual( + await query( + "SELECT CAST(toInt32(-5), 'Variant(Int32, String)') SETTINGS allow_experimental_variant_type = 1 FORMAT RowBinary", + ), + )); + + it("encodes the String alternative (discriminant 1)", async () => + expect(encode(writeV, [1, "hello"])).toEqual( + await query( + "SELECT CAST('hello', 'Variant(Int32, String)') SETTINGS allow_experimental_variant_type = 1 FORMAT RowBinary", + ), + )); + + it("encodes NULL as a single 0xFF byte", () => + expect([...encode(writeV, null)]).toEqual([0xff])); + + it("throws for an out-of-range discriminant", () => + expect(() => encode(writeV, [9, 0])).toThrow(RangeError)); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Date.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Date.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Date.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Date.test.ts index 7128474ea..504ef4677 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Date.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Date.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readDate } from "../src/datetime.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readDate } from "../src/readers/datetime.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Date32.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Date32.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Date32.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Date32.test.ts index 05b0ec414..3259adfa9 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Date32.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Date32.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readDate32 } from "../src/datetime.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readDate32 } from "../src/readers/datetime.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/DateTime.test.ts b/skills/clickhouse-js-node-rowbinary/tests/DateTime.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/DateTime.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/DateTime.test.ts index c65ca1c98..e6b9318f8 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/DateTime.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/DateTime.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readDateTime } from "../src/datetime.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readDateTime } from "../src/readers/datetime.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary/tests/DateTime.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/DateTime.write.test.ts new file mode 100644 index 000000000..51d78c90c --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/DateTime.write.test.ts @@ -0,0 +1,99 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { encode } from "./encode.js"; +import { + writeDate, + writeDate32, + writeDateTime, + writeDateTime64, + writeDateTime64P3, + writeDateTime64P6, + writeDateTime64P9, +} from "../src/writers/datetime.js"; + +describe("writeDate / writeDate32", () => { + it("encodes a Date as days since the epoch", async () => + expect(encode(writeDate, new Date(Date.UTC(2021, 6, 7)))).toEqual( + await query("SELECT toDate('2021-07-07') FORMAT RowBinary"), + )); + it("encodes a pre-1970 Date32 (negative day count)", async () => + expect(encode(writeDate32, new Date(Date.UTC(1950, 0, 2)))).toEqual( + await query("SELECT toDate32('1950-01-02') FORMAT RowBinary"), + )); +}); + +describe("writeDateTime", () => { + it("encodes a DateTime as Unix seconds", async () => + expect( + encode(writeDateTime, new Date(Date.UTC(2021, 6, 7, 18, 30, 0))), + ).toEqual( + await query( + "SELECT toDateTime('2021-07-07 18:30:00', 'UTC') FORMAT RowBinary", + ), + )); +}); + +// Sub-day / sub-second inputs are floored to the encoded unit, never rounded +// up. Asserted purely (encode-vs-encode against the truncated instant) so the +// cases run without a live ClickHouse and can't be masked by a reader. +describe("date/time flooring", () => { + it("floors a near-midnight Date down to its own calendar day", () => + expect( + encode(writeDate, new Date(Date.UTC(2021, 6, 7, 23, 59, 59))), + ).toEqual(encode(writeDate, new Date(Date.UTC(2021, 6, 7))))); + + it("floors a pre-1970 Date32 toward the earlier day, not the epoch", () => + expect( + encode(writeDate32, new Date(Date.UTC(1969, 11, 31, 12, 0, 0))), + ).toEqual(encode(writeDate32, new Date(Date.UTC(1969, 11, 31))))); + + it("floors a sub-second DateTime down, never up to the next second", () => + expect( + encode(writeDateTime, new Date(Date.UTC(2021, 6, 7, 18, 30, 0, 600))), + ).toEqual( + encode(writeDateTime, new Date(Date.UTC(2021, 6, 7, 18, 30, 0))), + )); +}); + +describe("writeDateTime64", () => { + const wholeSecond = new Date(Date.UTC(2021, 6, 7, 18, 30, 0)); + + it("encodes DateTime64(9) via the generic writer", async () => + expect(encode(writeDateTime64(9), [wholeSecond, 123456789])).toEqual( + await query( + "SELECT toDateTime64('2021-07-07 18:30:00.123456789', 9, 'UTC') FORMAT RowBinary", + ), + )); + + it("encodes a negative DateTime64(9) instant", async () => + expect( + encode(writeDateTime64(9), [new Date(Date.UTC(1960, 0, 1, 0, 0, 0)), 1]), + ).toEqual( + await query( + "SELECT toDateTime64('1960-01-01 00:00:00.000000001', 9, 'UTC') FORMAT RowBinary", + ), + )); + + it("encodes DateTime64(3) via writeDateTime64P3", async () => + expect( + encode(writeDateTime64P3, new Date(Date.UTC(2021, 6, 7, 18, 30, 0, 123))), + ).toEqual( + await query( + "SELECT toDateTime64('2021-07-07 18:30:00.123', 3, 'UTC') FORMAT RowBinary", + ), + )); + + it("encodes DateTime64(6) via writeDateTime64P6", async () => + expect(encode(writeDateTime64P6, [wholeSecond, 123456])).toEqual( + await query( + "SELECT toDateTime64('2021-07-07 18:30:00.123456', 6, 'UTC') FORMAT RowBinary", + ), + )); + + it("encodes DateTime64(9) via writeDateTime64P9", async () => + expect(encode(writeDateTime64P9, [wholeSecond, 123456789])).toEqual( + await query( + "SELECT toDateTime64('2021-07-07 18:30:00.123456789', 9, 'UTC') FORMAT RowBinary", + ), + )); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64.test.ts b/skills/clickhouse-js-node-rowbinary/tests/DateTime64.test.ts similarity index 93% rename from skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/DateTime64.test.ts index 0857d474f..fd6e3ff1a 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/DateTime64.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readDateTime64 } from "../src/datetime.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readDateTime64 } from "../src/readers/datetime.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64P3.test.ts b/skills/clickhouse-js-node-rowbinary/tests/DateTime64P3.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64P3.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/DateTime64P3.test.ts index 1c9fa5745..ea30213e3 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64P3.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/DateTime64P3.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readDateTime64P3 } from "../src/datetime.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readDateTime64P3 } from "../src/readers/datetime.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64P6.test.ts b/skills/clickhouse-js-node-rowbinary/tests/DateTime64P6.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64P6.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/DateTime64P6.test.ts index 7c92da007..f3b61c0c2 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64P6.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/DateTime64P6.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readDateTime64P6 } from "../src/datetime.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readDateTime64P6 } from "../src/readers/datetime.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64P9.test.ts b/skills/clickhouse-js-node-rowbinary/tests/DateTime64P9.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64P9.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/DateTime64P9.test.ts index 59e646b0a..b50010d38 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/DateTime64P9.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/DateTime64P9.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readDateTime64P9 } from "../src/datetime.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readDateTime64P9 } from "../src/readers/datetime.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary/tests/Decimal.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Decimal.write.test.ts new file mode 100644 index 000000000..e10cf1445 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/Decimal.write.test.ts @@ -0,0 +1,86 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { encode } from "./encode.js"; +import { + writeDecimal32, + writeDecimal64, + writeDecimal128, + writeDecimal256, + parseDecimal, +} from "../src/writers/decimals.js"; + +describe("writeDecimal32", () => { + it("encodes 1.5", async () => + expect(encode(writeDecimal32, parseDecimal("1.5", 4))).toEqual( + await query("SELECT CAST('1.5', 'Decimal32(4)') FORMAT RowBinary"), + )); + it("encodes -12345.6789", async () => + expect(encode(writeDecimal32, parseDecimal("-12345.6789", 4))).toEqual( + await query( + "SELECT CAST('-12345.6789', 'Decimal32(4)') FORMAT RowBinary", + ), + )); + it("encodes 0", async () => + expect(encode(writeDecimal32, parseDecimal("0", 4))).toEqual( + await query("SELECT CAST('0', 'Decimal32(4)') FORMAT RowBinary"), + )); +}); + +describe("writeDecimal64", () => { + it("encodes 1.50", async () => + expect(encode(writeDecimal64, parseDecimal("1.50", 2))).toEqual( + await query("SELECT CAST('1.50', 'Decimal64(2)') FORMAT RowBinary"), + )); + it("encodes -9999999999.99", async () => + expect(encode(writeDecimal64, parseDecimal("-9999999999.99", 2))).toEqual( + await query( + "SELECT CAST('-9999999999.99', 'Decimal64(2)') FORMAT RowBinary", + ), + )); +}); + +describe("writeDecimal128", () => { + it("encodes 3.1415926535", async () => + expect(encode(writeDecimal128, parseDecimal("3.1415926535", 10))).toEqual( + await query( + "SELECT CAST('3.1415926535', 'Decimal128(10)') FORMAT RowBinary", + ), + )); + it("encodes -1.0000000001", async () => + expect(encode(writeDecimal128, parseDecimal("-1.0000000001", 10))).toEqual( + await query( + "SELECT CAST('-1.0000000001', 'Decimal128(10)') FORMAT RowBinary", + ), + )); +}); + +describe("writeDecimal256", () => { + it("encodes 2.71828182845904523536", async () => + expect( + encode(writeDecimal256, parseDecimal("2.71828182845904523536", 20)), + ).toEqual( + await query( + "SELECT CAST('2.71828182845904523536', 'Decimal256(20)') FORMAT RowBinary", + ), + )); + it("encodes -0.00000000000000000001", async () => + expect( + encode(writeDecimal256, parseDecimal("-0.00000000000000000001", 20)), + ).toEqual( + await query( + "SELECT CAST('-0.00000000000000000001', 'Decimal256(20)') FORMAT RowBinary", + ), + )); +}); + +describe("parseDecimal", () => { + it("scales the integer and fraction parts", () => + expect(parseDecimal("1.5000", 4)).toEqual([15000n, 4])); + it("handles a negative value", () => + expect(parseDecimal("-12345.6789", 4)).toEqual([-123456789n, 4])); + it("handles zero", () => expect(parseDecimal("0.00", 2)).toEqual([0n, 2])); + it("right-pads a short fraction to the scale", () => + expect(parseDecimal("1.5", 4)).toEqual([15000n, 4])); + it("truncates a fraction longer than the scale", () => + expect(parseDecimal("1.56789", 2)).toEqual([156n, 2])); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Decimal128.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Decimal128.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Decimal128.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Decimal128.test.ts index 465169f29..7e520afd6 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Decimal128.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Decimal128.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { formatDecimal, readDecimal128 } from "../src/decimals.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { formatDecimal, readDecimal128 } from "../src/readers/decimals.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Decimal256.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Decimal256.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Decimal256.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Decimal256.test.ts index fca855973..b83cd75d9 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Decimal256.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Decimal256.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { formatDecimal, readDecimal256 } from "../src/decimals.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { formatDecimal, readDecimal256 } from "../src/readers/decimals.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Decimal32.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Decimal32.test.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Decimal32.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Decimal32.test.ts index 170954007..ffc934e3a 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Decimal32.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Decimal32.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { formatDecimal, readDecimal32 } from "../src/decimals.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { formatDecimal, readDecimal32 } from "../src/readers/decimals.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Decimal64.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Decimal64.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Decimal64.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Decimal64.test.ts index b82275a83..aebdb0949 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Decimal64.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Decimal64.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { formatDecimal, readDecimal64 } from "../src/decimals.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { formatDecimal, readDecimal64 } from "../src/readers/decimals.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Dynamic.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Dynamic.test.ts similarity index 97% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Dynamic.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Dynamic.test.ts index 4b5242688..ddeb93ba4 100644 Binary files a/skills/clickhouse-js-node-rowbinary-parser/tests/Dynamic.test.ts and b/skills/clickhouse-js-node-rowbinary/tests/Dynamic.test.ts differ diff --git a/skills/clickhouse-js-node-rowbinary/tests/Enum16.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Enum16.test.ts new file mode 100644 index 000000000..ee121b69a --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/Enum16.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readEnum16 } from "../src/readers/enums.js"; + +async function reader(expr: string): Promise { + return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); +} + +describe("readEnum16", () => { + it("resolves a 16-bit underlying value to its name", async () => { + const read = readEnum16( + new Map([ + [1, "small"], + [300, "big"], + ]), + ); + const r = await reader("CAST('big' AS Enum16('small' = 1, 'big' = 300))"); + expect(read(r)).toBe("big"); + expect(r.pos).toBe(2); + }); + + it("resolves a negative enum value", async () => { + const read = readEnum16( + new Map([ + [-1000, "lo"], + [1000, "hi"], + ]), + ); + expect( + read(await reader("CAST('lo' AS Enum16('lo' = -1000, 'hi' = 1000))")), + ).toBe("lo"); + }); + + it("falls back to the stringified integer for an unmapped value", () => { + // Wire bytes for Int16 300 (little-endian) with an empty map => "300". + expect(readEnum16(new Map())(new Cursor(Buffer.from([0x2c, 0x01])))).toBe( + "300", + ); + }); + + describe("advance() edge cases", () => { + it("throws NeedMoreData for every incomplete prefix (0 .. full.length-1)", async () => { + const read = readEnum16( + new Map([ + [1, "small"], + [300, "big"], + ]), + ); + const full = await query( + "SELECT CAST('big' AS Enum16('small' = 1, 'big' = 300)) FORMAT RowBinary", + ); + for (let len = 0; len < full.length; len++) { + const r = new Cursor(full.subarray(0, len)); + let thrown: unknown; + try { + read(r); + } catch (e) { + thrown = e; + } + expect(thrown, `prefix length ${len} of ${full.length}`).toBe( + NeedMoreData, + ); + } + }); + }); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Enum8.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Enum8.test.ts similarity index 50% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Enum8.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Enum8.test.ts index 84fe3e605..711c17b2d 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Enum8.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Enum8.test.ts @@ -1,32 +1,50 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readEnum8 } from "../src/enums.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readEnum8 } from "../src/readers/enums.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); } describe("readEnum8", () => { - it("decodes the underlying value and resolves the name via a lookup", async () => { + it("resolves the underlying value to its name via the supplied map", async () => { + const read = readEnum8( + new Map([ + [1, "a"], + [2, "b"], + ]), + ); const r = await reader("CAST('b' AS Enum8('a' = 1, 'b' = 2))"); - const value = readEnum8(r); - expect(value).toBe(2); + expect(read(r)).toBe("b"); expect(r.pos).toBe(1); - // The name map comes from the column's type definition, not the wire. - const NAMES: Record = { 1: "a", 2: "b" }; - expect(NAMES[value]).toBe("b"); }); - it("decodes a negative enum value", async () => { - const value = readEnum8( - await reader("CAST('x' AS Enum8('x' = -1, 'y' = 2))"), + it("resolves a negative enum value", async () => { + const read = readEnum8( + new Map([ + [-1, "x"], + [2, "y"], + ]), + ); + expect(read(await reader("CAST('x' AS Enum8('x' = -1, 'y' = 2))"))).toBe( + "x", ); - expect(value).toBe(-1); + }); + + it("falls back to the stringified integer for an unmapped value", () => { + // Wire byte 0x05 with an empty map => no name => "5". + expect(readEnum8(new Map())(new Cursor(Buffer.from([5])))).toBe("5"); }); describe("advance() edge cases", () => { it("throws NeedMoreData for every incomplete prefix (0 .. full.length-1)", async () => { + const read = readEnum8( + new Map([ + [1, "a"], + [2, "b"], + ]), + ); const full = await query( "SELECT CAST('b' AS Enum8('a' = 1, 'b' = 2)) FORMAT RowBinary", ); @@ -34,7 +52,7 @@ describe("readEnum8", () => { const r = new Cursor(full.subarray(0, len)); let thrown: unknown; try { - readEnum8(r); + read(r); } catch (e) { thrown = e; } diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/FixedString.test.ts b/skills/clickhouse-js-node-rowbinary/tests/FixedString.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/FixedString.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/FixedString.test.ts index eab2e985f..ef7ad2e9d 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/FixedString.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/FixedString.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readFixedString } from "../src/strings.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readFixedString } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/FixedStringBytes.test.ts b/skills/clickhouse-js-node-rowbinary/tests/FixedStringBytes.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/FixedStringBytes.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/FixedStringBytes.test.ts index 99a30fe91..c384b918b 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/FixedStringBytes.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/FixedStringBytes.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readFixedStringBytes } from "../src/strings.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readFixedStringBytes } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Float32.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Float32.test.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Float32.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Float32.test.ts index 6ce0dae1a..8f16ea702 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Float32.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Float32.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readFloat32 } from "../src/floats.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readFloat32 } from "../src/readers/floats.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Float64.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Float64.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Float64.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Float64.test.ts index 7de12973a..ccea34b43 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Float64.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Float64.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readFloat64 } from "../src/floats.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readFloat64 } from "../src/readers/floats.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary/tests/Geo.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Geo.write.test.ts new file mode 100644 index 000000000..3e429bcbb --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/Geo.write.test.ts @@ -0,0 +1,100 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { encode } from "./encode.js"; +import { Sink } from "../src/writers/core.js"; +import { + writePoint, + writeRing, + writeLineString, + writePolygon, + writeMultiLineString, + writeMultiPolygon, + writeGeometry, +} from "../src/writers/geo.js"; +import { type Point } from "../src/readers/geo.js"; + +const ring: Point[] = [ + [0, 0], + [1, 2], +]; +const polygon: Point[][] = [ + [ + [0, 0], + [1, 0], + [1, 1], + ], +]; + +describe("geo writers", () => { + it("encodes a Point", async () => + expect(encode(writePoint, [1.5, 2.5])).toEqual( + await query("SELECT CAST((1.5, 2.5) AS Point) FORMAT RowBinary"), + )); + it("encodes a Ring", async () => + expect(encode(writeRing, ring)).toEqual( + await query("SELECT CAST([(0, 0), (1, 2)] AS Ring) FORMAT RowBinary"), + )); + it("encodes a LineString", async () => + expect(encode(writeLineString, ring)).toEqual( + await query( + "SELECT CAST([(0, 0), (1, 2)] AS LineString) FORMAT RowBinary", + ), + )); + it("encodes a Polygon", async () => + expect(encode(writePolygon, polygon)).toEqual( + await query( + "SELECT CAST([[(0, 0), (1, 0), (1, 1)]] AS Polygon) FORMAT RowBinary", + ), + )); + it("encodes a MultiLineString", async () => + expect( + encode(writeMultiLineString, [ + [ + [0, 0], + [1, 2], + ], + [[3, 4]], + ]), + ).toEqual( + await query( + "SELECT CAST([[(0, 0), (1, 2)], [(3, 4)]] AS MultiLineString) FORMAT RowBinary", + ), + )); + it("encodes a MultiPolygon", async () => + expect(encode(writeMultiPolygon, [polygon])).toEqual( + await query( + "SELECT CAST([[[(0, 0), (1, 0), (1, 1)]]] AS MultiPolygon) FORMAT RowBinary", + ), + )); +}); + +describe("writeGeometry", () => { + it("encodes a Point (discriminant 3)", async () => + expect(encode(writeGeometry, [3, [1.5, 2.5]])).toEqual( + await query( + "SELECT CAST(CAST((1.5, 2.5) AS Point) AS Geometry) SETTINGS allow_suspicious_variant_types = 1 FORMAT RowBinary", + ), + )); + it("encodes a LineString (discriminant 0)", async () => + expect(encode(writeGeometry, [0, ring])).toEqual( + await query( + "SELECT CAST(CAST([(0, 0), (1, 2)] AS LineString) AS Geometry) SETTINGS allow_suspicious_variant_types = 1 FORMAT RowBinary", + ), + )); + it("encodes a Polygon (discriminant 4)", async () => + expect(encode(writeGeometry, [4, polygon])).toEqual( + await query( + "SELECT CAST(CAST([[(0, 0), (1, 0), (1, 1)]] AS Polygon) AS Geometry) SETTINGS allow_suspicious_variant_types = 1 FORMAT RowBinary", + ), + )); + it("encodes NULL as a single 0xFF byte", () => + expect([...encode(writeGeometry, null)]).toEqual([0xff])); + + it("throws on an unknown discriminant without writing the byte", () => { + const sink = new Sink(Buffer.allocUnsafe(16)); + expect(() => writeGeometry(sink, [6, [1, 2]])).toThrow(RangeError); + // The discriminant is validated before the byte is written, so a rejected + // value leaves no partial payload behind. + expect(sink.bytes().length).toBe(0); + }); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Geometry.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Geometry.test.ts similarity index 94% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Geometry.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Geometry.test.ts index 28cd86d3b..40d696837 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Geometry.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Geometry.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readGeometry } from "../src/geo.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readGeometry } from "../src/readers/geo.js"; // Geometry's variant has "similar" alternatives (LineString/Ring), so the type // needs allow_suspicious_variant_types; the value still casts through a geo type. diff --git a/skills/clickhouse-js-node-rowbinary/tests/IP.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/IP.write.test.ts new file mode 100644 index 000000000..4d2ce824b --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/IP.write.test.ts @@ -0,0 +1,75 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { encode } from "./encode.js"; +import { Sink } from "../src/writers/core.js"; +import { + writeIPv4, + writeIPv6, + parseIPv4, + parseIPv6, +} from "../src/writers/ip.js"; + +describe("writeIPv4", () => { + it("encodes 0.0.0.0", async () => + expect(encode(writeIPv4, parseIPv4("0.0.0.0"))).toEqual( + await query("SELECT toIPv4('0.0.0.0') FORMAT RowBinary"), + )); + it("encodes 1.2.3.4", async () => + expect(encode(writeIPv4, parseIPv4("1.2.3.4"))).toEqual( + await query("SELECT toIPv4('1.2.3.4') FORMAT RowBinary"), + )); + it("encodes 192.168.0.1", async () => + expect(encode(writeIPv4, parseIPv4("192.168.0.1"))).toEqual( + await query("SELECT toIPv4('192.168.0.1') FORMAT RowBinary"), + )); + it("encodes 255.255.255.255", async () => + expect(encode(writeIPv4, parseIPv4("255.255.255.255"))).toEqual( + await query("SELECT toIPv4('255.255.255.255') FORMAT RowBinary"), + )); +}); + +describe("parseIPv4", () => { + it("packs the dotted quad big-endian", () => + expect(parseIPv4("1.2.3.4")).toBe(0x01020304)); + it("rejects an out-of-range octet", () => + expect(() => parseIPv4("1.2.3.256")).toThrow(RangeError)); +}); + +describe("writeIPv6", () => { + it("encodes ::", async () => + expect(encode(writeIPv6, parseIPv6("::"))).toEqual( + await query("SELECT toIPv6('::') FORMAT RowBinary"), + )); + it("encodes ::1", async () => + expect(encode(writeIPv6, parseIPv6("::1"))).toEqual( + await query("SELECT toIPv6('::1') FORMAT RowBinary"), + )); + it("encodes 2001:db8::1", async () => + expect(encode(writeIPv6, parseIPv6("2001:db8::1"))).toEqual( + await query("SELECT toIPv6('2001:db8::1') FORMAT RowBinary"), + )); + it("encodes fe80::a6:6ad3:dba0:1", async () => + expect(encode(writeIPv6, parseIPv6("fe80::a6:6ad3:dba0:1"))).toEqual( + await query("SELECT toIPv6('fe80::a6:6ad3:dba0:1') FORMAT RowBinary"), + )); + it("encodes the IPv4-mapped ::ffff:1.2.3.4", async () => + expect(encode(writeIPv6, parseIPv6("::ffff:1.2.3.4"))).toEqual( + await query("SELECT toIPv6('::ffff:1.2.3.4') FORMAT RowBinary"), + )); + + it("rejects non-16-byte input", () => + expect(() => + writeIPv6(new Sink(Buffer.allocUnsafe(16)), Buffer.alloc(4)), + ).toThrow(RangeError)); +}); + +describe("parseIPv6 validation", () => { + it("rejects a non-hex group instead of silently encoding 0", () => + expect(() => parseIPv6("2001:db8::gggg")).toThrow(RangeError)); + it("rejects a negative group instead of wrapping to 0xffff", () => + expect(() => parseIPv6("2001:db8::-1")).toThrow(RangeError)); + it("rejects an over-long (5-digit) group", () => + expect(() => parseIPv6("2001:db8::12345")).toThrow(RangeError)); + it("rejects an empty group", () => + expect(() => parseIPv6("2001:db8:::1")).toThrow(RangeError)); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/IPv4.test.ts b/skills/clickhouse-js-node-rowbinary/tests/IPv4.test.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/tests/IPv4.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/IPv4.test.ts index 90db4a2e8..161e80080 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/IPv4.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/IPv4.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { formatIPv4, readIPv4 } from "../src/ip.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { formatIPv4, readIPv4 } from "../src/readers/ip.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/IPv6.test.ts b/skills/clickhouse-js-node-rowbinary/tests/IPv6.test.ts similarity index 94% rename from skills/clickhouse-js-node-rowbinary-parser/tests/IPv6.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/IPv6.test.ts index 672e23f9f..c3516bff5 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/IPv6.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/IPv6.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { formatIPv6, readIPv6 } from "../src/ip.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { formatIPv6, readIPv6 } from "../src/readers/ip.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Int128.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Int128.test.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Int128.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Int128.test.ts index 81d69d7af..b4cc9ce57 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Int128.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Int128.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readInt128 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readInt128 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Int16.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Int16.test.ts similarity index 95% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Int16.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Int16.test.ts index 0d1641d3e..41812e736 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Int16.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Int16.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readInt16 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readInt16 } from "../src/readers/integers.js"; /** * Int16 is 2 bytes, little-endian, two's-complement. Each case selects the diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Int256.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Int256.test.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Int256.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Int256.test.ts index 52af1bcb6..894c2d63e 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Int256.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Int256.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readInt256 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readInt256 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Int32.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Int32.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Int32.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Int32.test.ts index 8bacd22c7..edb6f8646 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Int32.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Int32.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readInt32 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readInt32 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Int64.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Int64.test.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Int64.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Int64.test.ts index 260a0553f..80af7acca 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Int64.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Int64.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readInt64 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readInt64 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Int8.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Int8.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Int8.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Int8.test.ts index c521dcf5d..0bc0760e8 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Int8.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Int8.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readInt8 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readInt8 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary/tests/Integers.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Integers.write.test.ts new file mode 100644 index 000000000..82595e54e --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/Integers.write.test.ts @@ -0,0 +1,186 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { encode } from "./encode.js"; +import { + writeUInt8, + writeInt8, + writeUInt16, + writeInt16, + writeUInt32, + writeInt32, + writeUInt64, + writeInt64, + writeUInt128, + writeInt128, + writeUInt256, + writeInt256, +} from "../src/writers/integers.js"; + +describe("writeUInt8", () => { + it("encodes 0", async () => + expect(encode(writeUInt8, 0)).toEqual( + await query("SELECT toUInt8('0') FORMAT RowBinary"), + )); + it("encodes 255", async () => + expect(encode(writeUInt8, 255)).toEqual( + await query("SELECT toUInt8('255') FORMAT RowBinary"), + )); +}); + +describe("writeInt8", () => { + it("encodes -128", async () => + expect(encode(writeInt8, -128)).toEqual( + await query("SELECT toInt8('-128') FORMAT RowBinary"), + )); + it("encodes 0", async () => + expect(encode(writeInt8, 0)).toEqual( + await query("SELECT toInt8('0') FORMAT RowBinary"), + )); + it("encodes 127", async () => + expect(encode(writeInt8, 127)).toEqual( + await query("SELECT toInt8('127') FORMAT RowBinary"), + )); +}); + +describe("writeUInt16", () => { + it("encodes 0", async () => + expect(encode(writeUInt16, 0)).toEqual( + await query("SELECT toUInt16('0') FORMAT RowBinary"), + )); + it("encodes 65535", async () => + expect(encode(writeUInt16, 65535)).toEqual( + await query("SELECT toUInt16('65535') FORMAT RowBinary"), + )); +}); + +describe("writeInt16", () => { + it("encodes -32768", async () => + expect(encode(writeInt16, -32768)).toEqual( + await query("SELECT toInt16('-32768') FORMAT RowBinary"), + )); + it("encodes 32767", async () => + expect(encode(writeInt16, 32767)).toEqual( + await query("SELECT toInt16('32767') FORMAT RowBinary"), + )); +}); + +describe("writeUInt32", () => { + it("encodes 0", async () => + expect(encode(writeUInt32, 0)).toEqual( + await query("SELECT toUInt32('0') FORMAT RowBinary"), + )); + it("encodes 4294967295", async () => + expect(encode(writeUInt32, 4294967295)).toEqual( + await query("SELECT toUInt32('4294967295') FORMAT RowBinary"), + )); +}); + +describe("writeInt32", () => { + it("encodes -2147483648", async () => + expect(encode(writeInt32, -2147483648)).toEqual( + await query("SELECT toInt32('-2147483648') FORMAT RowBinary"), + )); + it("encodes 2147483647", async () => + expect(encode(writeInt32, 2147483647)).toEqual( + await query("SELECT toInt32('2147483647') FORMAT RowBinary"), + )); +}); + +describe("writeUInt64", () => { + it("encodes 0", async () => + expect(encode(writeUInt64, 0n)).toEqual( + await query("SELECT toUInt64('0') FORMAT RowBinary"), + )); + it("encodes 2^64 - 1", async () => + expect(encode(writeUInt64, 18446744073709551615n)).toEqual( + await query("SELECT toUInt64('18446744073709551615') FORMAT RowBinary"), + )); +}); + +describe("writeInt64", () => { + it("encodes -2^63", async () => + expect(encode(writeInt64, -9223372036854775808n)).toEqual( + await query("SELECT toInt64('-9223372036854775808') FORMAT RowBinary"), + )); + it("encodes 2^63 - 1", async () => + expect(encode(writeInt64, 9223372036854775807n)).toEqual( + await query("SELECT toInt64('9223372036854775807') FORMAT RowBinary"), + )); +}); + +describe("writeUInt128", () => { + it("encodes 0", async () => + expect(encode(writeUInt128, 0n)).toEqual( + await query("SELECT toUInt128('0') FORMAT RowBinary"), + )); + it("encodes 2^128 - 1", async () => + expect( + encode(writeUInt128, 340282366920938463463374607431768211455n), + ).toEqual( + await query( + "SELECT toUInt128('340282366920938463463374607431768211455') FORMAT RowBinary", + ), + )); +}); + +describe("writeInt128", () => { + it("encodes -2^127", async () => + expect( + encode(writeInt128, -170141183460469231731687303715884105728n), + ).toEqual( + await query( + "SELECT toInt128('-170141183460469231731687303715884105728') FORMAT RowBinary", + ), + )); + it("encodes 2^127 - 1", async () => + expect( + encode(writeInt128, 170141183460469231731687303715884105727n), + ).toEqual( + await query( + "SELECT toInt128('170141183460469231731687303715884105727') FORMAT RowBinary", + ), + )); +}); + +describe("writeUInt256", () => { + it("encodes 0", async () => + expect(encode(writeUInt256, 0n)).toEqual( + await query("SELECT toUInt256('0') FORMAT RowBinary"), + )); + it("encodes 2^256 - 1", async () => + expect( + encode( + writeUInt256, + 115792089237316195423570985008687907853269984665640564039457584007913129639935n, + ), + ).toEqual( + await query( + "SELECT toUInt256('115792089237316195423570985008687907853269984665640564039457584007913129639935') FORMAT RowBinary", + ), + )); +}); + +describe("writeInt256", () => { + it("encodes -2^255", async () => + expect( + encode( + writeInt256, + -57896044618658097711785492504343953926634992332820282019728792003956564819968n, + ), + ).toEqual( + await query( + "SELECT toInt256('-57896044618658097711785492504343953926634992332820282019728792003956564819968') FORMAT RowBinary", + ), + )); + it("encodes 2^255 - 1", async () => + expect( + encode( + writeInt256, + 57896044618658097711785492504343953926634992332820282019728792003956564819967n, + ), + ).toEqual( + await query( + "SELECT toInt256('57896044618658097711785492504343953926634992332820282019728792003956564819967') FORMAT RowBinary", + ), + )); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Interval.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Interval.test.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Interval.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Interval.test.ts index d60d4700b..95644f50f 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Interval.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Interval.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readInterval } from "../src/interval.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readInterval } from "../src/readers/interval.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/JSON.test.ts b/skills/clickhouse-js-node-rowbinary/tests/JSON.test.ts similarity index 95% rename from skills/clickhouse-js-node-rowbinary-parser/tests/JSON.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/JSON.test.ts index 05505feea..6435605ab 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/JSON.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/JSON.test.ts @@ -1,8 +1,8 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readDynamic } from "../src/dynamic.js"; -import { readJSON } from "../src/json.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readDynamic } from "../src/readers/dynamic.js"; +import { readJSON } from "../src/readers/json.js"; const J = "SETTINGS allow_experimental_json_type = 1, enable_json_type = 1"; diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/LineString.test.ts b/skills/clickhouse-js-node-rowbinary/tests/LineString.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/LineString.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/LineString.test.ts index d646d2a58..479dc46ac 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/LineString.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/LineString.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readLineString } from "../src/geo.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readLineString } from "../src/readers/geo.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Map.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Map.test.ts similarity index 88% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Map.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Map.test.ts index 1f6783a23..400a96359 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Map.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Map.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readMap, readNullable } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt32, readUInt8 } from "../src/integers.js"; -import { readString } from "../src/strings.js"; +import { readMap, readNullable } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt32, readUInt8 } from "../src/readers/integers.js"; +import { readString } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/MultiLineString.test.ts b/skills/clickhouse-js-node-rowbinary/tests/MultiLineString.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/MultiLineString.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/MultiLineString.test.ts index cae87a61d..c467b8f41 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/MultiLineString.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/MultiLineString.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readMultiLineString } from "../src/geo.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readMultiLineString } from "../src/readers/geo.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/MultiPolygon.test.ts b/skills/clickhouse-js-node-rowbinary/tests/MultiPolygon.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/MultiPolygon.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/MultiPolygon.test.ts index 0ae3fe4a2..7d3b97e61 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/MultiPolygon.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/MultiPolygon.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readMultiPolygon } from "../src/geo.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readMultiPolygon } from "../src/readers/geo.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary/tests/Nothing.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Nothing.write.test.ts new file mode 100644 index 000000000..a81ee1ab0 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/Nothing.write.test.ts @@ -0,0 +1,19 @@ +import { describe, expect, it } from "vitest"; +import { encode } from "./encode.js"; +import { writeNothing } from "../src/writers/nothing.js"; +import { writeArray, writeNullable } from "../src/writers/composite.js"; + +describe("writeNothing", () => { + it("throws if ever invoked directly", () => + expect(() => encode(writeNothing, undefined as never)).toThrow( + /Nothing is zero-width/, + )); + + it("is never invoked for an empty Array(Nothing)", () => + // Just the varint length 0x00; the element writer never runs. + expect([...encode(writeArray(writeNothing), [])]).toEqual([0x00])); + + it("is never invoked for a NULL Nullable(Nothing)", () => + // Just the NULL flag byte 0x01; the inner writer never runs. + expect([...encode(writeNullable(writeNothing), null)]).toEqual([0x01])); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Nullable.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Nullable.test.ts similarity index 89% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Nullable.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Nullable.test.ts index ab631439a..384cb1f27 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Nullable.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Nullable.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readNullable } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt32, readUInt8 } from "../src/integers.js"; -import { readString } from "../src/strings.js"; +import { readNullable } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt32, readUInt8 } from "../src/readers/integers.js"; +import { readString } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Point.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Point.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Point.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Point.test.ts index bb47c19f4..9bf20315f 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Point.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Point.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readPoint } from "../src/geo.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readPoint } from "../src/readers/geo.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Polygon.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Polygon.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Polygon.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Polygon.test.ts index 31fd6a10b..7fec66acb 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Polygon.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Polygon.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readPolygon } from "../src/geo.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readPolygon } from "../src/readers/geo.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Ring.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Ring.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Ring.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Ring.test.ts index 1c9bfcf12..f5c69c5c2 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Ring.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Ring.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readRing } from "../src/geo.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readRing } from "../src/readers/geo.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary/tests/Rows.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Rows.write.test.ts new file mode 100644 index 000000000..6d9042468 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/Rows.write.test.ts @@ -0,0 +1,194 @@ +import { describe, expect, it, vi } from "vitest"; +import diagnostics_channel from "node:diagnostics_channel"; +import { query } from "./clickhouse.js"; +import { + writeRows, + FLUSH_CHANNEL_NAME, + type WriteRowsFlush, +} from "../src/writers/rows.js"; +import { type Writer } from "../src/writers/core.js"; +import { writeTupleNamed } from "../src/writers/composite.js"; +import { writeUInt64, writeUInt32 } from "../src/writers/integers.js"; +import { writeString } from "../src/writers/strings.js"; + +type Row = { + id: bigint; + n: number; + name: string; +}; + +const writeRow = writeTupleNamed({ + id: writeUInt64, + n: writeUInt32, + name: writeString, +}); + +/** + * Drive `writeRows` to completion and concatenate every yielded buffer — the + * canonical driver loop. `bufferSize` sizes each (fixed) sink, so a small value + * forces the overflow + flush path; the default fits the rows in one buffer. + */ +function encodeRows( + write: Writer, + rows: Iterable, + bufferSize = 4096, +): Buffer { + return Buffer.concat([...writeRows(write)(rows, bufferSize)]); +} + +describe("writeRows", () => { + const rows: Row[] = Array.from({ length: 5 }, (_, i) => ({ + id: BigInt(i), + n: i * 10, + name: `row${i}`, + })); + const sql = + "SELECT toUInt64(number) AS id, toUInt32(number * 10) AS n, concat('row', toString(number)) AS name " + + "FROM numbers(5) FORMAT RowBinary"; + + it("encodes a plain RowBinary result of several rows", async () => { + expect(encodeRows(writeRow, rows)).toEqual(await query(sql)); + }); + + it("writes nothing for an empty array", () => + expect(encodeRows(writeRow, []).length).toBe(0)); + + it("flushes on buffer overflow and resumes — same bytes across a tiny buffer", async () => { + // A buffer far smaller than the whole result: the driver must flush full + // buffers mid-stream at row boundaries and reassemble to the identical bytes. + const expected = await query(sql); + const tiny = encodeRows(writeRow, rows, 20); // 20 holds one 17-byte row, not two + expect(tiny).toEqual(expected); + }); + + it("yields at row boundaries, never a half-written row", () => { + // bufferSize holds two rows + change but not three, so the first yield must + // land exactly on a row boundary (a whole number of rows), not mid-row. + const gen = writeRows(writeRow)(rows, 40); + const first = gen.next(); + expect(first.done).toBe(false); + const flushed = first.value as Buffer; + // A prefix check alone is NOT enough — a mid-row split is also a prefix. Prove + // the flush ends EXACTLY on a row boundary: collect the per-row cumulative byte + // offsets and assert the flushed length is one of them (i.e. a whole number of + // rows), AND that the bytes are the matching prefix of the full encoding. + const boundaries = new Set(); + let acc = 0; + for (const row of rows) { + acc += encodeRows(writeRow, [row]).length; + boundaries.add(acc); + } + expect(boundaries.has(flushed.length)).toBe(true); // ends on a row boundary + const full = encodeRows(writeRow, rows); + expect(full.subarray(0, flushed.length)).toEqual(flushed); // and is that prefix + }); + + it("yields independent buffers — a flushed buffer survives the next iteration", () => { + // Each flush gets a fresh buffer, so an earlier yield isn't clobbered when + // the generator resumes. Collect two buffers, then assert the first is intact. + const gen = writeRows(writeRow)(rows, 20); + const a = gen.next().value as Buffer; + const snapshot = Buffer.from(a); // independent copy of what we saw first + gen.next(); // resume: writes the next row into a NEW buffer + expect(a).toEqual(snapshot); // `a` must be untouched + }); + + it("grows the buffer to fit a row larger than bufferSize, warning once", async () => { + const expected = await query(sql); + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + try { + // bufferSize 4 can't hold even one 17-byte row: the buffer doubles + // (4→8→16→32) until the row fits — no data lost, nothing thrown — and it + // warns exactly once even though it grew several times. + expect(encodeRows(writeRow, rows, 4)).toEqual(expected); + expect(warn).toHaveBeenCalledTimes(1); + expect(warn.mock.calls[0]?.[0]).toMatch(/didn't fit bufferSize=4/); + } finally { + warn.mockRestore(); + } + }); + + it("does not warn when every row fits the buffer", () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + try { + encodeRows(writeRow, rows); // default 4096 fits every row + expect(warn).not.toHaveBeenCalled(); + } finally { + warn.mockRestore(); + } + }); + + it("rejects a non-positive or non-integer bufferSize instead of looping forever", () => { + // 0 / NaN would make the growth loop spin (size *= 2 never escapes 0/NaN); + // fail fast on first .next() with a clear error. + for (const bad of [0, -1, NaN, 1.5, Infinity]) { + const gen = writeRows(writeRow)(rows, bad); + expect(() => gen.next()).toThrow(/bufferSize must be a positive integer/); + } + }); + + /** Run `body` with a subscriber on the flush channel, collecting every event. */ + function withFlushEvents(body: () => void): WriteRowsFlush[] { + const events: WriteRowsFlush[] = []; + const onMessage = (msg: unknown) => events.push(msg as WriteRowsFlush); + diagnostics_channel.subscribe(FLUSH_CHANNEL_NAME, onMessage); + try { + body(); + } finally { + diagnostics_channel.unsubscribe(FLUSH_CHANNEL_NAME, onMessage); + } + return events; + } + + it("publishes a flush event per buffer — every 'full' batch fills its capacity, then one 'end'", () => { + // bufferSize 20 holds one 17-byte row: rows 0..3 each flush a 'full' buffer + // when the next row overflows, row 4 comes out as the 'end' batch. + const events = withFlushEvents(() => encodeRows(writeRow, rows, 20)); + expect(events.map((e) => e.reason)).toEqual([ + "full", + "full", + "full", + "full", + "end", + ]); + // Every buffer reports its real capacity and the configured size; used never + // exceeds capacity; nothing grew, so capacity stays at bufferSize. + for (const e of events) { + expect(e.capacityBytes).toBe(20); + expect(e.bufferSize).toBe(20); + expect(e.usedBytes).toBeLessThanOrEqual(e.capacityBytes); + } + // The four mid-stream flushes each carried exactly one 17-byte row. + expect(events.slice(0, 4).every((e) => e.usedBytes === 17)).toBe(true); + // Summed used bytes equal the whole payload — nothing is double-counted. + const total = events.reduce((n, e) => n + e.usedBytes, 0); + expect(total).toBe(encodeRows(writeRow, rows).length); + }); + + it("reports the grown capacity and original bufferSize so overflow is identifiable", () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + try { + // bufferSize 4 grows to 32 to fit a 17-byte row; the published capacity is + // the grown size while bufferSize stays 4, so `capacityBytes > bufferSize` + // flags the overflow and `usedBytes / bufferSize` gives its magnitude. + const events = withFlushEvents(() => encodeRows(writeRow, rows, 4)); + expect(events.every((e) => e.capacityBytes === 32)).toBe(true); + expect(events.every((e) => e.bufferSize === 4)).toBe(true); + expect(events.every((e) => e.capacityBytes > e.bufferSize)).toBe(true); + } finally { + warn.mockRestore(); + } + }); + + it("stops publishing to a subscriber once it unsubscribes", () => { + const events: WriteRowsFlush[] = []; + const onMessage = (msg: unknown) => events.push(msg as WriteRowsFlush); + diagnostics_channel.subscribe(FLUSH_CHANNEL_NAME, onMessage); + encodeRows(writeRow, rows, 20); + const afterFirst = events.length; + expect(afterFirst).toBeGreaterThan(0); + diagnostics_channel.unsubscribe(FLUSH_CHANNEL_NAME, onMessage); + encodeRows(writeRow, rows, 20); // a second run with no subscriber + expect(events.length).toBe(afterFirst); // nothing more delivered + }); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/String.test.ts b/skills/clickhouse-js-node-rowbinary/tests/String.test.ts similarity index 93% rename from skills/clickhouse-js-node-rowbinary-parser/tests/String.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/String.test.ts index 335a247f1..5da1696b6 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/String.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/String.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readString } from "../src/strings.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readString } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary/tests/String.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/String.write.test.ts new file mode 100644 index 000000000..ad4f750e0 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/String.write.test.ts @@ -0,0 +1,64 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { encode } from "./encode.js"; +import { + writeString, + writeStringBytes, + writeFixedString, + writeFixedStringBytes, +} from "../src/writers/strings.js"; + +describe("writeString", () => { + it("encodes the empty string", async () => + expect(encode(writeString, "")).toEqual( + await query("SELECT '' FORMAT RowBinary"), + )); + it("encodes an ASCII string", async () => + expect(encode(writeString, "hello")).toEqual( + await query("SELECT 'hello' FORMAT RowBinary"), + )); + it("encodes a multi-byte UTF-8 string", async () => + expect(encode(writeString, "héllo · 日本")).toEqual( + await query("SELECT 'héllo · 日本' FORMAT RowBinary"), + )); + it("encodes a string longer than a 1-byte varint length", async () => + expect(encode(writeString, "x".repeat(300))).toEqual( + await query("SELECT repeat('x', 300) FORMAT RowBinary"), + )); +}); + +describe("writeStringBytes", () => { + it("writes raw (non-UTF-8) bytes from a Uint8Array", () => { + // varint length 3, then the raw bytes verbatim. + expect([ + ...encode(writeStringBytes, Buffer.from([0xff, 0x00, 0xfe])), + ]).toEqual([0x03, 0xff, 0x00, 0xfe]); + }); +}); + +describe("writeFixedString", () => { + it("pads to the column width with trailing NULs", async () => + expect(encode(writeFixedString(16), "abc")).toEqual( + await query("SELECT toFixedString('abc', 16) FORMAT RowBinary"), + )); + + it("throws when the value exceeds the size", () => + expect(() => encode(writeFixedString(2), "abc")).toThrow(RangeError)); +}); + +describe("writeFixedStringBytes", () => { + it("pads raw bytes to the column width with trailing NULs", async () => + expect(encode(writeFixedStringBytes(8), Buffer.from("abc"))).toEqual( + await query("SELECT toFixedString('abc', 8) FORMAT RowBinary"), + )); + + it("zero-pads a short value", () => + expect([...encode(writeFixedStringBytes(4), Buffer.from([1, 2]))]).toEqual([ + 1, 2, 0, 0, + ])); + + it("throws when the value exceeds the size", () => + expect(() => + encode(writeFixedStringBytes(2), Buffer.from([1, 2, 3])), + ).toThrow(RangeError)); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Time.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Time.test.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Time.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Time.test.ts index 95d67712c..e64ca575d 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Time.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Time.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { formatTime, readTime } from "../src/time.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { formatTime, readTime } from "../src/readers/time.js"; // Time / Time64 need enable_time_time64_type; pass it inline via SETTINGS. async function reader(expr: string): Promise { diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Time64.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Time64.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Time64.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Time64.test.ts index c7dccf71c..ec2281815 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Time64.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Time64.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { formatTime64, readTime64 } from "../src/time.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { formatTime64, readTime64 } from "../src/readers/time.js"; async function reader(expr: string): Promise { return new Cursor( diff --git a/skills/clickhouse-js-node-rowbinary/tests/TimeInterval.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/TimeInterval.write.test.ts new file mode 100644 index 000000000..207a74b93 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/TimeInterval.write.test.ts @@ -0,0 +1,68 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { encode } from "./encode.js"; +import { + writeTime, + writeTime64, + parseTime, + parseTime64, +} from "../src/writers/time.js"; +import { writeInterval } from "../src/writers/interval.js"; + +describe("writeTime", () => { + it("encodes 12:34:56", async () => + expect(encode(writeTime, parseTime("12:34:56"))).toEqual( + await query( + "SELECT CAST('12:34:56' AS Time) SETTINGS enable_time_time64_type = 1 FORMAT RowBinary", + ), + )); + it("encodes a negative -01:02:03", async () => + expect(encode(writeTime, parseTime("-01:02:03"))).toEqual( + await query( + "SELECT CAST('-01:02:03' AS Time) SETTINGS enable_time_time64_type = 1 FORMAT RowBinary", + ), + )); +}); + +describe("writeTime64", () => { + it("encodes Time64(3)", async () => + expect(encode(writeTime64, parseTime64("12:34:56.789", 3))).toEqual( + await query( + "SELECT toTime64('12:34:56.789', 3) SETTINGS enable_time_time64_type = 1 FORMAT RowBinary", + ), + )); + it("encodes a negative Time64(6)", async () => + expect(encode(writeTime64, parseTime64("-01:02:03.000004", 6))).toEqual( + await query( + "SELECT toTime64('-01:02:03.000004', 6) SETTINGS enable_time_time64_type = 1 FORMAT RowBinary", + ), + )); +}); + +describe("parseTime64", () => { + it("scales whole seconds and the fraction to ticks", () => + expect(parseTime64("12:34:56.789", 3)).toEqual([ + (12n * 3600n + 34n * 60n + 56n) * 1000n + 789n, + 3, + ])); + it("handles a negative value with a short fraction", () => + expect(parseTime64("-01:02:03.5", 1)).toEqual([ + -((1n * 3600n + 2n * 60n + 3n) * 10n + 5n), + 1, + ])); +}); + +describe("writeInterval", () => { + it("encodes toIntervalDay(7)", async () => + expect(encode(writeInterval, 7n)).toEqual( + await query("SELECT toIntervalDay(7) FORMAT RowBinary"), + )); + it("encodes a negative toIntervalSecond(-90)", async () => + expect(encode(writeInterval, -90n)).toEqual( + await query("SELECT toIntervalSecond(-90) FORMAT RowBinary"), + )); + it("encodes toIntervalYear(2)", async () => + expect(encode(writeInterval, 2n)).toEqual( + await query("SELECT toIntervalYear(2) FORMAT RowBinary"), + )); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Tuple.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Tuple.test.ts similarity index 85% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Tuple.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Tuple.test.ts index 5b37552b1..b171e5e64 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Tuple.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Tuple.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readNullable, readTuple } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt32, readUInt8 } from "../src/integers.js"; -import { readString } from "../src/strings.js"; +import { readNullable, readTuple } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt32, readUInt8 } from "../src/readers/integers.js"; +import { readString } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/TupleNamed.test.ts b/skills/clickhouse-js-node-rowbinary/tests/TupleNamed.test.ts similarity index 86% rename from skills/clickhouse-js-node-rowbinary-parser/tests/TupleNamed.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/TupleNamed.test.ts index c31d55f64..e14058e11 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/TupleNamed.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/TupleNamed.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readNullable, readTupleNamed } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt32, readUInt8 } from "../src/integers.js"; -import { readString } from "../src/strings.js"; +import { readNullable, readTupleNamed } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt32, readUInt8 } from "../src/readers/integers.js"; +import { readString } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt128.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UInt128.test.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/tests/UInt128.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/UInt128.test.ts index b90fc3b71..9b5845a19 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt128.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/UInt128.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt128 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt128 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt16.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UInt16.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/UInt16.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/UInt16.test.ts index 0d1b0ab3a..5793f244b 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt16.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/UInt16.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt16 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt16 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt256.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UInt256.test.ts similarity index 92% rename from skills/clickhouse-js-node-rowbinary-parser/tests/UInt256.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/UInt256.test.ts index 83478d7fd..cf0b355a1 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt256.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/UInt256.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt256 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt256 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt32.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UInt32.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/UInt32.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/UInt32.test.ts index a0fd2025e..c39b1e167 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt32.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/UInt32.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt32 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt32 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt64.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UInt64.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/UInt64.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/UInt64.test.ts index d2809929a..db2363597 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt64.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/UInt64.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt64 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt64 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt8.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UInt8.test.ts similarity index 90% rename from skills/clickhouse-js-node-rowbinary-parser/tests/UInt8.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/UInt8.test.ts index 0921192e6..61f2955ca 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/UInt8.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/UInt8.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt8 } from "../src/integers.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt8 } from "../src/readers/integers.js"; describe("readUInt8", () => { it("reads sequential unsigned bytes", () => { diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/UUID.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UUID.test.ts similarity index 96% rename from skills/clickhouse-js-node-rowbinary-parser/tests/UUID.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/UUID.test.ts index a28da0705..eb5548cc5 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/UUID.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/UUID.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { formatUUID, formatUUIDTable, readUUID } from "../src/uuid.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { formatUUID, formatUUIDTable, readUUID } from "../src/readers/uuid.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary/tests/UUID.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UUID.write.test.ts new file mode 100644 index 000000000..39f573282 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/UUID.write.test.ts @@ -0,0 +1,60 @@ +import { describe, expect, it } from "vitest"; +import { query } from "./clickhouse.js"; +import { encode } from "./encode.js"; +import { Sink } from "../src/writers/core.js"; +import { + writeUUID, + writeUUIDBigInt, + writeUUIDHiLo, + parseUUID, +} from "../src/writers/uuid.js"; + +const SAMPLE = "61f0c404-5cb3-11e7-907b-a6006ad3dba0"; +// ClickHouse stores a UUID as two little-endian UInt64 halves (high then low). +const HI = 0x61f0c4045cb311e7n; +const LO = 0x907ba6006ad3dba0n; +// The same UUID as one 128-bit value (hi in the high 64 bits, lo in the low). +const HI_LO_UUID: bigint = (HI << 64n) | LO; +const WIRE = [ + 0xe7, + 0x11, + 0xb3, + 0x5c, + 0x04, + 0xc4, + 0xf0, + 0x61, // high half, little-endian + 0xa0, + 0xdb, + 0xd3, + 0x6a, + 0x00, + 0xa6, + 0x7b, + 0x90, // low half, little-endian +]; + +describe("UUID writers", () => { + it("writeUUID copies the raw 16 wire bytes verbatim", async () => + expect(encode(writeUUID, Buffer.from(WIRE))).toEqual( + await query(`SELECT toUUID('${SAMPLE}') FORMAT RowBinary`), + )); + + it("writeUUIDBigInt splits a 128-bit bigint into the two LE halves", async () => + expect(encode(writeUUIDBigInt, HI_LO_UUID)).toEqual( + await query(`SELECT toUUID('${SAMPLE}') FORMAT RowBinary`), + )); + + it("writeUUIDHiLo writes the two raw [hi, lo] halves", async () => + expect(encode(writeUUIDHiLo, [HI, LO])).toEqual( + await query(`SELECT toUUID('${SAMPLE}') FORMAT RowBinary`), + )); + + it("parseUUID turns the canonical string into the wire bytes", () => + expect([...parseUUID(SAMPLE)]).toEqual(WIRE)); + + it("writeUUID rejects non-16-byte input", () => + expect(() => + writeUUID(new Sink(Buffer.allocUnsafe(16)), Buffer.alloc(15)), + ).toThrow(RangeError)); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/UUIDBigInt.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UUIDBigInt.test.ts similarity index 93% rename from skills/clickhouse-js-node-rowbinary-parser/tests/UUIDBigInt.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/UUIDBigInt.test.ts index 3902b6adb..406b85eac 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/UUIDBigInt.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/UUIDBigInt.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUUIDBigInt } from "../src/uuid.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUUIDBigInt } from "../src/readers/uuid.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/UUIDHiLo.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UUIDHiLo.test.ts similarity index 95% rename from skills/clickhouse-js-node-rowbinary-parser/tests/UUIDHiLo.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/UUIDHiLo.test.ts index f01ff23f2..b17e6d0ab 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/UUIDHiLo.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/UUIDHiLo.test.ts @@ -1,12 +1,12 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; import { formatUUID, readUUID, readUUIDBigInt, readUUIDHiLo, -} from "../src/uuid.js"; +} from "../src/readers/uuid.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/UVarint.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UVarint.test.ts similarity index 96% rename from skills/clickhouse-js-node-rowbinary-parser/tests/UVarint.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/UVarint.test.ts index 8229c530c..83c65baed 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/UVarint.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/UVarint.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUVarint } from "../src/varint.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUVarint } from "../src/readers/varint.js"; /** * RowBinary prefixes every String with its length as a LEB128 unsigned varint. diff --git a/skills/clickhouse-js-node-rowbinary/tests/UVarint.write.test.ts b/skills/clickhouse-js-node-rowbinary/tests/UVarint.write.test.ts new file mode 100644 index 000000000..951c73fe3 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/UVarint.write.test.ts @@ -0,0 +1,58 @@ +import { describe, expect, it } from "vitest"; +import { Sink, reserve, BufferFull } from "../src/writers/core.js"; +import { encode } from "./encode.js"; +import { writeUVarint } from "../src/writers/varint.js"; + +/** A fresh sink over a fixed buffer of `capacity` bytes. */ +function sink(capacity: number): Sink { + return new Sink(Buffer.allocUnsafe(capacity)); +} + +describe("Sink / reserve", () => { + it("advances and returns the start offset", () => { + const s = sink(8); + expect(reserve(s, 4)).toBe(0); + expect(reserve(s, 2)).toBe(4); + expect(s.pos).toBe(6); + }); + + it("throws BufferFull without advancing when the buffer is full", () => { + const s = sink(4); + expect(reserve(s, 4)).toBe(0); + let thrown: unknown; + try { + reserve(s, 1); + } catch (err) { + thrown = err; + } + expect(thrown).toBe(BufferFull); + expect(s.pos).toBe(4); // position is unchanged on overflow + }); + + it("bytes() returns only the written prefix", () => { + const s = sink(64); + s.buf[reserve(s, 1)] = 0x01; + expect(s.bytes().length).toBe(1); + }); +}); + +describe("writeUVarint", () => { + // Hard-coded LEB128 expectations — independent of the reader. + it("encodes 0 as a single 0x00 byte", () => + expect([...encode(writeUVarint, 0)]).toEqual([0x00])); + it("encodes 1", () => expect([...encode(writeUVarint, 1)]).toEqual([0x01])); + it("encodes 127 in one byte", () => + expect([...encode(writeUVarint, 127)]).toEqual([0x7f])); + it("encodes 128 in two bytes", () => + expect([...encode(writeUVarint, 128)]).toEqual([0x80, 0x01])); + it("encodes 300 as [0xac, 0x02]", () => + expect([...encode(writeUVarint, 300)]).toEqual([0xac, 0x02])); + it("encodes 16383 (2-byte boundary)", () => + expect([...encode(writeUVarint, 16383)]).toEqual([0xff, 0x7f])); + it("encodes 16384 (3-byte boundary)", () => + expect([...encode(writeUVarint, 16384)]).toEqual([0x80, 0x80, 0x01])); + it("encodes Number.MAX_SAFE_INTEGER (2^53 - 1)", () => + expect([...encode(writeUVarint, Number.MAX_SAFE_INTEGER)]).toEqual([ + 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x0f, + ])); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/Variant.test.ts b/skills/clickhouse-js-node-rowbinary/tests/Variant.test.ts similarity index 88% rename from skills/clickhouse-js-node-rowbinary-parser/tests/Variant.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/Variant.test.ts index 98e86a90f..cbcd90c4a 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/Variant.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/Variant.test.ts @@ -1,10 +1,10 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readVariant } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readFloat64 } from "../src/floats.js"; -import { readUInt64, readUInt8 } from "../src/integers.js"; -import { readString } from "../src/strings.js"; +import { readVariant } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readFloat64 } from "../src/readers/floats.js"; +import { readUInt64, readUInt8 } from "../src/readers/integers.js"; +import { readString } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor( diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/advance.test.ts b/skills/clickhouse-js-node-rowbinary/tests/advance.test.ts similarity index 95% rename from skills/clickhouse-js-node-rowbinary-parser/tests/advance.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/advance.test.ts index ae39a2e4a..9a5a62157 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/advance.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/advance.test.ts @@ -1,8 +1,8 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt64 } from "../src/integers.js"; -import { readString } from "../src/strings.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt64 } from "../src/readers/integers.js"; +import { readString } from "../src/readers/strings.js"; /** * `advance` / `NeedMoreData` tests: the per-read "need more bytes" throw that is diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/aggregateFunction.test.ts b/skills/clickhouse-js-node-rowbinary/tests/aggregateFunction.test.ts similarity index 94% rename from skills/clickhouse-js-node-rowbinary-parser/tests/aggregateFunction.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/aggregateFunction.test.ts index b1d22654b..d66b691bd 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/aggregateFunction.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/aggregateFunction.test.ts @@ -1,8 +1,8 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readAggregateFunction } from "../src/aggregateFunction.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt64, readUInt8 } from "../src/integers.js"; +import { readAggregateFunction } from "../src/readers/aggregateFunction.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt64, readUInt8 } from "../src/readers/integers.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/carts.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/carts.bench.ts similarity index 96% rename from skills/clickhouse-js-node-rowbinary-parser/tests/carts.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/carts.bench.ts index d93884897..2cdb26a74 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/carts.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/carts.bench.ts @@ -1,6 +1,6 @@ import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { type Reader, Cursor } from "../src/core.js"; +import { type Reader, Cursor } from "../src/readers/core.js"; import { type CartRow, readCartRow, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/carts.example.test.ts b/skills/clickhouse-js-node-rowbinary/tests/carts.example.test.ts similarity index 94% rename from skills/clickhouse-js-node-rowbinary-parser/tests/carts.example.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/carts.example.test.ts index 32229373c..702e11c7b 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/carts.example.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/carts.example.test.ts @@ -1,8 +1,8 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; +import { Cursor } from "../src/readers/core.js"; import { type CartRow, readCartRow } from "../src/examples/carts.js"; -import { readRows } from "../src/rows.js"; +import { readRows } from "../src/readers/rows.js"; /** * Runs the `carts` example end to end (nested generics): an Array of named diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/clickhouse.ts b/skills/clickhouse-js-node-rowbinary/tests/clickhouse.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/tests/clickhouse.ts rename to skills/clickhouse-js-node-rowbinary/tests/clickhouse.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/coalesceChunks.test.ts b/skills/clickhouse-js-node-rowbinary/tests/coalesceChunks.test.ts similarity index 95% rename from skills/clickhouse-js-node-rowbinary-parser/tests/coalesceChunks.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/coalesceChunks.test.ts index 2f1b946d7..a934ddfc1 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/coalesceChunks.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/coalesceChunks.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; -import { readUInt64 } from "../src/integers.js"; -import { coalesceChunks, streamRowBatches } from "../src/stream.js"; -import { readString } from "../src/strings.js"; +import { Cursor } from "../src/readers/core.js"; +import { readUInt64 } from "../src/readers/integers.js"; +import { coalesceChunks, streamRowBatches } from "../src/readers/stream.js"; +import { readString } from "../src/readers/strings.js"; /** * `coalesceChunks` merges a too-small chunk stream into chunks of at least diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/columnar.test.ts b/skills/clickhouse-js-node-rowbinary/tests/columnar.test.ts similarity index 98% rename from skills/clickhouse-js-node-rowbinary-parser/tests/columnar.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/columnar.test.ts index d04d5a53b..76648bb1b 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/columnar.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/columnar.test.ts @@ -1,6 +1,6 @@ import { afterEach, describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { streamSensorColumns } from "../src/columnar.js"; +import { streamSensorColumns } from "../src/readers/columnar.js"; /** * Eval for the streaming columnar decoder (`streamSensorColumns`). The example diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/combinations.generated.test.ts b/skills/clickhouse-js-node-rowbinary/tests/combinations.generated.test.ts similarity index 97% rename from skills/clickhouse-js-node-rowbinary-parser/tests/combinations.generated.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/combinations.generated.test.ts index 84dd4421c..0b1e735b4 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/combinations.generated.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/combinations.generated.test.ts @@ -6,10 +6,10 @@ import { readNullable, readTuple, readVariant, -} from "../src/composite.js"; -import { NeedMoreData, type Reader, Cursor } from "../src/core.js"; -import { readInt32, readUInt8 } from "../src/integers.js"; -import { readString } from "../src/strings.js"; +} from "../src/readers/composite.js"; +import { NeedMoreData, type Reader, Cursor } from "../src/readers/core.js"; +import { readInt32, readUInt8 } from "../src/readers/integers.js"; +import { readString } from "../src/readers/strings.js"; /** * GENERATED type-combination coverage — the systematic companion to the curated, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/compile.test.ts b/skills/clickhouse-js-node-rowbinary/tests/compile.test.ts similarity index 96% rename from skills/clickhouse-js-node-rowbinary-parser/tests/compile.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/compile.test.ts index 2c5e598ab..475cec9e3 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/compile.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/compile.test.ts @@ -1,8 +1,8 @@ import { describe, expect, it } from "vitest"; import { parseDataType } from "@clickhouse/datatype-parser"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; -import { astToReader, RowBinaryTypeError } from "../src/compile.js"; +import { Cursor } from "../src/readers/core.js"; +import { astToReader, RowBinaryTypeError } from "../src/readers/compile.js"; // compile.ts is AST in, reader out. These tests exercise that fold directly: // parse a type string, fold it to a Reader, and decode plain RowBinary value diff --git a/skills/clickhouse-js-node-rowbinary/tests/encode.ts b/skills/clickhouse-js-node-rowbinary/tests/encode.ts new file mode 100644 index 000000000..4f69d3a74 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/encode.ts @@ -0,0 +1,19 @@ +import { Sink, type Writer } from "../src/writers/core.js"; + +/** + * Encode a single value with `write` into a fresh {@link Sink} and return the + * bytes. The shared shape of every `*.write.test.ts`. + * + * Writer tests are INDEPENDENT of the readers: the test supplies the JS value + * itself and asserts the encoded bytes equal what ClickHouse emits (the source of + * truth, see `clickhouse.ts`) or a hard-coded expectation — never a value decoded + * by the reader. That way a reader bug cannot mask a writer bug, and vice versa. + * + * `capacity` sizes the (fixed-length) sink buffer; the default comfortably fits + * every value under test. + */ +export function encode(write: Writer, value: T, capacity = 4096): Buffer { + const sink = new Sink(Buffer.allocUnsafe(capacity)); + write(sink, value); + return Buffer.from(sink.bytes()); +} diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/events.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/events.bench.ts similarity index 96% rename from skills/clickhouse-js-node-rowbinary-parser/tests/events.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/events.bench.ts index a08a9b5fd..e5f89c34d 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/events.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/events.bench.ts @@ -1,6 +1,6 @@ import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { type Reader, Cursor } from "../src/core.js"; +import { type Reader, Cursor } from "../src/readers/core.js"; import { type EventRow, readEventRow, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/events.example.test.ts b/skills/clickhouse-js-node-rowbinary/tests/events.example.test.ts similarity index 94% rename from skills/clickhouse-js-node-rowbinary-parser/tests/events.example.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/events.example.test.ts index 65ec4b2da..49fc63b53 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/events.example.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/events.example.test.ts @@ -1,8 +1,8 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; +import { Cursor } from "../src/readers/core.js"; import { type EventRow, readEventRow } from "../src/examples/events.js"; -import { readRows } from "../src/rows.js"; +import { readRows } from "../src/readers/rows.js"; /** * Runs the `events` example end to end: CREATE a table, populate it (here via diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/framing-interleaved.test.ts b/skills/clickhouse-js-node-rowbinary/tests/framing-interleaved.test.ts similarity index 97% rename from skills/clickhouse-js-node-rowbinary-parser/tests/framing-interleaved.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/framing-interleaved.test.ts index 55b1e2ced..183269535 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/framing-interleaved.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/framing-interleaved.test.ts @@ -6,12 +6,12 @@ import { readNullable, readTuple, readVariant, -} from "../src/composite.js"; -import { Cursor } from "../src/core.js"; -import { readDynamic } from "../src/dynamic.js"; -import { readInt32, readUInt8 } from "../src/integers.js"; -import { readJSON } from "../src/json.js"; -import { readString } from "../src/strings.js"; +} from "../src/readers/composite.js"; +import { Cursor } from "../src/readers/core.js"; +import { readDynamic } from "../src/readers/dynamic.js"; +import { readInt32, readUInt8 } from "../src/readers/integers.js"; +import { readJSON } from "../src/readers/json.js"; +import { readString } from "../src/readers/strings.js"; /** * Interleaving framing tests: TWO variable-length / self-describing columns are diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/framing-nested.test.ts b/skills/clickhouse-js-node-rowbinary/tests/framing-nested.test.ts similarity index 97% rename from skills/clickhouse-js-node-rowbinary-parser/tests/framing-nested.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/framing-nested.test.ts index f34b27d60..425a85cc1 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/framing-nested.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/framing-nested.test.ts @@ -6,12 +6,12 @@ import { readNullable, readTuple, readVariant, -} from "../src/composite.js"; -import { Cursor } from "../src/core.js"; -import { readDynamic } from "../src/dynamic.js"; -import { readInt32, readUInt8 } from "../src/integers.js"; -import { readJSON } from "../src/json.js"; -import { readString } from "../src/strings.js"; +} from "../src/readers/composite.js"; +import { Cursor } from "../src/readers/core.js"; +import { readDynamic } from "../src/readers/dynamic.js"; +import { readInt32, readUInt8 } from "../src/readers/integers.js"; +import { readJSON } from "../src/readers/json.js"; +import { readString } from "../src/readers/strings.js"; /** * Framing tests for NESTED self-describing / variable-length types — the place diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/framing.test.ts b/skills/clickhouse-js-node-rowbinary/tests/framing.test.ts similarity index 93% rename from skills/clickhouse-js-node-rowbinary-parser/tests/framing.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/framing.test.ts index 8d19363a6..54026b145 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/framing.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/framing.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readBool } from "../src/bool.js"; +import { readBool } from "../src/readers/bool.js"; import { readArray, readMap, @@ -9,23 +9,27 @@ import { readTuple, readTupleNamed, readVariant, -} from "../src/composite.js"; -import { Cursor } from "../src/core.js"; +} from "../src/readers/composite.js"; +import { Cursor } from "../src/readers/core.js"; import { readDate, readDate32, readDateTime, readDateTime64, -} from "../src/datetime.js"; +} from "../src/readers/datetime.js"; import { readDecimal128, readDecimal256, readDecimal32, readDecimal64, -} from "../src/decimals.js"; -import { readDynamic } from "../src/dynamic.js"; -import { readEnum16, readEnum8 } from "../src/enums.js"; -import { readBFloat16, readFloat32, readFloat64 } from "../src/floats.js"; +} from "../src/readers/decimals.js"; +import { readDynamic } from "../src/readers/dynamic.js"; +import { readEnum16, readEnum8 } from "../src/readers/enums.js"; +import { + readBFloat16, + readFloat32, + readFloat64, +} from "../src/readers/floats.js"; import { readGeometry, readLineString, @@ -34,7 +38,7 @@ import { readPoint, readPolygon, readRing, -} from "../src/geo.js"; +} from "../src/readers/geo.js"; import { readInt128, readInt16, @@ -48,13 +52,18 @@ import { readUInt32, readUInt64, readUInt8, -} from "../src/integers.js"; -import { readInterval } from "../src/interval.js"; -import { formatIPv4, formatIPv6, readIPv4, readIPv6 } from "../src/ip.js"; -import { readJSON } from "../src/json.js"; -import { readFixedString, readString } from "../src/strings.js"; -import { readTime, readTime64 } from "../src/time.js"; -import { formatUUID, readUUID } from "../src/uuid.js"; +} from "../src/readers/integers.js"; +import { readInterval } from "../src/readers/interval.js"; +import { + formatIPv4, + formatIPv6, + readIPv4, + readIPv6, +} from "../src/readers/ip.js"; +import { readJSON } from "../src/readers/json.js"; +import { readFixedString, readString } from "../src/readers/strings.js"; +import { readTime, readTime64 } from "../src/readers/time.js"; +import { formatUUID, readUUID } from "../src/readers/uuid.js"; /** * Framing tests: every type is placed as the MIDDLE column between two distinct @@ -340,14 +349,28 @@ describe("framing: i32, X, i32 — the middle reader must stop at the exact byte it("Enum8", async () => { const r = await framed("CAST('b' AS Enum8('a' = 1, 'b' = 2))"); expect(readInt32(r)).toBe(LEAD); - expect(readEnum8(r)).toBe(2); + expect( + readEnum8( + new Map([ + [1, "a"], + [2, "b"], + ]), + )(r), + ).toBe("b"); expect(readInt32(r)).toBe(TRAIL); }); it("Enum16", async () => { const r = await framed("CAST('big' AS Enum16('small' = 1, 'big' = 300))"); expect(readInt32(r)).toBe(LEAD); - expect(readEnum16(r)).toBe(300); + expect( + readEnum16( + new Map([ + [1, "small"], + [300, "big"], + ]), + )(r), + ).toBe("big"); expect(readInt32(r)).toBe(TRAIL); }); }); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/iot.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/iot.bench.ts similarity index 98% rename from skills/clickhouse-js-node-rowbinary-parser/tests/iot.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/iot.bench.ts index f104170c7..fa5c2c226 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/iot.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/iot.bench.ts @@ -1,6 +1,6 @@ import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { type Reader, Cursor } from "../src/core.js"; +import { type Reader, Cursor } from "../src/readers/core.js"; import { type IotRow, readIotRow, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/iot.columnar.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/iot.columnar.bench.ts similarity index 97% rename from skills/clickhouse-js-node-rowbinary-parser/tests/iot.columnar.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/iot.columnar.bench.ts index 6148b7b6f..844e2dfd5 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/iot.columnar.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/iot.columnar.bench.ts @@ -1,6 +1,6 @@ import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; +import { Cursor } from "../src/readers/core.js"; import { type IotRow, decodeIotColumnar, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/iot.wasm-headroom.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/iot.wasm-headroom.bench.ts similarity index 98% rename from skills/clickhouse-js-node-rowbinary-parser/tests/iot.wasm-headroom.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/iot.wasm-headroom.bench.ts index 0a6eb7292..f1d66c375 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/iot.wasm-headroom.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/iot.wasm-headroom.bench.ts @@ -1,6 +1,6 @@ import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; +import { Cursor } from "../src/readers/core.js"; import { type IotRow, readIotRowFast } from "../src/examples/iot.js"; /** diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/ledger.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/ledger.bench.ts similarity index 98% rename from skills/clickhouse-js-node-rowbinary-parser/tests/ledger.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/ledger.bench.ts index b2bffc00f..84c725e8e 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/ledger.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/ledger.bench.ts @@ -1,7 +1,7 @@ import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { type Reader, Cursor } from "../src/core.js"; -import { type DecimalValue, formatDecimal } from "../src/decimals.js"; +import { type Reader, Cursor } from "../src/readers/core.js"; +import { type DecimalValue, formatDecimal } from "../src/readers/decimals.js"; import { type LedgerRow, readLedgerRow, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/logs.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/logs.bench.ts similarity index 98% rename from skills/clickhouse-js-node-rowbinary-parser/tests/logs.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/logs.bench.ts index 89c520524..267b955c3 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/logs.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/logs.bench.ts @@ -1,7 +1,7 @@ import { gzipSync, zstdCompressSync } from "node:zlib"; import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { type Reader, Cursor } from "../src/core.js"; +import { type Reader, Cursor } from "../src/readers/core.js"; import { type LogRow, readLogRow, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/lowCardinality.test.ts b/skills/clickhouse-js-node-rowbinary/tests/lowCardinality.test.ts similarity index 88% rename from skills/clickhouse-js-node-rowbinary-parser/tests/lowCardinality.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/lowCardinality.test.ts index dbc39d66b..fd8dd2219 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/lowCardinality.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/lowCardinality.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readNullable } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readLowCardinality } from "../src/lowCardinality.js"; -import { readString } from "../src/strings.js"; +import { readNullable } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readLowCardinality } from "../src/readers/lowCardinality.js"; +import { readString } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/nested.test.ts b/skills/clickhouse-js-node-rowbinary/tests/nested.test.ts similarity index 86% rename from skills/clickhouse-js-node-rowbinary-parser/tests/nested.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/nested.test.ts index 7723b5f42..135804b36 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/nested.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/nested.test.ts @@ -1,10 +1,10 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readArray, readTupleNamed } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt8 } from "../src/integers.js"; -import { readNested } from "../src/nested.js"; -import { readString } from "../src/strings.js"; +import { readArray, readTupleNamed } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt8 } from "../src/readers/integers.js"; +import { readNested } from "../src/readers/nested.js"; +import { readString } from "../src/readers/strings.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/nothing.test.ts b/skills/clickhouse-js-node-rowbinary/tests/nothing.test.ts similarity index 94% rename from skills/clickhouse-js-node-rowbinary-parser/tests/nothing.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/nothing.test.ts index 9e26c4414..597945e0a 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/nothing.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/nothing.test.ts @@ -1,8 +1,8 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readArray, readNullable } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readNothing } from "../src/nothing.js"; +import { readArray, readNullable } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readNothing } from "../src/readers/nothing.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/observability.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/observability.bench.ts similarity index 97% rename from skills/clickhouse-js-node-rowbinary-parser/tests/observability.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/observability.bench.ts index 631b7f969..e1b76499b 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/observability.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/observability.bench.ts @@ -1,6 +1,6 @@ import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { type Reader, Cursor } from "../src/core.js"; +import { type Reader, Cursor } from "../src/readers/core.js"; import { type ObsRow, readObsRow, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/observability.example.test.ts b/skills/clickhouse-js-node-rowbinary/tests/observability.example.test.ts similarity index 97% rename from skills/clickhouse-js-node-rowbinary-parser/tests/observability.example.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/observability.example.test.ts index 22e4c4570..3d2b3e5e9 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/observability.example.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/observability.example.test.ts @@ -1,12 +1,12 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; +import { Cursor } from "../src/readers/core.js"; import { type ObsRow, readObsRow, readObsRowFast, } from "../src/examples/observability.js"; -import { readRows } from "../src/rows.js"; +import { readRows } from "../src/readers/rows.js"; /** * The gotcha-heavy example end to end: a single SELECT (no table needed) builds diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/orders.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/orders.bench.ts similarity index 96% rename from skills/clickhouse-js-node-rowbinary-parser/tests/orders.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/orders.bench.ts index 3a8ffc8cf..5f289c4c3 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/orders.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/orders.bench.ts @@ -1,6 +1,6 @@ import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { type Reader, Cursor } from "../src/core.js"; +import { type Reader, Cursor } from "../src/readers/core.js"; import { type OrderRow, readOrderRow, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/orders.example.test.ts b/skills/clickhouse-js-node-rowbinary/tests/orders.example.test.ts similarity index 95% rename from skills/clickhouse-js-node-rowbinary-parser/tests/orders.example.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/orders.example.test.ts index 1e4bef328..d74fa7dc5 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/orders.example.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/orders.example.test.ts @@ -1,8 +1,8 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; +import { Cursor } from "../src/readers/core.js"; import { type OrderRow, readOrderRow } from "../src/examples/orders.js"; -import { readRows } from "../src/rows.js"; +import { readRows } from "../src/readers/rows.js"; /** * Runs the `orders` example end to end (UUID / Decimal / Enum). These types are diff --git a/skills/clickhouse-js-node-rowbinary/tests/parseIP.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/parseIP.bench.ts new file mode 100644 index 000000000..d0df1ec68 --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/tests/parseIP.bench.ts @@ -0,0 +1,43 @@ +import { bench, describe } from "vitest"; +import { parseIPv4, parseIPv6 } from "../src/writers/ip.js"; + +/** + * Benchmark: the string -> raw-bytes IP parsers. `parseIPv6` is the heavier one + * (zero-run expansion, optional embedded IPv4) and was singled out in review, so + * it gets the spread of forms; `parseIPv4` is included as the cheap baseline. + * + * Bench-independent of the writers: each case parses a static string, so it + * measures only parsing cost. An equivalence guard runs first — a faster wrong + * answer is worthless. + */ + +const V4 = "192.168.0.1"; +const V6_FULL = "2001:0db8:0000:0000:0000:ff00:0042:8329"; +const V6_COMPRESSED = "2001:db8::ff00:42:8329"; +const V6_MAPPED = "::ffff:1.2.3.4"; + +// Equivalence guards: compressed and full forms must parse to the same bytes. +if (parseIPv4(V4) !== 0xc0a80001) { + throw new Error("parseIPv4 sanity check failed"); +} +if (!parseIPv6(V6_FULL).equals(parseIPv6(V6_COMPRESSED))) { + throw new Error("parseIPv6 full/compressed mismatch"); +} + +describe("parseIPv4", () => { + bench("dotted quad", () => { + parseIPv4(V4); + }); +}); + +describe("parseIPv6", () => { + bench("fully expanded", () => { + parseIPv6(V6_FULL); + }); + bench("zero-run compressed", () => { + parseIPv6(V6_COMPRESSED); + }); + bench("IPv4-mapped", () => { + parseIPv6(V6_MAPPED); + }); +}); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/profiles.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/profiles.bench.ts similarity index 96% rename from skills/clickhouse-js-node-rowbinary-parser/tests/profiles.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/profiles.bench.ts index 7276bc15f..045ee34de 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/profiles.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/profiles.bench.ts @@ -1,6 +1,6 @@ import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { type Reader, Cursor } from "../src/core.js"; +import { type Reader, Cursor } from "../src/readers/core.js"; import { type ProfileRow, readProfileRow, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/profiles.example.test.ts b/skills/clickhouse-js-node-rowbinary/tests/profiles.example.test.ts similarity index 93% rename from skills/clickhouse-js-node-rowbinary-parser/tests/profiles.example.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/profiles.example.test.ts index c42a54d15..a1b7a0d7b 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/profiles.example.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/profiles.example.test.ts @@ -1,8 +1,8 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; +import { Cursor } from "../src/readers/core.js"; import { type ProfileRow, readProfileRow } from "../src/examples/profiles.js"; -import { readRows } from "../src/rows.js"; +import { readRows } from "../src/readers/rows.js"; /** * Runs the `profiles` example end to end (Array + Nullable). Populated via diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/qbit.test.ts b/skills/clickhouse-js-node-rowbinary/tests/qbit.test.ts similarity index 91% rename from skills/clickhouse-js-node-rowbinary-parser/tests/qbit.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/qbit.test.ts index d5637c7ba..e02913f4b 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/qbit.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/qbit.test.ts @@ -1,9 +1,13 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readQBit } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readBFloat16, readFloat32, readFloat64 } from "../src/floats.js"; -import { readUInt8 } from "../src/integers.js"; +import { readQBit } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { + readBFloat16, + readFloat32, + readFloat64, +} from "../src/readers/floats.js"; +import { readUInt8 } from "../src/readers/integers.js"; // QBit is experimental; the type needs allow_experimental_qbit_type. async function reader(expr: string): Promise { diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/readHeader.test.ts b/skills/clickhouse-js-node-rowbinary/tests/readHeader.test.ts similarity index 87% rename from skills/clickhouse-js-node-rowbinary-parser/tests/readHeader.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/readHeader.test.ts index ab5a171d5..35e6e9939 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/readHeader.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/readHeader.test.ts @@ -1,7 +1,7 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; -import { readHeader } from "../src/header.js"; +import { Cursor } from "../src/readers/core.js"; +import { readHeader } from "../src/readers/header.js"; /** Fetch a `RowBinaryWithNamesAndTypes` response (header + rows) as a cursor. */ async function withNamesAndTypes(select: string): Promise { diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/readUUID.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/readUUID.bench.ts similarity index 94% rename from skills/clickhouse-js-node-rowbinary-parser/tests/readUUID.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/readUUID.bench.ts index 5f894b5ca..0075667f8 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/readUUID.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/readUUID.bench.ts @@ -1,5 +1,5 @@ import { bench, describe } from "vitest"; -import { formatUUID, formatUUIDTable } from "../src/reader.js"; +import { formatUUID, formatUUIDTable } from "../src/readers/reader.js"; /** * Benchmark: the BigInt-based formatUUID vs the lookup-table formatUUIDTable diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/rowBinaryWithNamesAndTypes.order.test.ts b/skills/clickhouse-js-node-rowbinary/tests/rowBinaryWithNamesAndTypes.order.test.ts similarity index 94% rename from skills/clickhouse-js-node-rowbinary-parser/tests/rowBinaryWithNamesAndTypes.order.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/rowBinaryWithNamesAndTypes.order.test.ts index 748cc6222..0deb9ab0b 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/rowBinaryWithNamesAndTypes.order.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/rowBinaryWithNamesAndTypes.order.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from "vitest"; -import { Cursor } from "../src/core.js"; -import { compileRowBinaryWithNamesAndTypes } from "../src/rowBinaryWithNamesAndTypes.js"; +import { Cursor } from "../src/readers/core.js"; +import { compileRowBinaryWithNamesAndTypes } from "../src/readers/rowBinaryWithNamesAndTypes.js"; // Offline regression tests (no server) for the wire-order guarantee of the // row reader: every row must read EXACTLY one reader per header column, in diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/rowBinaryWithNamesAndTypes.test.ts b/skills/clickhouse-js-node-rowbinary/tests/rowBinaryWithNamesAndTypes.test.ts similarity index 98% rename from skills/clickhouse-js-node-rowbinary-parser/tests/rowBinaryWithNamesAndTypes.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/rowBinaryWithNamesAndTypes.test.ts index a745d5466..28c3f1aac 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/rowBinaryWithNamesAndTypes.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/rowBinaryWithNamesAndTypes.test.ts @@ -1,13 +1,13 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; +import { Cursor } from "../src/readers/core.js"; import { compileRowBinaryWithNamesAndTypes, createTypeReaderCache, typeStringToReader, type Row, -} from "../src/rowBinaryWithNamesAndTypes.js"; -import { RowBinaryTypeError } from "../src/compile.js"; +} from "../src/readers/rowBinaryWithNamesAndTypes.js"; +import { RowBinaryTypeError } from "../src/readers/compile.js"; /** Raw value bytes for one expression (`FORMAT RowBinary`, no header). */ async function rowBinary(expr: string): Promise { @@ -248,16 +248,16 @@ describe("decimals (width chosen by precision)", () => { }); }); -describe("enums (value is the underlying int)", () => { +describe("enums (resolve to the value's name)", () => { it("Enum8", async () => { expect( await value("SELECT CAST('b' AS Enum8('a' = 1, 'b' = 2)) AS v"), - ).toEqual(2); + ).toEqual("b"); }); it("Enum16", async () => { expect( await value("SELECT CAST('y' AS Enum16('x' = -1, 'y' = 100)) AS v"), - ).toEqual(100); + ).toEqual("y"); }); }); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/rows.test.ts b/skills/clickhouse-js-node-rowbinary/tests/rows.test.ts similarity index 97% rename from skills/clickhouse-js-node-rowbinary-parser/tests/rows.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/rows.test.ts index ca45f3bca..139b6ce4c 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/rows.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/rows.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; -import { readInt32, readUInt64 } from "../src/integers.js"; -import { readRows } from "../src/rows.js"; -import { readString } from "../src/strings.js"; +import { Cursor } from "../src/readers/core.js"; +import { readInt32, readUInt64 } from "../src/readers/integers.js"; +import { readRows } from "../src/readers/rows.js"; +import { readString } from "../src/readers/strings.js"; /** * Multi-row tests: plain `RowBinary` concatenates rows back-to-back with no row diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/simpleAggregateFunction.test.ts b/skills/clickhouse-js-node-rowbinary/tests/simpleAggregateFunction.test.ts similarity index 88% rename from skills/clickhouse-js-node-rowbinary-parser/tests/simpleAggregateFunction.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/simpleAggregateFunction.test.ts index 817ef55e3..adb971bcb 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/simpleAggregateFunction.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/simpleAggregateFunction.test.ts @@ -1,9 +1,9 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { readArray } from "../src/composite.js"; -import { NeedMoreData, Cursor } from "../src/core.js"; -import { readUInt64, readUInt8 } from "../src/integers.js"; -import { readSimpleAggregateFunction } from "../src/simpleAggregateFunction.js"; +import { readArray } from "../src/readers/composite.js"; +import { NeedMoreData, Cursor } from "../src/readers/core.js"; +import { readUInt64, readUInt8 } from "../src/readers/integers.js"; +import { readSimpleAggregateFunction } from "../src/readers/simpleAggregateFunction.js"; async function reader(expr: string): Promise { return new Cursor(await query(`SELECT ${expr} FORMAT RowBinary`)); diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/streamRowBatches.test.ts b/skills/clickhouse-js-node-rowbinary/tests/streamRowBatches.test.ts similarity index 95% rename from skills/clickhouse-js-node-rowbinary-parser/tests/streamRowBatches.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/streamRowBatches.test.ts index 0239bfc4e..e057e85a1 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/streamRowBatches.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/streamRowBatches.test.ts @@ -1,9 +1,12 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; -import { readUInt64 } from "../src/integers.js"; -import { type SmallChunkStats, streamRowBatches } from "../src/stream.js"; -import { readString } from "../src/strings.js"; +import { Cursor } from "../src/readers/core.js"; +import { readUInt64 } from "../src/readers/integers.js"; +import { + type SmallChunkStats, + streamRowBatches, +} from "../src/readers/stream.js"; +import { readString } from "../src/readers/strings.js"; /** * `streamRowBatches` is the async front door over `readRows`: an async iterable diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/streamingRow.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/streamingRow.bench.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/tests/streamingRow.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/streamingRow.bench.ts diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/telemetry.bench.ts b/skills/clickhouse-js-node-rowbinary/tests/telemetry.bench.ts similarity index 96% rename from skills/clickhouse-js-node-rowbinary-parser/tests/telemetry.bench.ts rename to skills/clickhouse-js-node-rowbinary/tests/telemetry.bench.ts index 499d23272..314f88598 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/telemetry.bench.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/telemetry.bench.ts @@ -1,6 +1,6 @@ import { bench, describe } from "vitest"; import { query } from "./clickhouse.js"; -import { type Reader, Cursor } from "../src/core.js"; +import { type Reader, Cursor } from "../src/readers/core.js"; import { type TelemetryRow, readTelemetryRow, diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/telemetry.example.test.ts b/skills/clickhouse-js-node-rowbinary/tests/telemetry.example.test.ts similarity index 95% rename from skills/clickhouse-js-node-rowbinary-parser/tests/telemetry.example.test.ts rename to skills/clickhouse-js-node-rowbinary/tests/telemetry.example.test.ts index 9e5ab1fff..1d6f701e6 100644 --- a/skills/clickhouse-js-node-rowbinary-parser/tests/telemetry.example.test.ts +++ b/skills/clickhouse-js-node-rowbinary/tests/telemetry.example.test.ts @@ -1,11 +1,11 @@ import { describe, expect, it } from "vitest"; import { query } from "./clickhouse.js"; -import { Cursor } from "../src/core.js"; +import { Cursor } from "../src/readers/core.js"; import { type TelemetryRow, readTelemetryRow, } from "../src/examples/telemetry.js"; -import { readRows } from "../src/rows.js"; +import { readRows } from "../src/readers/rows.js"; /** * Runs the `telemetry` example end to end (Map / Array / Nullable / named diff --git a/skills/clickhouse-js-node-rowbinary-parser/tests/wasm-int128.experiment.mjs b/skills/clickhouse-js-node-rowbinary/tests/wasm-int128.experiment.mjs similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/tests/wasm-int128.experiment.mjs rename to skills/clickhouse-js-node-rowbinary/tests/wasm-int128.experiment.mjs diff --git a/skills/clickhouse-js-node-rowbinary-parser/tsconfig.build.json b/skills/clickhouse-js-node-rowbinary/tsconfig.build.json similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/tsconfig.build.json rename to skills/clickhouse-js-node-rowbinary/tsconfig.build.json diff --git a/skills/clickhouse-js-node-rowbinary-parser/tsconfig.json b/skills/clickhouse-js-node-rowbinary/tsconfig.json similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/tsconfig.json rename to skills/clickhouse-js-node-rowbinary/tsconfig.json diff --git a/skills/clickhouse-js-node-rowbinary-parser/vitest.config.ts b/skills/clickhouse-js-node-rowbinary/vitest.config.ts similarity index 100% rename from skills/clickhouse-js-node-rowbinary-parser/vitest.config.ts rename to skills/clickhouse-js-node-rowbinary/vitest.config.ts diff --git a/skills/clickhouse-js-node-rowbinary/writer.md b/skills/clickhouse-js-node-rowbinary/writer.md new file mode 100644 index 000000000..f3e3be75c --- /dev/null +++ b/skills/clickhouse-js-node-rowbinary/writer.md @@ -0,0 +1,96 @@ +# RowBinary writer (encode) for Node.js + +Encoding JS values into a `RowBinary` payload to send to ClickHouse. Read +[SKILL.md](SKILL.md) first for the format gate ("is RowBinary even the right +format?") and the principles that apply to **both** directions; this file covers +what's specific to **writing**. Reading? See [reader.md](reader.md). + +Each `writeX` encodes one value, appending its RowBinary bytes to a `Sink` (a +caller-supplied byte buffer plus the current write offset — state only, no write +methods). Leaf writers (`writeUInt8`, `writeString`, …) encode directly; +combinators (`writeArray`, `writeTuple`, …) take sub-writers and return a writer, +so composite types compose with no per-element closures. For the `Sink`/`Writer` +types and how to drain the encoded bytes, see `src/writers/core.ts`. Import the +barrel as `@clickhouse/rowbinary/writer`, or a per-type module for just what you +need. (Structurally this is the mirror of the decode side in +[reader.md](reader.md), but you don't need the read side to write.) + +## Writer guidance + +On top of the shared principles in [SKILL.md](SKILL.md), the write path has its own: + +- **Reserve before you write.** Every fixed-width write goes through `reserve()`, + which bounds-checks against the fixed-length buffer and throws the `BufferFull` + sentinel when the chunk is full — your cue to flush what's written and continue + into a fresh buffer. Exact signature, return value, and throw contract are in + `src/writers/core.ts`. + +- **Coalesce `reserve()` across a run of adjacent fixed-width columns.** Their + combined size is statically known, so reserve ONCE for the whole run and write + each value at a constant offset off the returned base — one bounds-check instead + of one per column. Only applies where every column in the run is fixed-width (a + variable-width writer like `writeString` reserves on its own). + +- **Hoist sink state into locals in the generated writer.** `Sink.buf`/`Sink.view` + are `readonly`, so bind them to locals once at the top and address them directly + instead of through `sink.` on every write. Keep the write position (`sink.pos`) + in a local too — but sync it back to `sink.pos` before any `reserve()` or + `BufferFull` throw, since those read and mutate it to decide capacity and where a + flushed buffer resumes. + +- **Stream the whole result with `writeRows`, not a one-shot writer.** When you + need to encode a large or unbounded row source, reach for `writeRows` rather than + a `Writer`: it owns a fixed buffer and yields it as a generator, + streaming the result out chunk by chunk instead of demanding it all fit at once. + It never leaks a half-written row (it rewinds to the last whole-row boundary + before flushing) and never fails on a single big row (it grows the buffer to fit). + It also publishes a per-flush diagnostics-channel event for buffer-utilization + metrics. Signature, default buffer size, channel name and payload type, the + growth/rewind details, and a usage example are all in `src/writers/rows.ts`. + +- **No defensive validation on the hot path.** Don't add `isFinite`/`NaN`/range + checks to `writeX`; document the precondition in JSDoc instead. Two narrow + exceptions — framing-keeping checks and zero-cost parse-time helpers — are + spelled out in [AGENTS.md](AGENTS.md); `src/writers/ip.ts` is the worked + example (`writeIPv6`, `parseIPv6`). + +- **Lossy time conversions floor, never round.** Every date/time writer in + `src/writers/datetime.ts` drops the sub-unit it can't encode by flooring toward + −∞, so a caller's value is never silently shifted _up_ to the wrong + day/second/tick and pre-1970 instants stay correct (not rounded toward the + epoch). See its JSDoc for the per-function specifics. + +## Writer type family references + +The writers live as real code under `src/writers/`, one file per type family +(same basenames as the readers, under the `writers/` directory). + +| Value to encode (trigger) | Open | +| --------------------------------------------------------------------------------------- | ---------------------------------------- | +| **Always** — sink state, `reserve()`, `BufferFull`, `Writer` | `src/writers/core.ts` | +| LEB128 length/count prefixes for `String`/`Array`/`Map` (`writeUVarint`) | `src/writers/varint.ts` | +| `Int8`–`Int256`, `UInt8`–`UInt256` | `src/writers/integers.ts` | +| `Bool` (`writeBool`) | `src/writers/bool.ts` | +| `Enum8`, `Enum16` (`writeEnum8`/`writeEnum16`; pass the raw int) | `src/writers/enums.ts` | +| `Float32`, `Float64`, `BFloat16` | `src/writers/floats.ts` | +| `Decimal32/64/128/256`, `Decimal(P, S)` (`parseDecimal`) | `src/writers/decimals.ts` | +| `String`, `FixedString(N)` (`writeString`/`writeStringBytes`/`writeFixedString`) | `src/writers/strings.ts` | +| `UUID` (`writeUUID`, `parseUUID`) | `src/writers/uuid.ts` | +| `IPv4`, `IPv6` (`writeIPv4`/`writeIPv6`, `parseIPv4`/`parseIPv6`) | `src/writers/ip.ts` | +| `Date`, `Date32`, `DateTime`, `DateTime(tz)`, `DateTime64(P[, tz])` | `src/writers/datetime.ts` | +| `Time`, `Time64(P)` (`parseTime`/`parseTime64`) | `src/writers/time.ts` | +| `IntervalNanosecond` … `IntervalYear` | `src/writers/interval.ts` | +| `Array(T)`, `Map(K, V)`, `Tuple(...)`, `Nullable(T)`, `Variant(...)`, `QBit(...)` | `src/writers/composite.ts` | +| `Point`, `Ring`, `LineString`, `MultiLineString`, `Polygon`, `MultiPolygon`, `Geometry` | `src/writers/geo.ts` | +| The whole result — write rows from a value source (`writeRows`) | `src/writers/rows.ts` | +| `LowCardinality(T)` — transparent, encode as `T` | `src/writers/lowCardinality.ts` | +| `SimpleAggregateFunction(f, T)` — transparent, encode as `T` | `src/writers/simpleAggregateFunction.ts` | +| `Nested(...)` — no wire of its own; `Array(Tuple(...))` | `src/writers/nested.ts` | +| `Nothing` — zero-width, never encoded (only wrapped) | `src/writers/nothing.ts` | +| `AggregateFunction(...)` — opaque state; produce server-side | `src/writers/aggregateFunction.ts` | + +**No writer counterpart yet** — these reader paths are decode-only for now: +`dynamic.ts`, `json.ts`, `stream.ts`, the `RowBinaryWithNamesAndTypes` +header/compile/runtime path (`header.ts`, `compile.ts`, +`rowBinaryWithNamesAndTypes.ts`), and the columnar typed-array path +(`columnar.ts`). The AST-based dynamic encode path is intentionally not built. diff --git a/tests/clickhouse-test-runner/AGENTS.md b/tests/clickhouse-test-runner/AGENTS.md new file mode 100644 index 000000000..73d97eb0e --- /dev/null +++ b/tests/clickhouse-test-runner/AGENTS.md @@ -0,0 +1,25 @@ +# Recommendations for AI agents — upstream SQL test harness + +Guidance for the [`clickhouse-test-runner`](.) harness. See the [repo-root `AGENTS.md`](../../AGENTS.md) for cross-cutting guidance. + +This harness is a Node.js port of `clickhouse-client` that allows the official ClickHouse Python test runner (`tests/clickhouse-test`) to drive a subset of the upstream SQL test suite against `@clickhouse/client`. + +## What the harness does + +- Wraps `@clickhouse/client` in a tiny CLI (`bin/clickhouse` → `dist/main.js`) that mimics enough of the upstream `clickhouse-client` binary (same flags, `extract-from-config` shortcut, stdin/`--query` behavior) for the Python `tests/clickhouse-test` runner to drive it without modification. +- The runner is an npm workspace of the root `clickhouse-js` package, so `npm install` from the repo root links `@clickhouse/client` and `@clickhouse/client-common` from the local checkout instead of resolving them from the npm registry. Always install + build from the repo root (`npm install && npm run build`) so the harness exercises the code under review rather than the last published client. +- The CI matrix runs the harness against ClickHouse `latest` and `head` so that we exercise `@clickhouse/client` against both server versions and detect server regressions. The allowlist is also split into round-robin shards (`SHARD_INDEX` / `SHARD_TOTAL`) so each matrix job stays at roughly one minute; bump both the `shard` matrix values and the `SHARD_TOTAL` env value in the workflow together if per-shard runtime climbs back above ~1 minute. +- Reads the curated test list from [`upstream-allowlist.txt`](upstream-allowlist.txt) (one test name per line, `#` for comments) and forwards them as positional arguments to `tests/clickhouse-test`. +- The `SERVER_SETTINGS`/`CLIENT_ONLY_SETTINGS` allowlists in [`src/settings.ts`](src/settings.ts) are copied from the Java port and may need periodic resync as ClickHouse adds or reclassifies settings. + +See [`README.md`](README.md) for build, usage, and environment-variable documentation. When harness behavior changes (new wrapper flags, new short-circuited keys in `bin/clickhouse`, new entries in the settings allowlists), review the README and [`.github/workflows/upstream-sql-tests.yml`](../../.github/workflows/upstream-sql-tests.yml) to keep them in sync with the implementation. + +## Strategy for growing the allowlist + +The allowlist is grown in **batches of ~100 candidate tests at a time**, in upstream filename order, following this loop: + +1. **Pre-filter the candidate batch.** Skip non-SQL tests (`.sh`, `.py`, `.j2`) and tests tagged for unsupported infrastructure (`shard`, `distributed`, `replicated`, `zookeeper`, `kafka`, `s3`, `mysql`, `tls`, etc.). These will never pass through this harness as it stands today. +2. **Run each candidate through the harness** with `--no-stateful --no-long`. **Only keep tests that report `[ OK ]`**; drop failures and skips. +3. **Validate against the CI matrix before committing**, not just one local server version. The CI workflow runs `{ClickHouse latest, head} × {shard 1..N}` — a test that passes locally on `head` may fail on `latest` (or vice versa) and break CI. +4. **Beware substring/prefix expansion.** `tests/clickhouse-test` treats positional arguments as **substring/prefix matches** rather than exact names, so an allowlist entry like `00396_uuid` will silently pull in `00396_uuid_v7`, `00712_prewhere_with_alias` will pull in `00712_prewhere_with_alias_bug_2`, etc. When adding an entry whose name is a prefix of any other test in `0_stateless`, prefer the longest unambiguous form, or accept that the siblings come along and verify they all pass. +5. **Prune flakes promptly.** If a previously-passing test starts to flake on the nightly run, remove it (or its prefix-expanded siblings) from the allowlist rather than retrying — the allowlist exists to be a stable green signal, not a TODO list. diff --git a/tests/clickhouse-test-runner/README.md b/tests/clickhouse-test-runner/README.md index 67340d0fa..8c5f99183 100644 --- a/tests/clickhouse-test-runner/README.md +++ b/tests/clickhouse-test-runner/README.md @@ -15,6 +15,44 @@ client. It lets us see exactly which upstream SQL tests pass or fail when run through `@clickhouse/client`, without having to reimplement the test runner itself. +## Backends + +The runner can execute queries two ways, selected by the `TEST_RUNNER_BACKEND` +environment variable: + +- **`passthrough`** (default) — set `default_format = TabSeparated` and stream + ClickHouse's own output bytes straight to stdout. ClickHouse does all the + formatting; this exercises `@clickhouse/client`'s transport, session and + settings handling. Comparison is the upstream `.reference` diff. + +- **`rowbinary`** — for each result-returning statement that has no explicit + `FORMAT` clause, request `RowBinaryWithNamesAndTypes`, decode it with the + published [`@clickhouse/rowbinary`](https://www.npmjs.com/package/@clickhouse/rowbinary) + package's dynamic header→reader path, and re-render the rows as `TabSeparated` so the + same `.reference` diff still applies. This makes the upstream SQL suite a + breadth test of the RowBinary parser: ClickHouse is the byte oracle, the + `.reference` is the value oracle. DDL / `INSERT` / `SET` / explicit-`FORMAT` + statements fall through to passthrough. A decode or render error fails the + statement (it is never silently swallowed), so a test only "passes" if the + parser actually reproduced the server's output. + +Because the `rowbinary` backend can only validate tests whose decoded types it +can render back to `TabSeparated`, it runs a dedicated, smaller allowlist +(`rowbinary-allowlist.txt`) rather than the full `upstream-allowlist.txt`. Point +`UPSTREAM_TEST_LIST` at it: + +```bash +UPSTREAM_CLICKHOUSE_DIR=/path/to/ClickHouse \ + TEST_RUNNER_BACKEND=rowbinary \ + UPSTREAM_TEST_LIST=tests/clickhouse-test-runner/rowbinary-allowlist.txt \ + tests/clickhouse-test-runner/scripts/run-upstream-tests.sh --no-stateful +``` + +The `rowbinary` backend depends on the published `@clickhouse/rowbinary` package +(installed by the normal `npm install`), so it validates the same parser build +that ships to users. The skill's in-repo source is covered separately by its own +suite (`.github/workflows/tests-skill-rowbinary.yml`). + ## Build This package is a workspace of the root `clickhouse-js` repository, so it @@ -59,13 +97,14 @@ export PATH="/path/to/clickhouse-js/tests/clickhouse-test-runner/bin:$PATH" ## Environment variables -| Variable | Default | Description | -| --------------------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | -| `CLICKHOUSE_CLIENT_CLI_LOG` | `tests/clickhouse-test-runner/.upstream/clickhouse-client-cli.log` | Path to a log file used to record every shim invocation. Useful for troubleshooting. | -| `UPSTREAM_CLICKHOUSE_DIR` | `tests/clickhouse-test-runner/.upstream/ClickHouse` | Path to a checkout of `ClickHouse/ClickHouse` containing the upstream test suite. | -| `UPSTREAM_TEST_LIST` | `tests/clickhouse-test-runner/upstream-allowlist.txt` | Path to a file listing the upstream tests to run (one test name per line, `#` for comments). | -| `SHARD_INDEX` | `1` | 1-based index of the shard to run when sharding the allowlist (must be `<= SHARD_TOTAL`). | -| `SHARD_TOTAL` | `1` | Total number of shards. When `> 1`, only tests at positions where `i % SHARD_TOTAL == SHARD_INDEX - 1` are run (round-robin selection). | +| Variable | Default | Description | +| --------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `TEST_RUNNER_BACKEND` | `passthrough` | Which backend executes queries: `passthrough` (stream ClickHouse's own `TabSeparated`) or `rowbinary` (decode `RowBinaryWithNamesAndTypes` via `@clickhouse/rowbinary` and re-render). See [Backends](#backends). | +| `CLICKHOUSE_CLIENT_CLI_LOG` | `tests/clickhouse-test-runner/.upstream/clickhouse-client-cli.log` | Path to a log file used to record every shim invocation. Useful for troubleshooting. | +| `UPSTREAM_CLICKHOUSE_DIR` | `tests/clickhouse-test-runner/.upstream/ClickHouse` | Path to a checkout of `ClickHouse/ClickHouse` containing the upstream test suite. | +| `UPSTREAM_TEST_LIST` | `tests/clickhouse-test-runner/upstream-allowlist.txt` | Path to a file listing the upstream tests to run (one test name per line, `#` for comments). | +| `SHARD_INDEX` | `1` | 1-based index of the shard to run when sharding the allowlist (must be `<= SHARD_TOTAL`). | +| `SHARD_TOTAL` | `1` | Total number of shards. When `> 1`, only tests at positions where `i % SHARD_TOTAL == SHARD_INDEX - 1` are run (round-robin selection). | ## Running against the upstream test suite diff --git a/tests/clickhouse-test-runner/__tests__/should-decode.test.ts b/tests/clickhouse-test-runner/__tests__/should-decode.test.ts new file mode 100644 index 000000000..c0e5eb50c --- /dev/null +++ b/tests/clickhouse-test-runner/__tests__/should-decode.test.ts @@ -0,0 +1,47 @@ +import { describe, expect, it } from "vitest"; +import { shouldDecode } from "../src/backends/rowbinary.js"; + +describe("shouldDecode", () => { + it("routes result-returning statements to the decode path", () => { + expect(shouldDecode("SELECT 1")).toBe(true); + expect(shouldDecode("select 1")).toBe(true); + expect(shouldDecode("WITH 1 AS x SELECT x")).toBe(true); + expect(shouldDecode("SHOW TABLES")).toBe(true); + expect(shouldDecode("DESCRIBE TABLE system.one")).toBe(true); + expect(shouldDecode("EXISTS TABLE system.one")).toBe(true); + expect(shouldDecode("EXPLAIN SELECT 1")).toBe(true); + expect(shouldDecode("(SELECT 1) UNION ALL (SELECT 2)")).toBe(true); + }); + + it("does not decode statements with an explicit FORMAT clause", () => { + expect(shouldDecode("SELECT 1 FORMAT JSON")).toBe(false); + expect(shouldDecode("SELECT 1 FORMAT TabSeparated")).toBe(false); + expect(shouldDecode("SELECT 1 SETTINGS max_threads=1 FORMAT Pretty")).toBe( + false, + ); + }); + + it("does not decode DDL / INSERT / SET / other non-result statements", () => { + expect(shouldDecode("CREATE TABLE t (a UInt8) ENGINE = Memory")).toBe( + false, + ); + expect(shouldDecode("INSERT INTO t VALUES (1)")).toBe(false); + expect(shouldDecode("INSERT INTO t FORMAT Values (1)")).toBe(false); + expect(shouldDecode("DROP TABLE IF EXISTS t")).toBe(false); + expect(shouldDecode("SET max_threads = 1")).toBe(false); + expect(shouldDecode("ALTER TABLE t ADD COLUMN b UInt8")).toBe(false); + }); + + it("does not confuse the FORMAT keyword with formatXxx functions", () => { + expect(shouldDecode("SELECT formatDateTime(now(), '%Y')")).toBe(true); + expect(shouldDecode("SELECT formatReadableSize(1024)")).toBe(true); + }); + + it("skips leading comments and whitespace when finding the keyword", () => { + expect(shouldDecode("-- a comment\nSELECT 1")).toBe(true); + expect(shouldDecode("/* block */ SELECT 1")).toBe(true); + expect( + shouldDecode(" \n CREATE TABLE t (a UInt8) ENGINE = Memory"), + ).toBe(false); + }); +}); diff --git a/tests/clickhouse-test-runner/__tests__/test-hint.test.ts b/tests/clickhouse-test-runner/__tests__/test-hint.test.ts new file mode 100644 index 000000000..b6a888acd --- /dev/null +++ b/tests/clickhouse-test-runner/__tests__/test-hint.test.ts @@ -0,0 +1,142 @@ +import { describe, expect, it } from "vitest"; +import { + buildStatements, + errorMatchesExpectation, + parseHintComment, +} from "../src/test-hint.js"; +import { splitQueries } from "../src/split-queries.js"; + +const statements = (sql: string) => buildStatements(splitQueries(sql)); + +describe("parseHintComment", () => { + it("parses a named serverError hint", () => { + const hint = parseHintComment( + "-- { serverError SIZES_OF_ARRAYS_DONT_MATCH }", + ); + expect(hint).not.toBeNull(); + expect([...hint!.names]).toEqual(["SIZES_OF_ARRAYS_DONT_MATCH"]); + expect([...hint!.codes]).toEqual([]); + expect(hint!.label).toBe("serverError SIZES_OF_ARRAYS_DONT_MATCH"); + }); + + it("parses a numeric serverError hint", () => { + const hint = parseHintComment("-- { serverError 190 }"); + expect([...hint!.codes]).toEqual(["190"]); + expect([...hint!.names]).toEqual([]); + }); + + it("parses a comma-separated list of codes", () => { + const hint = parseHintComment("-- { serverError 153, 6 }"); + expect([...hint!.codes].sort()).toEqual(["153", "6"]); + }); + + it("parses a clientError hint", () => { + const hint = parseHintComment("-- { clientError SYNTAX_ERROR }"); + expect([...hint!.names]).toEqual(["SYNTAX_ERROR"]); + }); + + it("supports block comments", () => { + const hint = parseHintComment("/* { serverError 241 } */"); + expect([...hint!.codes]).toEqual(["241"]); + }); + + it("returns null for non-error hints", () => { + expect(parseHintComment("-- { echoOn }")).toBeNull(); + expect(parseHintComment("-- a plain comment")).toBeNull(); + expect(parseHintComment("-- { unterminated")).toBeNull(); + }); +}); + +describe("buildStatements", () => { + it("leaves un-annotated statements without an expectation", () => { + expect(statements("SELECT 1; SELECT 2")).toEqual([ + { sql: "SELECT 1", expectedError: null }, + { sql: "SELECT 2", expectedError: null }, + ]); + }); + + it("attaches a trailing hint to the preceding statement", () => { + expect( + statements( + "SELECT throwIf(1); -- { serverError FUNCTION_THROW_IF_VALUE_IS_NON_ZERO }\nSELECT 2;", + ), + ).toEqual([ + { + sql: "SELECT throwIf(1)", + expectedError: { + label: "serverError FUNCTION_THROW_IF_VALUE_IS_NON_ZERO", + codes: new Set(), + names: new Set(["FUNCTION_THROW_IF_VALUE_IS_NON_ZERO"]), + }, + }, + { + sql: "-- { serverError FUNCTION_THROW_IF_VALUE_IS_NON_ZERO }\nSELECT 2", + expectedError: null, + }, + ]); + }); + + it("attaches a hint left dangling after the final statement", () => { + expect(statements("SELECT bad(); -- { serverError 42 }")).toEqual([ + { + sql: "SELECT bad()", + expectedError: { + label: "serverError 42", + codes: new Set(["42"]), + names: new Set(), + }, + }, + ]); + }); + + it("ignores a leading hint with no preceding statement", () => { + expect(statements("-- { serverError 42 }\nSELECT 1;")).toEqual([ + { sql: "-- { serverError 42 }\nSELECT 1", expectedError: null }, + ]); + }); + + it("attaches consecutive hints to their own statements", () => { + const result = statements( + "SELECT a; -- { serverError 1 }\nSELECT b; -- { serverError 2 }\nSELECT c;", + ); + expect(result.map((s) => s.sql.includes("SELECT a"))).toContain(true); + const a = result.find((s) => s.sql.endsWith("SELECT a"))!; + const b = result.find((s) => s.sql.includes("SELECT b"))!; + const c = result.find((s) => s.sql.includes("SELECT c"))!; + expect([...a.expectedError!.codes]).toEqual(["1"]); + expect([...b.expectedError!.codes]).toEqual(["2"]); + expect(c.expectedError).toBeNull(); + }); +}); + +describe("errorMatchesExpectation", () => { + const expected = parseHintComment( + "-- { serverError SIZES_OF_ARRAYS_DONT_MATCH }", + )!; + + it("matches by error type name", () => { + expect( + errorMatchesExpectation( + { code: "190", type: "SIZES_OF_ARRAYS_DONT_MATCH" }, + expected, + ), + ).toBe(true); + }); + + it("matches by numeric code", () => { + const byCode = parseHintComment("-- { serverError 190 }")!; + expect(errorMatchesExpectation({ code: "190" }, byCode)).toBe(true); + expect(errorMatchesExpectation({ code: 190 }, byCode)).toBe(true); + }); + + it("does not match a different error", () => { + expect( + errorMatchesExpectation({ code: "60", type: "UNKNOWN_TABLE" }, expected), + ).toBe(false); + }); + + it("does not match non-error values", () => { + expect(errorMatchesExpectation(null, expected)).toBe(false); + expect(errorMatchesExpectation("boom", expected)).toBe(false); + }); +}); diff --git a/tests/clickhouse-test-runner/__tests__/tsv-serialize.test.ts b/tests/clickhouse-test-runner/__tests__/tsv-serialize.test.ts new file mode 100644 index 000000000..185e3f004 --- /dev/null +++ b/tests/clickhouse-test-runner/__tests__/tsv-serialize.test.ts @@ -0,0 +1,208 @@ +import { describe, expect, it } from "vitest"; +import { parseDataType } from "@clickhouse/datatype-parser"; +import { formatUUID } from "@clickhouse/rowbinary/uuid"; +import { renderValue, compileRowRenderers } from "../src/tsv-serialize.js"; + +/** + * Render a single value at top level or nested, given a ClickHouse type string. + * The value shapes here mirror exactly what `@clickhouse/rowbinary` decodes — + * Decimal as `[bigint, scale]`, Date/DateTime as `Date`, UUID/IPv6 as `Buffer`, + * Map as a JS `Map`, etc. — so the test doubles as documentation of that model. + */ +function render(type: string, value: unknown, nested = false): string { + const ast = parseDataType(type).ast!; + return renderValue(ast, value, nested); +} + +describe("renderValue — top level (escaped, unquoted)", () => { + it("integers and bigints", () => { + expect(render("UInt8", 255)).toBe("255"); + expect(render("Int64", -9223372036854775808n)).toBe("-9223372036854775808"); + expect(render("UInt256", 7n)).toBe("7"); + }); + + it("floats incl. specials", () => { + expect(render("Float64", 1.5)).toBe("1.5"); + expect(render("Float64", 1)).toBe("1"); + expect(render("Float64", Infinity)).toBe("inf"); + expect(render("Float64", -Infinity)).toBe("-inf"); + expect(render("Float64", NaN)).toBe("nan"); + }); + + it("float64 signed zero and exponent style match ClickHouse", () => { + expect(render("Float64", -0)).toBe("-0"); + expect(render("Float64", 0)).toBe("0"); + // positive exponents drop the '+' (1e+21 -> 1e21); negatives keep the '-' + expect(render("Float64", 1e21)).toBe("1e21"); + expect(render("Float64", 1e-10)).toBe("1e-10"); + }); + + it("float32 renders the shortest single-precision round-trip, not the widened double", () => { + // Math.fround(0.26894) === the value ClickHouse decodes; String() of that + // double is 0.2689400017261505, but ClickHouse prints the shortest form. + expect(render("Float32", Math.fround(0.26894))).toBe("0.26894"); + expect(render("Float32", Math.fround(-0.76159))).toBe("-0.76159"); + expect(render("Float32", Math.fround(0.1))).toBe("0.1"); + expect(render("Float32", Math.fround(1 / 3))).toBe("0.33333334"); + expect(render("Float32", Math.fround(3.4028235e38))).toBe("3.4028235e38"); + expect(render("Float32", -0)).toBe("-0"); + expect(render("Float32", Infinity)).toBe("inf"); + expect(render("Float32", NaN)).toBe("nan"); + }); + + it("bool", () => { + expect(render("Bool", true)).toBe("true"); + expect(render("Bool", false)).toBe("false"); + }); + + it("strings escape backslash, tab, newline, CR, NUL", () => { + expect(render("String", "a\tb\nc\\d\r\0")).toBe("a\\tb\\nc\\\\d\\r\\0"); + }); + + it("decimal via [unscaled, scale]", () => { + expect(render("Decimal(10, 2)", [314n, 2])).toBe("3.14"); + expect(render("Decimal64(3)", [-1005n, 3])).toBe("-1.005"); + }); + + it("date and datetime from a UTC Date", () => { + expect(render("Date", new Date(Date.UTC(2020, 0, 2)))).toBe("2020-01-02"); + expect(render("DateTime", new Date(Date.UTC(2020, 0, 2, 3, 4, 5)))).toBe( + "2020-01-02 03:04:05", + ); + }); + + it("datetime64 fraction scaled to precision", () => { + // [whole-seconds Date, nanoseconds] + const d = new Date(Date.UTC(2020, 0, 2, 3, 4, 5)); + expect(render("DateTime64(3)", [d, 500_000_000])).toBe( + "2020-01-02 03:04:05.500", + ); + expect(render("DateTime64(6)", [d, 123_456_000])).toBe( + "2020-01-02 03:04:05.123456", + ); + // Sub-precision nanoseconds are TRUNCATED, not rounded (round would give .124). + expect(render("DateTime64(3)", [d, 123_999_999])).toBe( + "2020-01-02 03:04:05.123", + ); + }); + + it("uuid / ipv4 / ipv6 from their decoded byte/number forms", () => { + // readUUID hands back the raw 16 wire bytes; formatUUID applies ClickHouse's + // byte ordering, so assert renderValue delegates to it (quoting is checked + // in the nested suite) rather than re-deriving the layout here. + const uuid = Buffer.from("61f0c4045cb311e7907ba6006ad3dba0", "hex"); + expect(render("UUID", uuid)).toBe(formatUUID(uuid)); + expect(render("IPv4", (1 << 24) | (2 << 16) | (3 << 8) | 4)).toBe( + "1.2.3.4", + ); + const ipv6 = Buffer.alloc(16); + ipv6[15] = 1; + expect(render("IPv6", ipv6)).toBe("::1"); + }); + + it("enum maps the wire integer to its name", () => { + expect(render("Enum8('x' = 1, 'y' = 2)", 2)).toBe("y"); + expect(render("Enum16('a' = 10, 'b' = -20)", -20)).toBe("b"); + }); + + it("NULL is backslash-N at top level", () => { + expect(render("Nullable(Int32)", null)).toBe("\\N"); + expect(render("Nullable(Int32)", 5)).toBe("5"); + }); + + it("LowCardinality is transparent", () => { + expect(render("LowCardinality(String)", "hi")).toBe("hi"); + }); +}); + +describe("renderValue — nested (single-quoted)", () => { + it("stringish values gain quotes and escape the quote", () => { + expect(render("String", "b\tc", true)).toBe("'b\\tc'"); + expect(render("String", "it's", true)).toBe("'it\\'s'"); + }); + + it("numbers and decimals stay bare when nested", () => { + expect(render("Int32", -5, true)).toBe("-5"); + expect(render("Decimal(10, 2)", [314n, 2], true)).toBe("3.14"); + }); + + it("NULL is the word NULL when nested", () => { + expect(render("Nullable(Int32)", null, true)).toBe("NULL"); + }); +}); + +describe("renderValue — composites", () => { + it("array brackets with nested elements", () => { + expect(render("Array(Int32)", [1, 2, 3])).toBe("[1,2,3]"); + expect(render("Array(String)", ["a", "b\tc"])).toBe("['a','b\\tc']"); + expect(render("Array(Nullable(Int32))", [1, null, 3])).toBe("[1,NULL,3]"); + expect(render("Array(String)", [])).toBe("[]"); + }); + + it("positional and named tuples both print positionally", () => { + expect(render("Tuple(UInt8, String)", [1, "x"])).toBe("(1,'x')"); + expect(render("Tuple(a UInt8, b String)", { a: 1, b: "x" })).toBe( + "(1,'x')", + ); + }); + + it("map prints {k:v} with quoted stringish keys/values", () => { + const m = new Map([ + ["k", "v"], + ["k2", "w"], + ]); + expect(render("Map(String, String)", m)).toBe("{'k':'v','k2':'w'}"); + }); + + it("nested composites recurse", () => { + const m = new Map([ + ["a", [1, 2]], + ["b", [3]], + ]); + expect(render("Map(String, Array(UInt8))", m)).toBe("{'a':[1,2],'b':[3]}"); + }); + + it("geo types render as Point tuples and their array nestings", () => { + // shapes mirror @clickhouse/rowbinary: Point [x,y]; Ring/LineString + // Point[]; Polygon/MultiLineString Point[][]; MultiPolygon Point[][][]. + expect(render("Point", [1.5, 2.5])).toBe("(1.5,2.5)"); + expect( + render("Ring", [ + [0, 0], + [1, 0], + [1, 1], + ]), + ).toBe("[(0,0),(1,0),(1,1)]"); + expect(render("LineString", [[0, 0]])).toBe("[(0,0)]"); + expect( + render("Polygon", [ + [ + [0, 0], + [1, 0], + ], + [[0.1, 0.1]], + ]), + ).toBe("[[(0,0),(1,0)],[(0.1,0.1)]]"); + expect( + render("MultiPolygon", [ + [ + [ + [0, 0], + [1, 0], + ], + ], + ]), + ).toBe("[[[(0,0),(1,0)]]]"); + // geo renders identically in a nested context + expect(render("Array(Point)", [[1.5, 2.5]])).toBe("[(1.5,2.5)]"); + }); +}); + +describe("compileRowRenderers", () => { + it("builds one top-level renderer per column type", () => { + const renderers = compileRowRenderers(["UInt8", "String", "Array(Int32)"]); + expect(renderers[0]!(7)).toBe("7"); + expect(renderers[1]!("hi")).toBe("hi"); + expect(renderers[2]!([1, 2])).toBe("[1,2]"); + }); +}); diff --git a/tests/clickhouse-test-runner/package.json b/tests/clickhouse-test-runner/package.json index c1e24caea..eb56719c9 100644 --- a/tests/clickhouse-test-runner/package.json +++ b/tests/clickhouse-test-runner/package.json @@ -19,7 +19,8 @@ "test": "vitest run --root ." }, "dependencies": { - "@clickhouse/client": "*" + "@clickhouse/client": "*", + "@clickhouse/rowbinary": "^0.1.2" }, "devDependencies": { "@eslint/js": "^10.0.1", diff --git a/tests/clickhouse-test-runner/rowbinary-allowlist.txt b/tests/clickhouse-test-runner/rowbinary-allowlist.txt new file mode 100644 index 000000000..6b1af3efa --- /dev/null +++ b/tests/clickhouse-test-runner/rowbinary-allowlist.txt @@ -0,0 +1,2287 @@ +# Upstream ClickHouse SQL tests known to pass through the RowBinary backend of +# the tests/clickhouse-test-runner harness (TEST_RUNNER_BACKEND=rowbinary). +# +# For these tests, decoding RowBinaryWithNamesAndTypes with @clickhouse/rowbinary +# and re-rendering to TabSeparated reproduces, byte-for-byte, what the passthrough +# backend (ClickHouse's own output) produces. That round-trip is what exercises +# the dynamic header->reader path; see README.md > Backends. +# +# Conventions: +# - One test name per line (matches the pattern argument of tests/clickhouse-test). +# - Lines starting with '#' and blank lines are ignored. +# - Remove (or comment out with a reason) tests that begin to flake. +# +# This list is a SUBSET of upstream-allowlist.txt: it is the set whose decoded +# output matched passthrough in a differential sweep. Tests that differ (e.g. a +# type the renderer does not yet support) or that are non-deterministic are +# intentionally excluded. Regenerate by running both backends over +# upstream-allowlist.txt and keeping the byte-identical results. +# + +00001_select_1 +00003_reinterpret_as_string +00006_extremes_and_subquery_from +00007_array +00008_array_join +00009_array_join_subquery +00010_big_array_join +00012_array_join_alias_2 +00013_create_table_with_arrays +00014_select_from_table_with_nested +00017_in_subquery_with_empty_result +00018_distinct_in_subquery +00020_sorting_arrays +00021_sorting_arrays +00022_func_higher_order_and_constants +00023_agg_select_agg_subquery +00024_unused_array_join_in_subquery +00025_implicitly_used_subquery_column +00027_distinct_and_order_by +00027_simple_argMinArray +00030_alter_table +00031_parser_number +00032_fixed_string_to_string +00033_fixed_string_to_string +00034_fixed_string_to_number +00035_function_array_return_type +00036_array_element +00037_totals_limit +00040_array_enumerate_uniq +00041_aggregation_remap +00041_big_array_join +00042_set +00043_summing_empty_part +00044_sorting_by_string_descending +00045_sorting_by_fixed_string_descending +00046_stored_aggregates_simple +00047_stored_aggregates_complex +00048_a_stored_aggregates_merge +00048_b_stored_aggregates_merge +00049_any_left_join +00050_any_left_join +00051_any_inner_join +00052_all_left_join +00053_all_inner_join +00054_join_string +00055_join_two_numbers +00056_join_number_string +00060_date_lut +00064_negate_bug +00066_group_by_in +00067_replicate_segfault +00068_empty_tiny_log +00069_date_arithmetic +00071_insert_fewer_columns +00072_in_types +00073_merge_sorting_empty_array_joined +00077_set_keys_fit_128_bits_many_blocks +00078_string_concat +00087_distinct_of_empty_arrays +00087_math_functions +00088_distinct_of_arrays_of_strings +00089_group_by_arrays_of_fixed +00098_1_union_all +00098_2_union_all +00098_3_union_all +00098_4_union_all +00098_5_union_all +00098_6_union_all +00098_7_union_all +00098_8_union_all +00098_9_union_all +00098_a_union_all +00098_b_union_all +00098_c_union_all +00098_d_union_all +00098_e_union_all +00098_f_union_all +00098_g_union_all +00098_h_union_all +00098_j_union_all +00098_l_union_all +00099_join_many_blocks_segfault +00103_ipv4_num_to_string_class_c +00114_float_type_result_of_division +00116_storage_set +00117_parsing_arrays +00120_join_and_group_by +00122_join_with_subquery_with_subquery +00125_array_element_of_array_of_tuple +00127_group_by_concat +00128_group_by_number_and_fixed_string +00129_quantile_timing_weighted +00131_set_hashed +00132_sets +00134_aggregation_by_fixed_string_of_size_1_2_4_8 +00136_duplicate_order_by_elems +00137_in_constants +00138_table_aliases +00140_parse_unix_timestamp_as_datetime +00142_parse_timestamp_as_datetime +00143_number_classification_functions +00144_empty_regexp +00145_empty_likes +00149_function_url_hash +00151_tuple_with_array +00156_array_map_to_constant +00157_aliases_and_lambda_formal_parameters +00159_whitespace_in_columns_list +00160_merge_and_index_in_in +00164_not_chain +00165_transform_non_const_default +00167_settings_inside_query +00169_join_constant_keys +00170_lower_upper_utf8 +00172_constexprs_in_set +00173_compare_date_time_with_constant_string +00174_compare_date_time_with_constant_string_in_in +00175_if_num_arrays +00175_partition_by_ignore +00176_if_string_arrays +00178_function_replicate +00178_query_datetime64_index +00179_lambdas_with_common_expressions_and_filter +00180_attach_materialized_view +00185_array_literals +00187_like_regexp_prefix +00188_constants_as_arguments_of_aggregate_functions +00190_non_constant_array_of_constant_data +00192_least_greatest +00194_identity +00196_float32_formatting +00197_if_fixed_string +00198_group_by_empty_arrays +00201_array_uniq +00202_cross_join +00204_extract_url_parameter +00205_emptyscalar_subquery_type_mismatch_bug +00207_left_array_join +00208_agg_state_merge +00209_insert_select_extremes +00216_bit_test_function_family +00218_like_regexp_newline +00219_full_right_join_column_order +00222_sequence_aggregate_function_family +00227_quantiles_timing_arbitrary_order +00230_array_functions_has_count_equal_index_of_non_const_second_arg +00231_format_vertical_raw +00232_format_readable_decimal_size +00232_format_readable_size +00233_position_function_family +00233_position_function_sql_comparibilty +00234_disjunctive_equality_chains_optimization +00237_group_by_arrays +00238_removal_of_temporary_columns +00239_type_conversion_in_in +00240_replace_substring_loop +00250_tuple_comparison +00251_has_types +00255_array_concat_string +00256_reverse +00258_materializing_tuples +00259_hashing_tuples +00260_like_and_curly_braces +00263_merge_aggregates_and_overflow +00264_uniq_many_args +00266_read_overflow_mode +00267_tuple_array_access_operators_priority +00268_aliases_without_as_keyword +00269_database_table_whitespace +00270_views_query_processing_stage +00272_union_all_and_in_subquery +00277_array_filter +00280_hex_escape_sequence +00283_column_cut +00287_column_const_with_nan +00288_empty_stripelog +00291_array_reduce +00292_parser_tuple_element +00296_url_parameters +00299_stripe_log_multiple_inserts +00300_csv +00306_insert_values_and_expressions +00312_position_case_insensitive_utf8 +00315_quantile_off_by_one +00316_rounding_functions_and_empty_block +00317_in_tuples_and_out_of_range_values +00320_between +00323_quantiles_timing_bug +00324_hashing_enums +00330_view_subqueries +00331_final_and_prewhere_condition_ver_column +00332_quantile_timing_memory_leak +00333_parser_number_bug +00334_column_aggregate_function_limit +00338_replicate_array_of_strings +00342_escape_sequences +00343_array_element_generic +00346_if_tuple +00347_has_tuple +00348_tuples +00349_visible_width +00350_count_distinct +00351_select_distinct_arrays_tuples +00353_join_by_tuple +00355_array_of_non_const_convertible_types +00356_analyze_aggregations_and_union_all +00358_from_string_complex_types +00359_convert_or_zero_functions +00360_to_date_from_string_with_datetime +00362_great_circle_distance +00364_java_style_denormals +00367_visible_width_of_array_tuple_enum +00369_int_div_of_float +00371_union_all +00373_group_by_tuple +00374_any_last_if_merge +00381_first_significant_subdomain +00389_concat_operator +00393_if_with_constant_condition +00397_tsv_format_synonym +00399_group_uniq_array_date_datetime +00401_merge_and_stripelog +00403_to_start_of_day +00404_null_literal +00406_tuples_with_nulls +00413_least_greatest_new_behavior +00414_time_zones_direct_conversion +00420_null_in_scalar_subqueries +00422_hash_function_constexpr +00423_storage_log_single_thread +00425_count_nullable +00426_nulls_sorting +00429_point_in_ellipses +00431_if_nulls +00433_ifnull +00434_tonullable +00435_coalesce +00436_convert_charset +00436_fixed_string_16_comparisons +00437_nulls_first_last +00438_bit_rotate +00439_fixed_string_filter +00441_nulls_in +00442_filter_by_nullable +00445_join_nullable_keys +00447_foreach_modifier +00448_replicate_nullable_tuple_generic +00448_to_string_cut_to_zero +00449_filter_array_nullable_tuple +00450_higher_order_and_nullable +00451_left_array_join_and_constants +00452_left_array_join_and_nullable +00453_top_k +00457_log_tinylog_stripelog_nullable +00459_group_array_insert_at +00461_default_value_of_argument_type +00462_json_true_false_literals +00464_array_element_out_of_range +00464_sort_all_constant_columns +00465_nullable_default +00466_comments_in_keyword +00468_array_join_multiple_arrays_and_use_original_column +00469_comparison_of_strings_containing_null_char +00470_identifiers_in_double_quotes +00472_compare_uuid_with_constant_string +00472_create_view_if_not_exists +00475_in_join_db_table +00477_parsing_data_types +00479_date_and_datetime_to_number +00480_mac_addresses +00481_create_view_for_null +00482_subqueries_and_aliases +00483_cast_syntax +00486_if_fixed_string +00487_if_array_fixed_string +00488_column_name_primary +00488_non_ascii_column_names +00490_special_line_separators_and_characters_outside_of_bmp +00490_with_select +00498_array_functions_concat_slice_push_pop +00498_bitwise_aggregate_functions +00499_json_enum_insert +00500_point_in_polygon_2d_const +00500_point_in_polygon_3d_const +00500_point_in_polygon_bug_2 +00500_point_in_polygon_nan +00500_point_in_polygon_non_const_poly +00502_string_concat_with_array +00503_cast_const_nullable +00507_sumwithoverflow +00511_get_size_of_enum +00513_fractional_time_zones +00516_modulo +00517_date_parsing +00518_extract_all_and_empty_matches +00520_tuple_values_interpreter +00521_multidimensional +00522_multidimensional +00523_aggregate_functions_in_group_array +00524_time_intervals_months_underflow +00525_aggregate_functions_of_nullable_that_return_non_nullable +00526_array_join_with_arrays_of_nullable +00528_const_of_nullable +00529_orantius +00530_arrays_of_nothing +00532_topk_generic +00533_uniq_array +00534_exp10 +00535_parse_float_scientific +00537_quarters +00538_datediff +00538_datediff_plural_units +00539_functions_for_working_with_json +00541_kahan_sum +00541_to_start_of_fifteen_minutes +00544_agg_foreach_of_two_arg +00544_insert_with_select +00545_weird_aggregate_functions +00547_named_tuples +00548_slice_of_nested +00551_parse_or_null +00552_logical_functions_simple +00552_logical_functions_ternary +00552_logical_functions_uint8_as_bool +00552_or_nullable +00553_buff_exists_materlized_column +00553_invalid_nested_name +00554_nested_and_table_engines +00555_right_join_excessive_rows +00556_array_intersect +00556_remove_columns_from_subquery +00557_alter_null_storage_tables +00558_parse_floats +00559_filter_array_generic +00562_in_subquery_merge_tree +00562_rewrite_select_expression_with_union +00566_enum_min_max +00568_empty_function_with_fixed_string +00570_empty_array_is_const +00571_alter_nullable +00576_nested_and_prewhere +00578_merge_table_and_table_virtual_column +00579_merge_tree_partition_and_primary_keys_using_same_expression +00580_cast_nullable_to_non_nullable +00582_not_aliasing_functions +00583_limit_by_expressions +00585_union_all_subquery_aggregation_column_removal +00587_union_all_type_conversions +00589_removal_unused_columns_aggregation +00590_limit_by_column_removal +00591_columns_removal_union_all +00592_union_all_different_aliases +00593_union_all_assert_columns_removed +00597_with_totals_on_empty_set +00599_create_view_with_subquery +00603_system_parts_nonexistent_database +00605_intersections_aggregate_functions +00606_quantiles_and_nans +00607_index_in_in +00608_uniq_array +00609_prewhere_and_default +00612_union_query_with_subquery +00617_array_in +00618_nullable_in +00619_union_highlite +00622_select_in_parens +00624_length_utf8 +00625_arrays_in_nested +00626_in_syntax +00627_recursive_alias +00628_in_lambda_on_merge_table_bug +00633_func_or_in +00634_rename_view +00639_startsWith +00644_different_expressions_with_same_alias +00647_histogram +00647_histogram_negative +00647_select_numbers_with_offset +00649_quantile_tdigest_negative +00650_array_enumerate_uniq_with_tuples +00653_monotonic_integer_cast +00661_array_has_silviucpp +00662_array_has_nullable +00662_has_nullable +00663_tiny_log_empty_insert +00664_cast_from_string_to_nullable +00665_alter_nullable_string_to_nullable_uint8 +00666_uniq_complex_types +00667_compare_arrays_of_different_types +00668_compare_arrays_silviucpp +00671_max_intersections +00672_arrayDistinct +00673_subquery_prepared_set_performance +00674_has_array_enum +00676_group_by_in +00678_murmurhash +00679_uuid_in_key +00680_duplicate_columns_inside_union_all +00681_duplicate_columns_inside_union_all_stas_sviridov +00687_insert_into_mv +00688_aggregation_retention +00688_case_without_else +00688_low_cardinality_alter_add_column +00688_low_cardinality_defaults +00688_low_cardinality_dictionary_deserialization +00688_low_cardinality_prewhere +00688_low_cardinality_serialization +00689_join_table_function +00691_array_distinct +00696_system_columns_limit +00700_decimal_with_default_precision_and_scale +00701_context_use_after_free +00702_join_with_using_dups +00702_where_with_quailified_names +00703_join_crash +00704_arrayCumSumLimited_arrayDifference +00710_array_enumerate_dense +00711_array_enumerate_variants +00712_prewhere_with_alias_and_virtual_column +00712_prewhere_with_missing_columns +00712_prewhere_with_missing_columns_2 +00713_collapsing_merge_tree +00715_bounding_ratio_merge_empty +00717_default_join_type +00717_low_cardinaliry_group_by +00718_format_datetime_1 +00719_format_datetime_f_varsize_bug +00719_format_datetime_rand +00720_combinations_of_aggregate_combinators +00722_inner_join +00723_remerge_sort +00725_join_on_bug_1 +00725_join_on_bug_3 +00725_join_on_bug_4 +00726_length_aliases +00726_materialized_view_concurrent +00726_modulo_for_date +00733_if_datetime +00735_or_expr_optimize_bug +00738_nested_merge_multidimensional_array +00740_optimize_predicate_expression +00745_compile_scalar_subquery +00746_compile_non_deterministic_function +00746_hashing_tuples +00747_contributors +00750_merge_tree_merge_with_o_direct +00751_hashing_ints +00752_low_cardinality_array_result +00752_low_cardinality_permute +00753_alter_destination_for_storage_buffer +00753_quantile_format +00753_with_with_single_alias +00754_alter_modify_column_partitions +00754_first_significant_subdomain_more +00755_avg_value_size_hint_passing +00756_power_alias +00759_kodieg +00760_insert_json_with_defaults +00760_url_functions_overflow +00761_lower_utf8_bug +00765_sql_compatibility_aliases +00780_unaligned_array_join +00799_function_dry_run +00800_low_cardinality_array_group_by_arg +00800_low_cardinality_empty_array +00801_daylight_saving_time_hour_underflow +00802_daylight_saving_time_shift_backwards_at_midnight +00802_system_parts_with_datetime_partition +00803_odbc_driver_2_format +00803_xxhash +00804_rollup_with_having +00807_regexp_quote_meta +00810_in_operators_segfault +00812_prewhere_alias_array +00813_parse_date_time_best_effort_more +00814_parsing_ub +00816_join_column_names_sarg +00817_with_simple +00818_join_bug_4271 +00819_ast_refactoring_bugs +00820_multiple_joins +00820_multiple_joins_subquery_requires_alias +00822_array_insert_default +00823_sequence_match_dfa +00824_filesystem +00829_bitmap64_function +00834_date_datetime_cmp +00834_not_between +00836_numbers_table_function_zero +00839_bitmask_negative +00840_top_k_weighted +00844_join_lightee2 +00845_join_on_aliases +00847_multiple_join_same_column +00853_join_with_nulls_crash +00854_multiple_join_asterisks +00856_no_column_issue_4242 +00859_distinct_with_join +00860_unknown_identifier_bug +00870_t64_codec +00871_t64_codec_signed +00873_t64_codec_date +00874_issue_3495 +00876_wrong_arraj_join_column +00880_decimal_in_key +00881_unknown_identifier_in_in +00882_multiple_join_no_alias +00897_flatten +00898_quantile_timing_parameter_check +00901_joint_entropy +00902_entropy +00903_array_with_constant_function +00904_array_with_constant_2 +00905_compile_expressions_compare_big_dates +00905_field_with_aggregate_function_state +00906_low_cardinality_const_argument +00906_low_cardinality_rollup +00907_set_index_with_nullable_and_low_cardinality +00907_set_index_with_nullable_and_low_cardinality_bug +00908_analyze_query +00909_ngram_distance +00912_string_comparison +00914_join_bgranvea +00915_tuple_orantius +00916_add_materialized_column_after +00916_create_or_replace_view +00917_least_sqr +00917_multiple_joins_denny_crane +00919_histogram_merge +00919_sum_aggregate_states_constants +00920_multiply_aggregate_states_constants +00926_adaptive_index_granularity_collapsing_merge_tree +00926_adaptive_index_granularity_merge_tree +00926_adaptive_index_granularity_replacing_merge_tree +00926_adaptive_index_granularity_versioned_collapsing_merge_tree +00926_multimatch +00927_asof_join_noninclusive +00927_asof_joins +00928_multi_match_constant_constant +00930_arrayIntersect +00931_low_cardinality_nullable_aggregate_function_type +00931_low_cardinality_read_with_empty_array +00931_low_cardinality_set_index_in_key_condition +00932_array_intersect_bug +00932_geohash_support +00933_reserved_word +00933_ttl_with_default +00934_is_valid_utf8 +00936_crc_functions +00938_basename +00939_limit_by_offset +00939_test_null_in +00944_minmax_null +00950_default_prewhere +00950_test_gorilla_codec +00952_part_frozen_info +00954_resample_combinator +00956_join_use_nulls_with_array_column +00957_coalesce_const_nullable_crash +00960_eval_ml_method_const +00961_checksums_in_system_parts_columns_table +00961_visit_param_buffer_underflow +00962_visit_param_various +00963_startsWith_force_primary_key +00964_os_thread_priority +00966_invalid_json_must_not_parse +00968_roundAge +00969_roundDuration +00973_create_table_as_table_function +00974_bitmapContains_with_primary_key +00974_full_outer_join +00975_json_hang +00977_join_use_nulls_denny_crane +00978_ml_math +00978_sum_map_bugfix +00978_table_function_values_alias +00979_quantileExcatExclusive_and_Inclusive +00979_set_index_not +00980_full_join_crash_fancyqlx +00981_no_virtual_columns +00982_array_enumerate_uniq_ranked +00990_function_current_user +00994_table_function_numbers_mt +00995_optimize_read_in_order_with_aggregation +00996_limit_with_ties +00997_extract_all_crash_6627 +00997_trim +00999_settings_no_extra_quotes +01001_enums_in_in_section +01009_insert_select_data_loss +01009_insert_select_nicelulu +01010_partial_merge_join_const_and_lc +01011_group_uniq_array_memsan +01012_select_limit_x_0 +01013_hex_decimal +01013_hex_float +01015_array_split +01015_attach_part +01015_random_constant +01016_index_tuple_field_type +01016_null_part_minmax +01016_uniqCombined64 +01018_optimize_read_in_order_with_in_subquery +01019_array_fill +01020_function_array_compact +01020_function_char +01020_having_without_group_by +01025_array_compact_generic +01026_char_utf8 +01030_concatenate_equal_fixed_strings +01030_final_mark_empty_primary_key +01032_cityHash64_for_decimal +01032_cityHash64_for_UUID +01034_order_by_pk_prefix +01034_with_fill_and_push_down_predicate +01035_prewhere_with_alias +01040_h3_get_resolution +01041_h3_is_valid +01042_check_query_and_last_granule_size +01043_categorical_iv +01043_h3_edge_length_m +01044_great_circle_angle +01044_h3_edge_angle +01045_bloom_filter_null_array +01047_no_alias_columns_with_table_aliases +01047_nullable_rand +01047_simple_aggregate_sizes_of_columns_bug +01050_group_array_sample +01051_random_printable_ascii +01051_scalar_optimization +01053_if_chain_check +01055_prewhere_bugs +01056_negative_with_bloom_filter +01061_alter_codec_with_type +01062_pm_multiple_all_join_same_value +01063_create_column_set +01064_pm_all_join_const_and_nullable +01065_array_zip_mixed_const +01066_bit_count +01067_join_null +01068_parens +01069_set_in_group_by +01070_h3_get_base_cell +01070_h3_hex_area_m2 +01070_h3_indexes_are_neighbors +01070_h3_to_parent +01070_h3_to_string +01070_string_to_h3 +01071_in_array +01072_nullable_jit +01072_select_constant_limit +01073_blockSerializedSize +01073_crlf_end_of_line +01075_in_arrays_enmk +01076_array_join_prewhere_const_folding +01076_range_reader_segfault +01078_bloom_filter_operator_not_has +01079_bit_operations_using_bitset +01079_new_range_reader_segfault +01079_order_by_pk +01079_reinterpret_as_fixed_string +01081_keywords_formatting +01083_functional_index_in_mergetree +01083_log_first_column_alias +01085_extract_all_empty +01085_simdjson_uint64 +01086_modulo_or_zero +01087_index_set_ubsan +01090_fixed_string_bit_ops +01091_query_profiler_does_not_hang +01096_array_reduce_in_ranges +01096_block_serialized_state +01096_zeros +01097_one_more_range_reader_test_wide_part +01097_pre_limit +01100_split_by_string +01104_fixed_string_like +01105_string_like +01106_const_fixed_string_like +01109_sc0rp10_string_hash_map_zero_bytes +01112_check_table_with_index +01114_alter_modify_compact_parts +01114_clear_column_compact_parts +01115_prewhere_array_join +01116_cross_count_asterisks +01117_chain_finalize_bug +01117_greatest_least_case +01119_optimize_trivial_insert_select +01120_join_constants +01123_parse_date_time_best_effort_even_more +01124_view_bad_types +01136_multiple_sets +01137_order_by_func_final +01137_sample_final +01143_trivial_count_with_join +01144_join_rewrite_with_ambiguous_column_and_view +01144_multiple_joins_rewriter_v2_and_lambdas +01145_with_fill_const +01163_search_case_insensetive_utf8 +01189_create_as_table_as_table_function +01197_summing_enum +01198_plus_inf +01199_url_functions_path_without_schema_yiurule +01212_empty_join_and_totals +01213_alter_rename_compact_part +01220_scalar_optimization_in_alter +01234_to_string_monotonic +01247_least_greatest_filimonov +01248_least_greatest_mixed_const +01250_fixed_string_comparison +01251_string_comparison +01255_geo_types_livace +01262_low_cardinality_remove +01264_nested_baloo_bear +01266_default_prewhere_reqq +01268_mergine_sorted_limit +01269_alias_type_differs +01272_offset_without_limit +01273_lc_fixed_string_field +01276_random_string +01276_system_licenses +01277_large_tuples +01278_alter_rename_combination +01278_variance_nonnegative +01279_dist_group_by +01280_null_in +01280_unicode_whitespaces_lexer +01281_join_with_prewhere_fix +01281_sum_nullable +01283_strict_resize_bug +01284_view_and_extremes_bug +01285_date_datetime_key_condition +01289_min_execution_speed_not_too_early +01290_empty_array_index_analysis +01291_aggregation_in_order +01292_quantile_array_bug +01293_external_sorting_limit_bug +01296_pipeline_stuck +01300_polygon_convex_hull +01303_polygons_equals +01305_array_join_prewhere_in_subquery +01307_polygon_perimeter +01308_row_policy_and_trivial_count_query +01312_case_insensitive_regexp +01314_position_in_system_columns +01315_count_distinct_return_not_nullable +01318_parallel_final_stuck +01321_monotonous_functions_in_order_by_bug +01323_too_many_threads_bug +01324_settings_documentation +01326_build_id +01326_fixed_string_comparison_denny_crane +01326_hostname_alias +01328_bad_peephole_optimization +01338_sha256_fixedstring +01338_uuid_without_separator +01341_datetime64_wrong_supertype +01345_array_join_LittleMaverick +01345_index_date_vs_datetime +01346_array_join_mrxotey +01347_partition_date_vs_datetime +01351_geohash_assert +01351_parse_date_time_best_effort_us +01352_add_datetime_bad_get +01353_nullable_tuple +01353_topk_enum +01354_order_by_tuple_collate_const +01354_tuple_low_cardinality_array_mapped_bug +01355_defaultValueOfArgumentType_bug +01356_initialize_aggregation +01357_result_rows +01358_mutation_delete_null_rows +01359_codeql +01359_geodistance_loop +01360_division_overflow +01361_buffer_table_flush_with_materialized_view +01362_year_of_ISO8601_week_modificators_for_formatDateTime +01372_wrong_order_by_removal +01374_if_nullable_filimonov +01375_null_issue_3767 +01375_storage_file_write_prefix_csv_with_names +01375_storage_file_write_prefix_tsv_with_names +01376_array_fill_empty +01376_null_logical +01379_with_fill_several_columns +01385_not_function +01389_filter_by_virtual_columns +01390_check_table_codec +01396_negative_datetime_saturate_to_zero +01398_in_tuple_func +01400_join_get_with_multi_keys +01403_datetime64_constant_arg +01409_topK_merge +01410_full_join_and_null_predicates +01410_nullable_key_and_index_negate_cond +01411_from_unixtime +01411_xor_itai_shirav +01412_mod_float +01412_optimize_deduplicate_bug +01413_if_array_uuid +01413_truncate_without_table_keyword +01414_bloom_filter_index_with_const_column +01416_join_totals_header_bug +01417_update_permutation_crash +01418_index_analysis_bug +01419_materialize_null +01419_skip_index_compact_parts +01420_logical_functions_materialized_null +01421_array_nullable_element_nullable_index +01422_array_nullable_element_nullable_index +01423_if_nullable_cond +01426_geohash_constants +01427_pk_and_expression_with_different_type +01428_hash_set_nan_key +01430_fix_any_rewrite_aliases +01431_finish_sorting_with_consts +01431_utf8_ubsan +01434_netloc_fuzz +01440_big_int_shift +01441_array_combinator +01450_set_null_const +01451_normalize_query +01452_normalized_query_hash +01453_fixsed_string_sort +01453_normalize_query_alias_uuid +01455_nullable_type_with_if_agg_combinator +01455_time_zones +01456_low_cardinality_sorting_bugfix +01456_min_negative_decimal_formatting +01457_compile_expressions_fuzzer +01457_order_by_limit +01457_order_by_nulls_first +01458_count_digits +01458_is_decimal_overflow +01458_named_tuple_millin +01460_allow_dollar_and_number_in_identifier +01460_mark_inclusion_search_crash +01471_with_format +01475_fix_bigint_shift +01475_mutation_with_if +01479_cross_join_9855 +01480_binary_operator_monotonicity +01481_join_with_materialized +01485_256_bit_multiply +01490_nullable_string_to_enum +01491_nested_multiline_comments +01492_array_join_crash_13829 +01492_format_readable_quantity +01493_table_function_null +01495_subqueries_in_with_statement_2 +01495_subqueries_in_with_statement_4 +01496_signedness_conversion_monotonicity +01497_alias_on_default_array +01497_extract_all_groups_empty_match +01497_now_support_timezone +01502_bar_overflow +01503_fixed_string_primary_key +01503_if_const_optimization +01504_view_type_conversion +01508_explain_header +01511_different_expression_with_same_alias +01511_format_readable_timedelta +01511_prewhere_with_virtuals +01513_ilike_like_cache +01514_tid_function +01518_cast_nullable_virtual_system_column +01518_filtering_aliased_materialized_column +01518_nullable_aggregate_states1 +01518_select_in_null +01521_alter_enum_and_reverse_read +01521_max_length_alias +01523_date_time_compare_with_date_literal +01523_interval_operator_support_string_literal +01532_tuple_with_name_type +01533_distinct_depends_on_max_threads +01533_distinct_nullable_uuid +01534_lambda_array_join +01537_fuzz_count_equal +01540_verbatim_partition_pruning +01543_parse_datetime_besteffort_or_null_empty_string +01544_errorCodeToName +01548_lzy305 +01550_mutation_subquery +01552_impl_aggfunc_cloneresize +01554_bloom_filter_index_big_integer_uuid +01554_interpreter_integer_float +01555_or_fill +01556_if_null +01560_cancel_agg_func_combinator_native_name_constraint +01560_monotonicity_check_multiple_args_bug +01561_aggregate_functions_of_key_with_join +01567_system_processes_current_database +01576_if_null_external_aggregation +01580_column_const_comparision +01582_any_join_supertype +01583_const_column_in_set_index +01585_fuzz_bits_with_bugfix +01592_length_map +01592_toUnixTimestamp_Date +01596_full_join_chertus +01600_encode_XML +01600_min_max_compress_block_size +01603_decimal_mult_float +01605_dictinct_two_level +01605_key_condition_enum_int +01614_with_fill_with_limit +01615_two_args_function_index_fix +01616_untuple_access_field +01623_byte_size_const +01631_date_overflow_as_partition_key +01632_select_all_syntax +01633_limit_fuzz +01634_summap_nullable +01635_nullable_fuzz +01638_div_mod_ambiguities +01646_fix_window_funnel_inconistency +01648_normalize_query_keep_names +01649_with_alias_key_condition +01650_expressions_merge_bug +01654_bar_nan +01655_quarter_modificator_for_formatDateTime +01655_test_isnull_mysql_dialect +01655_window_functions_bug +01655_window_functions_null +01656_ipv4_bad_formatting +01656_join_defaul_enum +01656_test_hex_mysql_dialect +01657_array_element_ubsan +01657_test_toHour_mysql_compatibility +01658_test_base64Encode_mysql_compatibility +01659_array_aggregation_ubsan +01659_test_base64Decode_mysql_compatibility +01660_join_or_all +01660_system_parts_smoke +01660_test_toDayOfYear_mysql_compatibility +01661_join_complex +01661_test_toDayOfWeek_mysql_compatibility +01662_join_mixed +01662_test_toDayOfMonth_mysql_compatibility +01663_aes_msan +01663_quantile_weighted_overflow +01663_test_toDate_mysql_compatibility +01664_array_slice_ubsan +01664_ntoa_aton_mysql_compatibility +01665_merge_tree_min_for_concurrent_read +01665_substring_ubsan +01666_date_lut_buffer_overflow +01666_great_circle_distance_ubsan +01668_avg_weighted_ubsan +01668_test_toMonth_mysql_dialect +01669_join_or_duplicates +01669_test_toYear_mysql_dialect +01670_sign_function +01670_test_repeat_mysql_dialect +01671_aggregate_function_group_bitmap_data +01671_test_toQuarter_mysql_dialect +01672_actions_dag_merge_crash +01672_test_toSecond_mysql_dialect +01673_test_toMinute_mysql_dialect +01674_filter_by_uint8 +01674_htm_xml_coarse_parse +01674_unicode_asan +01676_range_hashed_dictionary +01676_round_int_ubsan +01677_array_enumerate_bug +01678_great_circle_angle +01679_format_readable_time_delta_inf +01680_predicate_pushdown_union_distinct_subquery +01681_arg_min_max_if_fix +01684_geohash_ubsan +01685_json_extract_double_as_float +01690_quantilesTiming_ubsan +01700_deltasum +01700_mod_negative_type_promotion +01702_bitmap_native_integers +01704_transform_with_float_key +01707_join_use_nulls +01710_aggregate_projection_with_hashing +01710_aggregate_projection_with_monotonic_key_expr +01710_minmax_count_projection_constant_query +01710_minmax_count_projection_count_nullable +01710_minmax_count_projection_modify_partition_key +01710_normal_projection_fix1 +01710_normal_projection_format +01710_normal_projection_join_plan_fix +01710_normal_projection_with_query_plan_optimization +01710_projection_additional_filters +01710_projection_aggregate_functions_null_for_empty +01710_projection_array_join +01710_projection_detach_part +01710_projection_external_aggregate +01710_projection_in_index +01710_projection_in_set +01710_projection_materialize_with_missing_columns +01710_projection_mutation +01710_projection_optimize_aggregators_of_group_by_keys +01710_projection_optimize_group_by_function_keys +01710_projection_part_check +01710_projection_pk_trivial_count +01710_projection_row_policy +01710_projection_with_ast_rewrite_settings +01710_projection_with_column_transformers +01710_projection_with_joins +01710_projections_group_by_no_key +01711_cte_subquery_fix +01712_no_adaptive_granularity_vertical_merge +01716_array_difference_overflow +01718_subtract_seconds_date +01720_union_distinct_with_limit +01732_explain_syntax_union_query +01733_transform_ubsan +01735_join_get_low_card_fix +01735_to_datetime64 +01736_null_as_default +01744_tuple_cast_to_map_bugfix +01746_convert_type_with_default +01746_lc_values_format_bug +01746_test_for_tupleElement_must_be_constant_issue +01747_transform_empty_arrays +01748_partition_id_pruning +01753_mutate_table_predicated_with_table +01760_ddl_dictionary_use_current_database_name +01761_cast_to_enum_nullable +01761_round_year_bounds +01762_datetime64_extended_parsing +01763_support_map_lowcardinality_type +01764_collapsing_merge_adaptive_granularity +01764_table_function_dictionary +01765_move_to_table_overlapping_block_number +01766_todatetime64_no_timezone_arg +01768_array_product +01770_add_months_ubsan +01772_intdiv_minus_one_ubsan +01773_case_sensitive_revision +01773_case_sensitive_version +01773_min_max_time_system_parts_datetime64 +01774_case_sensitive_connection_id +01774_tuple_null_in +01778_where_with_column_name +01779_quantile_deterministic_msan +01780_column_sparse_distinct +01780_column_sparse_filter +01780_column_sparse_pk +01780_column_sparse_tuple +01780_dict_get_or_null +01781_map_op_ubsan +01781_merge_tree_deduplication +01781_token_extractor_buffer_overflow +01783_merge_engine_join_key_condition +01785_pmj_lc_bug +01786_group_by_pk_many_streams +01787_arena_assert_column_nothing +01795_TinyLog_rwlock_ub +01796_Log_rwlock_ub +01797_StripeLog_rwlock_ub +01798_uniq_theta_union_intersect_not +01800_log_nested +01803_const_nullable_map +01803_untuple_subquery +01809_inactive_parts_to_delay_throw_insert +01811_datename +01811_filter_by_null +01812_has_generic +01813_quantileBfloat16_nans +01818_case_float_value_fangyc +01818_move_partition_simple +01820_unhex_case_insensitive +01822_union_and_constans_error +01831_max_streams +01832_memory_write_suffix +01833_test_collation_alvarotuso +01835_alias_to_primary_key_cyfdecyf +01837_cast_to_array_from_empty_array +01838_system_dictionaries_virtual_key_column +01839_join_to_subqueries_rewriter_columns_matcher +01845_add_testcase_for_arrayElement +01846_alter_column_without_type_bugfix +01846_null_as_default_for_insert_select +01851_fix_row_policy_empty_result +01851_s2_to_geo +01852_jit_if +01852_s2_get_neighbours +01855_jit_comparison_constant_result +01866_aggregate_function_interval_length_sum +01866_bit_positions_to_array +01866_datetime64_cmp_with_constant +01867_fix_storage_memory_mutation +01869_function_modulo_legacy +01869_reinterpret_as_fixed_string_uuid +01871_merge_tree_compile_expressions +01881_aggregate_functions_versioning +01881_create_as_tuple +01881_to_week_monotonic_fix +01881_total_bytes_storage_buffer +01881_union_header_mismatch_bug +01889_tokenize +01891_not_in_partition_prune +01891_not_like_partition_prune +01891_partition_by_uuid +01906_h3_to_geo +01906_partition_by_multiply_by_zero +01908_with_unknown_column +01909_mbtolou +01911_logical_error_minus +01912_bad_cast_join_fuzz +01913_fix_column_transformer_replace_format +01913_join_push_down_bug +01914_index_bgranvea +01914_ubsan_quantile_timing +01915_json_extract_raw_string +01916_low_cardinality_interval +01916_lowcard_dict_type +01916_multiple_join_view_optimize_predicate_chertus +01917_prewhere_column_type +01922_array_join_with_index +01925_date_date_time_comparison +01925_json_as_string_data_in_square_brackets +01925_merge_prewhere_table +01925_test_group_by_const_consistency +01926_bin_unbin +01926_union_all_schmak +01932_alter_index_with_order +01936_empty_function_support_uuid +01936_quantiles_cannot_return_null +01938_joins_identifiers +01940_pad_string +01940_totimezone_operator_monotonicity +01941_dict_get_has_complex_single_key +01942_untuple_transformers_msan +01943_log_column_sizes +01960_lambda_precedence +02001_join_on_const +02001_select_with_filter +02002_sampling_and_unknown_column_bug +02003_bug_from_23515 +02006_use_constants_in_with_and_select +02013_emptystring_cast +02015_order_by_with_fill_misoptimization +02016_order_by_with_fill_monotonic_functions_removal +02017_columns_with_dot_2 +02017_order_by_with_fill_redundant_functions +02019_multiple_weird_with_fill +02020_cast_integer_overflow +02020_exponential_smoothing_cross_block +02021_map_bloom_filter_index +02021_map_has +02021_prewhere_always_true_where +02022_array_full_text_bloom_filter_index +02023_parser_number_binary_literal +02025_having_filter_column +02025_subcolumns_compact_parts +02026_arrayDifference_const +02027_arrayCumSumNonNegative_const +02028_system_data_skipping_indices_size +02028_tokens +02030_function_mapContainsKeyLike +02030_quantiles_underflow +02032_short_circuit_least_greatest_bug +02036_jit_short_circuit +02041_openssl_hash_functions_test +02042_map_get_non_const_key +02045_like_function +02047_alias_for_table_and_database_name +02095_function_get_os_kernel_version +02097_initializeAggregationNullable +02098_date32_comparison +02100_limit_push_down_bug +02100_now64_types_bug +02100_replaceRegexpAll_bug +02111_with_fill_no_rows +02112_skip_index_set_and_or +02113_base64encode_trailing_bytes_1 +02113_untuple_func_alias +02124_empty_uuid +02124_uncompressed_cache +02125_low_cardinality_int256 +02129_window_functions_disable_optimizations +02131_materialize_column_cast +02131_remove_columns_in_subquery +02131_row_policies_combination +02131_skip_index_not_materialized +02132_empty_mutation_livelock +02133_final_prewhere_where_lowcardinality_replacing +02148_cast_type_parsing +02148_issue_32737 +02149_issue_32487 +02150_replace_regexp_all_empty_match +02151_lc_prefetch +02151_replace_regexp_all_empty_match_alternative +02152_count_distinct_optimization +02152_short_circuit_throw_if +02154_bitmap_contains +02155_nested_lc_defalut_bug +02155_parse_date_lowcard_default_throw +02157_line_as_string_output_format +02158_interval_length_sum +02160_h3_hex_area_Km2 +02160_monthname +02160_special_functions +02161_array_first_last +02162_array_first_last_index +02163_operators +02165_h3_edge_length_km +02167_columns_with_dots_default_values +02169_fix_view_offset_limit_setting +02176_toStartOfWeek_overflow_pruning +02178_column_function_insert_from +02179_bool_type +02180_group_by_lowcardinality +02187_test_final_and_limit_modifier +02188_table_function_format +02189_join_type_conversion +02190_current_metrics_query +02191_parse_date_time_best_effort_more_cases +02205_map_populate_series_non_const +02205_postgresql_functions +02206_array_starts_ends_with +02207_key_condition_floats +02209_short_circuit_node_without_parents +02210_append_to_dev_dull +02210_toColumnTypeName_toLowCardinality_const +02212_cte_and_table_alias +02220_array_join_format +02224_s2_test_const_columns +02226_low_cardinality_text_bloom_filter_index +02232_partition_pruner_mixed_constant_type +02232_partition_pruner_single_point +02234_position_case_insensitive_utf8 +02240_asof_join_biginteger +02240_get_type_serialization_streams +02241_array_first_last_or_null +02241_short_circuit_short_column +02242_negative_datetime64 +02242_optimize_to_subcolumns_no_storage +02242_throw_if_constant_argument +02243_in_ip_address +02243_ipv6_long_parsing +02245_format_string_stack_overflow +02245_join_with_nullable_lowcardinality_crash +02247_fix_extract_parser +02248_nullable_custom_types_to_string +02251_last_day_of_month +02265_cross_join_empty_list +02265_limit_push_down_over_window_functions_bug +02267_output_format_prometheus +02267_special_operator_parse_alias_check +02267_type_inference_for_insert_into_function_null +02277_full_sort_join_misc +02285_hex_bin_support_more_types +02286_convert_decimal_type +02286_function_wyhash +02292_hash_array_tuples +02293_ilike_on_fixed_strings +02293_optimize_aggregation_in_order_Array_functions +02294_optimize_aggregation_in_order_prefix_Array_functions +02294_system_certificates +02295_global_with_in_subquery +02296_nullable_arguments_in_array_filter +02302_clash_const_aggegate_join +02304_grouping_set_order_by +02306_window_move_row_number_fix +02307_join_get_array_null +02310_generate_multi_columns_with_uuid +02310_uuid_v7 +02312_is_not_null_prewhere +02313_cross_join_dup_col_names +02313_dump_column_structure_low_cardinality +02313_multiple_limits +02313_negative_datetime64 +02313_test_fpc_codec +02315_pmj_union_ubsan_35857 +02316_const_string_intersact +02316_literal_no_octal +02316_values_table_func_bug +02320_alter_columns_with_dots +02320_mapped_array_witn_const_nullable +02321_nested_short_circuit_functions +02325_dates_schema_inference +02336_sort_optimization_with_fill +02337_multiple_joins_original_names +02343_analyzer_lambdas_issue_28083 +02343_analyzer_lambdas_issue_36677 +02345_analyzer_subqueries +02345_create_table_allow_trailing_comma +02345_partial_sort_transform_optimization +02346_position_countsubstrings_zero_byte +02346_text_index_bug47393 +02346_text_index_bug54541 +02346_text_index_bug84805 +02346_text_index_bug87887 +02346_text_index_collapsingmergetree +02346_text_index_default_granularity +02346_text_index_dictionary_frontcoding +02346_text_index_direct_read_crash +02346_text_index_direct_read_nullable +02346_text_index_direct_read_with_query_condition_cache +02346_text_index_functions_with_empty_needle +02346_to_hour_monotonicity_fix_2 +02347_rank_corr_nan +02347_rank_corr_size_overflow +02351_Map_combinator_dist +02353_ascii +02353_isnullable +02353_partition_prune_nullable_key +02354_array_lowcardinality +02354_numeric_literals_with_underscores +02354_tuple_element_with_default +02354_tuple_lowcardinality +02354_vector_search_and_other_skipping_indexes +02354_vector_search_default_granularity +02354_vector_search_multiple_indexes +02354_vector_search_subquery +02354_vector_search_unquoted_index_parameters +02355_column_type_name_lc +02355_control_block_size_in_aggregator +02360_small_notation_h_for_hour_interval +02363_mapupdate_improve +02364_window_case +02366_asof_optimize_predicate_bug_37813 +02366_explain_query_tree +02366_normalize_aggregate_function_types_and_states +02367_analyzer_table_alias_columns +02367_optimize_trivial_count_with_array_join +02370_extractAll_regress +02371_select_projection_normal_agg +02374_combine_multi_if_and_count_if_opt +02374_in_tuple_index +02374_regexp_replace +02375_double_escaping_json +02375_scalar_lc_cte +02380_analyzer_join_sample +02381_parseDateTime64BestEffortUS +02383_schema_inference_hints +02386_analyzer_in_function_nested_subqueries +02387_analyzer_cte +02392_every_setting_must_have_documentation +02393_every_metric_must_have_documentation +02394_every_profile_event_must_have_documentation +02395_every_merge_tree_setting_must_have_documentation +02401_merge_tree_old_tmp_dirs_cleanup +02405_pmj_issue_40335 +02406_try_read_datetime64_bug +02408_to_fixed_string_short_circuit +02409_url_format_detection +02410_csv_empty_fields_inference +02414_all_new_table_functions_must_be_documented +02415_all_new_functions_must_be_documented +02416_row_policy_always_false_index +02417_from_select_syntax +02418_tautological_if_index +02421_decimal_in_precision_issue_41125 +02423_json_quote_float64 +02426_to_string_nullable_fixedstring +02427_column_nullable_ubsan +02427_msan_group_array_resample +02428_batch_nullable_assert +02428_delete_with_settings +02428_index_analysis_with_null_literal +02452_check_low_cardinality +02454_compressed_marks_in_compact_part +02455_extract_fixed_string_from_nested_json +02456_aggregate_state_conversion +02456_BLAKE3_hash_function_test +02457_datediff_via_unix_epoch +02457_key_condition_with_types_that_cannot_be_nullable +02457_morton_coding_with_mask +02460_prewhere_row_level_policy +02462_distributions +02467_cross_join_three_table_functions +02467_set_with_lowcardinality_type +02468_has_any_tuple +02471_wrong_date_monotonicity +02473_extract_low_cardinality_from_json +02473_map_element_nullable +02474_extract_fixedstring_from_json +02474_timeDiff_UTCTimestamp +02474_unhex_in_fix_string +02475_analyzer_join_tree_subquery +02475_analyzer_subquery_compound_expression +02475_bad_cast_low_cardinality_to_string_bug +02475_date_time_schema_inference_bug +02475_or_function_alias_and_const_where +02475_positive_modulo +02476_analyzer_join_with_unused_columns +02477_analyzer_ast_key_condition_crash +02477_logical_expressions_optimizer_issue_89803 +02478_analyzer_table_expression_aliases +02479_analyzer_aggregation_crash +02479_if_with_null_and_cullable_const +02479_nullable_primary_key_non_first_column +02480_every_asynchronous_metric_must_have_documentation +02480_interval_casting_and_subquery +02480_parse_date_time_best_effort_math_overflow +02481_aggregation_in_order_plan +02481_low_cardinality_with_short_circuit_functins_mutations +02481_xxh3_hash_function +02482_if_with_nothing_argument +02489_analyzer_indexes +02493_analyzer_sum_if_to_count_if +02493_analyzer_table_functions_untuple +02494_analyzer_cte_resolution_in_subquery_fix +02494_array_function_range +02494_parser_string_binary_literal +02495_sum_if_to_count_if_bug +02497_analyzer_sum_if_count_if_pass_crash_fix +02497_storage_join_right_assert +02499_analyzer_set_index +02499_escaped_quote_schema_inference +02500_analyzer_storage_view_crash_fix +02501_analyzer_expired_context_crash_fix +02502_analyzer_insert_select_crash_fix +02503_in_lc_const_args_bug +02503_join_switch_alias_fuzz +02504_bar_fractions +02509_h3_arguments +02510_group_by_prewhere_null +02512_array_join_name_resolution +02513_analyzer_duplicate_alias_crash_fix +02513_analyzer_sort_msan +02513_broken_datetime64_init_on_mac +02514_null_dictionary_source +02514_tsv_zero_started_number +02515_aggregate_functions_statistics +02515_analyzer_null_for_empty +02515_and_or_if_multiif_not_return_lc +02515_distinct_zero_size_key_bug_44831 +02515_generate_ulid +02516_projections_with_rollup +02518_qualified_asterisks_alias_table_name +02521_analyzer_aggregation_without_column +02521_cannot_find_column_in_projection +02523_range_const_start +02524_fuzz_and_fuss_2 +02525_analyzer_function_in_crash_fix +02525_jit_logical_functions_nan +02525_range_hashed_dictionary_update_field +02530_ip_part_id +02531_semi_join_null_const_bug +02532_analyzer_aggregation_with_rollup +02532_profileevents_server_startup_time +02533_generate_random_schema_inference +02535_analyzer_limit_offset +02535_ip_parser_not_whole +02536_replace_with_nonconst_needle_and_replacement +02537_system_formats +02538_analyzer_create_table_as_select +02538_ngram_bf_index_with_null +02538_nullable_array_tuple_timeseries +02539_generate_random_ip +02539_generate_random_low_cardinality +02539_generate_random_map +02540_date_column_consistent_insert_behaviour +02541_analyzer_grouping_sets_crash_fix +02541_empty_function_support_ip +02541_multiple_ignore_with_nested_select +02541_tuple_element_with_null +02542_case_no_else +02542_table_function_format +02551_ipv4_implicit_uint64 +02552_sparse_columns_intersect +02554_format_json_columns_for_empty +02559_add_parts +02559_ip_types_bloom +02559_multiple_read_steps_in_prewhere_missing_columns_2 +02559_nested_multiple_levels_default +02560_analyzer_materialized_view +02560_count_digits +02560_null_as_default +02560_quantile_min_max +02561_sorting_constants_and_distinct_crash +02564_date_format +02564_read_in_order_final_desc +02565_update_empty_nested +02567_and_consistency +02568_and_consistency +02568_array_map_const_low_cardinality +02572_max_intersections +02576_predicate_push_down_sorting_fix +02577_analyzer_array_join_calc_twice +02578_ipv4_codec_t64 +02580_like_substring_search_bug +02582_async_reading_with_small_limit +02584_range_ipv4 +02587_csv_big_numbers_inference +02591_bson_long_tuple +02662_sparse_columns_mutations_4 +02662_sparse_columns_mutations_5 +02668_column_block_number_with_projections +02674_and_consistency +02674_date_int_string_json_inference +02674_null_default_structure +02675_is_ipv6_function_fix +02676_distinct_reading_in_order_analyzer +02677_decode_url_component +02677_get_subcolumn_array_of_tuples +02677_grace_hash_limit_race +02679_explain_merge_tree_prewhere_row_policy +02680_instr_alias_for_position_case_insensitive +02690_subquery_identifiers +02691_multiple_joins_backtick_identifiers +02692_multiple_joins_unicode +02699_polygons_sym_difference_rollup +02700_regexp_operator +02705_grouping_keys_equal_keys +02705_projection_and_ast_optimizations_bug +02707_analyzer_nested_lambdas_types +02708_parallel_replicas_not_found_column +02709_generate_random_valid_decimals_and_bools +02709_storage_memory_compressed +02710_aggregation_nested_map_ip_uuid +02710_date_diff_aliases +02710_topk_with_empty_array +02711_trim_aliases +02713_array_low_cardinality_string +02713_ip4_uint_compare +02714_date_date32_in +02715_or_null +02716_int256_arrayfunc +02717_pretty_json +02719_aggregate_with_empty_string_key +02720_row_policy_column_with_dots +02724_function_in_left_table_clause_asof_join +02724_jit_logical_functions +02725_agg_projection_respect_PK +02725_alias_with_restricted_keywords +02725_cnf_large_check +02730_dictionary_hashed_load_factor_element_count +02731_auto_convert_dictionary_layout_to_complex_by_complex_keys +02731_in_operator_with_one_size_tuple +02733_distinct +02733_fix_distinct_in_order_bug_49622 +02733_sparse_columns_reload +02734_big_int_from_float_ubsan +02734_optimize_group_by +02734_sparse_columns_short_circuit +02735_array_map_array_of_tuples +02736_bit_count_big_int +02746_index_analysis_binary_operator_with_null +02751_match_constant_needle +02751_multiif_to_if_crash +02752_custom_separated_ignore_spaces_bug +02763_jit_compare_functions_nan +02763_mutate_compact_part_with_skip_indices_and_projections +02764_index_analysis_fix +02766_bitshift_with_const_arguments +02769_compare_functions_nan +02770_jit_aggregation_nullable_key_fix +02771_if_constant_folding +02771_jit_functions_comparison_crash +02771_log_faminy_truncate_count +02782_inconsistent_formatting_and_constant_folding +02782_values_null_to_lc_nullable +02783_date_predicate_optimizations +02784_move_all_conditions_to_prewhere_analyzer_asan +02784_projections_read_in_order_bug +02784_schema_inference_null_as_default +02785_summing_merge_tree_datetime64 +02786_transform_float +02787_transform_null +02789_functions_after_sorting_and_columns_with_same_names_bug +02789_jit_cannot_convert_column +02789_set_index_nullable_condition_bug +02790_keyed_hash_bug +02790_url_multiple_tsv_files +02795_full_join_assert_cast +02796_projection_date_filter_on_view +02797_aggregator_huge_mem_usage_bug +02797_transform_narrow_types +02799_transform_empty_arrays +02801_transform_nullable +02804_intersect_bad_cast +02806_cte_block_cannot_be_empty +02807_lower_utf8_msan +02807_math_unary_crash +02808_aliases_inside_case +02809_has_subsequence +02809_has_token +02809_prewhere_and_in +02810_initcap +02810_row_binary_with_defaults +02810_system_jemalloc_bins +02811_insert_schema_inference +02811_read_in_order_and_array_join_bug +02812_bug_with_unused_join_columns +02812_csv_date_time_with_comma +02812_large_varints +02812_subquery_operators +02813_any_value +02813_array_agg +02813_float_parsing +02813_func_today_and_alias +02813_system_licenses_base +02814_order_by_tuple_window_function +02815_alias_to_length +02815_empty_subquery_nullable_bug +02815_first_line +02815_fix_not_found_constants_col_in_block +02816_has_token_empty +02828_create_as_table_function_rename +02831_trash +02832_integer_type_inference +02832_transform_fixed_string_no_default +02833_array_join_columns +02833_sparse_columns_tuple_function +02834_sparse_columns_sort_with_limit +02835_nested_array_lowcardinality +02841_join_filter_set_sparse +02841_not_ready_set_join_on +02841_tuple_modulo +02841_with_clause_resolve +02843_date_predicate_optimizations_bugs +02845_domain_rfc_support_ipv6 +02845_join_on_cond_sparse +02845_prewhere_preserve_column +02861_filter_pushdown_const_bug +02861_interpolate_alias_precedence +02861_uuid_format_serialization +02862_uuid_reinterpret_as_numeric +02863_mutation_where_in_set_result_cache_pipeline_stuck_bug +02864_filtered_url_with_globs +02864_profile_event_part_lock +02864_test_ipv4_type_mismatch +02866_size_of_marks_skip_idx_explain +02867_null_lc_in_bug +02867_nullable_primary_key_final +02869_unicode_minus +02871_join_on_system_errors +02874_infer_objects_as_named_tuples +02874_parse_json_as_json_each_row_on_no_metadata +02875_final_invalid_read_ranges_bug +02875_json_array_as_string +02876_json_incomplete_types_as_strings_inference +02876_sort_union_of_sorted +02882_primary_key_index_in_function_different_types +02883_read_in_reverse_order_virtual_column +02884_interval_operator_support_plural_literal +02884_parallel_window_functions_bug +02884_virtual_column_order_by +02886_binary_like +02888_integer_type_inference_in_if_function +02890_partition_prune_in_extra_columns +02891_functions_over_sparse_columns +02893_bad_sample_view +02893_trash_optimization +02895_cast_operator_bug +02896_optimize_array_exists_to_has_with_date +02902_show_databases_limit +02903_bug_43644 +02907_filter_pushdown_crash +02910_nullable_enum_cast +02911_analyzer_remove_unused_projection_columns +02911_cte_invalid_query_analysis +02911_system_symbols +02912_group_array_sample +02913_sum_map_state +02915_analyzer_fuzz_1 +02916_analyzer_set_in_join +02917_transform_tsan +02919_ddsketch_quantile +02919_segfault_nullable_materialized_update +02920_fix_json_merge_patch +02920_unary_operators_functions +02921_bit_hamming_distance_big_int +02921_fuzzbits_with_array_join +02923_explain_expired_context +02923_join_use_nulls_modulo +02931_alter_materialized_view_query_inconsistent +02931_ubsan_error_arena_aligned_alloc +02932_set_ttl_where +02933_compare_with_bool_as_string +02933_ephemeral_mv +02935_ipv6_bit_operations +02935_ipv6_from_uint128_equality +02935_ipv6_from_uint128_one +02935_ipv6_from_uint128_two +02935_ipv6_from_uint128_with_bit_and +02935_ipv6_to_and_from_uint128 +02941_any_RESPECT_NULL_sparse_column +02943_create_query_interpreter_sample_block_fix +02943_exprs_order_in_group_by_with_rollup +02943_positional_arguments_bugs +02943_tokenbf_and_ngrambf_indexes_support_match_function +02943_use_full_text_skip_index_with_has_any +02943_variant_element +02945_blake3_msan +02946_literal_alias_misclassification +02946_merge_tree_final_split_ranges_by_primary_key +02947_dropped_tables_parts +02947_parallel_replicas_remote +02949_ttl_group_by_bug +02950_reading_array_tuple_subcolumns +02953_slow_create_view +02955_avro_format_zstd_encode_support +02955_sparkBar_alias_sparkbar +02956_fix_to_start_of_milli_microsecond +02956_format_constexpr +02959_system_database_engines +02961_sumMapFiltered_keepKey +02962_analyzer_const_in_count_distinct +02962_analyzer_constant_set +02962_max_joined_block_rows +02963_single_value_destructor +02965_projection_with_partition_pruning +02966_float32_promotion +02968_analyzer_join_column_not_found +02968_sumMap_with_nan +02969_functions_to_subcolumns_if_null +02970_generate_series +02971_functions_to_subcolumns_column_names +02971_functions_to_subcolumns_map +02974_analyzer_array_join_subcolumn +02974_if_with_map +02975_intdiv_with_decimal +02981_translate_fixedstring +02982_create_mv_inner_extra +02982_dont_infer_exponent_floats +02986_leftpad_fixedstring +02987_group_array_intersect +02989_group_by_tuple +02990_optimize_uniq_to_count_alias +02990_parts_splitter_invalid_ranges +02991_count_rewrite_analyzer +02992_all_columns_should_have_comment +02993_lazy_index_loading +02994_cosineDistanceNullable +02996_index_compaction_counterexample +02997_fix_datetime64_scale_conversion +02998_pretty_format_print_readable_number_if_last_column +02998_primary_key_skip_columns +02998_system_dns_cache_table +03000_minmax_index_first +03000_virtual_columns_in_prewhere +03001_analyzer_nullable_nothing +03001_block_offset_column_2 +03001_data_version_column +03002_analyzer_prewhere +03002_map_array_functions_with_low_cardinality +03002_modify_query_cte +03002_sample_factor_where +03003_count_asterisk_filter +03003_enum_and_string_compatible +03003_sql_json_nonsense +03008_groupSortedArray_field +03008_index_small +03008_uniq_exact_equal_ranges +03009_range_dict_get_or_default +03010_read_system_parts_table_test +03010_sum_to_to_count_if_nullable +03010_view_prewhere_in +03010_virtual_memory_mappings_asynchronous_metrics +03011_adaptative_timeout_compatibility +03013_fuzz_arrayPartialReverseSort +03013_ignore_drop_queries_probability +03013_repeat_with_nonnative_integers +03014_analyzer_group_by_use_nulls +03015_analyzer_groupby_fuzz_60772 +03015_peder1001 +03016_analyzer_groupby_fuzz_59796 +03018_analyzer_greater_null +03023_analyzer_optimize_group_by_function_keys_with_nulls +03023_remove_unused_column_distinct +03031_input_format_allow_errors_num_bad_escape_sequence +03031_table_function_fuzzquery +03032_multi_search_const_low_cardinality +03032_numbers_zeros +03032_redundant_equals +03033_create_as_copies_comment +03033_cte_numbers_memory +03033_distinct_transform_const_columns +03033_final_undefined_last_mark +03033_from_unixtimestamp_joda_by_int64 +03033_scalars_context_data_race +03033_virtual_column_override +03033_with_fill_interpolate +03035_argMinMax_numeric_non_extreme_bug +03035_materialized_primary_key +03035_morton_encode_no_rows +03036_prewhere_lambda_function +03036_with_numbers +03037_dot_product_overflow +03037_union_view +03038_ambiguous_column +03038_move_partition_to_oneself_deadlock +03039_unknown_identifier_window_function +03040_alias_column_join +03040_array_sum_and_join +03041_analyzer_gigachad_join +03041_select_with_query_result +03043_group_array_result_is_expected +03044_array_join_columns_in_nested_table +03046_column_in_block_array_join +03047_analyzer_alias_join +03047_group_by_field_identified_aggregation +03048_not_found_column_xxx_in_block +03050_select_one_one_one +03051_many_ctes +03052_query_hash_includes_aliases +03054_analyzer_join_alias +03055_analyzer_subquery_group_array +03057_analyzer_subquery_alias_join +03064_analyzer_named_subqueries +03065_analyzer_cross_join_and_array_join +03066_analyzer_global_with_statement +03067_analyzer_complex_alias_join +03069_analyzer_with_alias_in_array_join +03070_analyzer_CTE_scalar_as_numbers +03072_analyzer_missing_columns_from_subquery +03075_analyzer_subquery_alias +03085_analyzer_alias_column_group_by +03086_analyzer_window_func_part_of_group_by +03087_analyzer_subquery_with_alias +03089_analyzer_alias_replacement +03090_analyzer_multiple_using_statements +03093_analyzer_column_alias +03093_analyzer_miel_test +03093_with_fill_support_constant_expression +03094_analyzer_fiddle_multiif +03094_named_tuple_bug24607 +03094_transform_return_first +03095_join_filter_push_down_right_stream_filled +03096_largest_triangle_3b_crash +03096_order_by_system_tables +03096_update_non_indexed_columns +03097_query_log_join_processes +03100_lwu_05_basics +03100_lwu_07_merge_patches +03100_lwu_08_multiple_blocks +03100_lwu_18_sequence +03100_lwu_19_nullable +03100_lwu_32_on_fly_filter +03100_lwu_33_add_column +03100_lwu_34_multistep_prewhere +03100_lwu_37_update_all_columns +03100_lwu_44_missing_default +03100_lwu_45_query_condition_cache +03101_analyzer_invalid_join_on +03102_prefer_column_name_to_alias +03104_create_view_join +03105_table_aliases_in_mv +03108_describe_union_all +03109_ast_too_big +03110_unicode_alias +03112_analyzer_not_found_column_in_block +03115_alias_exists_column +03116_analyzer_explicit_alias_as_column_name +03121_analyzer_filed_redefenition_in_subquery +03127_argMin_combinator_state +03128_merge_tree_index_lazy_load +03129_cte_with_final +03129_low_cardinality_nullable_non_first_primary_key +03130_abs_in_key_condition_bug +03130_analyzer_array_join_prefer_column +03132_sqlancer_union_all +03142_skip_ANSI_in_UTF8_compute_width +03142_window_function_limit_by +03143_join_filter_push_down_filled_join_fix +03143_ttl_in_system_parts_columns_table +03144_aggregate_states_with_different_types +03144_alter_column_and_read +03145_unicode_quotes +03146_bug47862 +03146_tpc_ds_grouping +03149_analyzer_join_projection_name +03149_analyzer_join_projection_name_2 +03150_url_hash_non_constant_level +03151_analyzer_view_read_only_necessary_columns +03151_external_cross_join +03151_pmj_join_non_procssed_clash +03151_redundant_distinct_with_window +03155_datasketches_ubsan +03155_explain_current_transaction +03156_tuple_map_low_cardinality +03161_decimal_binary_math +03161_ipv4_ipv6_equality +03164_analyzer_rewrite_aggregate_function_with_if +03164_optimize_read_in_order_nullable +03165_distinct_with_window_func_crash +03167_fancy_quotes_off_by_one +03167_parametrized_view_with_cte +03168_cld2_tsan +03168_fuzz_multiIf_short_circuit +03169_cache_complex_dict_short_circuit_bug +03169_display_column_names_in_footer +03169_modify_column_data_loss +03169_optimize_injective_functions_inside_uniq_crash +03171_direct_dict_short_circuit_bug +03172_system_detached_tables_no_loop +03173_distinct_combinator_alignment +03174_multiple_authentication_methods_show_create +03175_sparse_and_skip_index +03195_group_concat_deserialization_fix +03196_max_intersections_arena_crash +03197_fix_parse_mysql_iso_date +03198_group_array_intersect +03198_h3_polygon_to_cells +03198_json_extract_more_types +03199_has_lc_fixed_string +03199_join_with_materialized_column +03200_subcolumns_join_use_nulls +03201_sumIf_to_countIf_return_type +03203_drop_detached_partition_all +03203_fill_missed_subcolumns +03203_multiif_and_where_2_conditions_old_analyzer_bug +03203_system_numbers_limit_and_offset_simple +03204_index_hint_fuzzer +03204_storage_join_optimize +03205_hashing_empty_tuples +03205_json_syntax +03205_system_sync_replica_format +03208_datetime_cast_losing_precision +03208_groupArrayIntersect_serialization +03208_multiple_joins_with_storage_join +03208_numbers_total_rows_approx +03208_uniq_with_empty_tuple +03209_functions_json_msan_fuzzer_issue +03210_lag_lead_inframe_types +03210_nested_short_circuit_functions_bug +03213_array_element_msan +03214_join_on_tuple_comparison_elimination_bug +03215_fix_get_index_in_tuple +03215_toStartOfWeek_with_dateTime64_fix +03215_udf_with_union +03217_fliter_pushdown_no_keys +03217_primary_index_memory_leak +03221_key_condition_bug +03221_refreshable_matview_progress +03222_ignore_nulls_query_tree_elimination +03224_trim_empty_string +03225_const_prewhere_non_ataptive +03227_test_sample_n +03228_join_to_rerange_right_table +03228_url_engine_response_headers +03229_query_condition_cache_in_operator +03230_anyHeavy_merge +03230_system_projections +03232_workload_create_and_drop +03236_create_query_ttl_where +03236_squashing_high_memory +03237_get_subcolumn_low_cardinality_column +03237_max_map_state_decimal_serialization +03238_analyzer_unknown_function +03239_if_constant_folding +03240_array_element_or_null_for_map +03240_cte_in_subquery +03241_view_block_structure +03242_view_block_structure +03243_array_join_lambda +03243_to_start_of_interval_aliases +03244_skip_index_in_final_query_part_of_pk +03244_skip_index_in_final_query_with_pk_rescan_extremes +03244_skip_index_in_final_query_with_pk_rescan_no_final_mark +03244_skip_index_in_final_query_with_pk_rescan_pk_subset +03245_ripemd160 +03245_views_and_filter_push_down_bug +03246_range_literal_replacement_works +03246_toStartOfInterval_date_timezone_bug +03247_json_extract_lc_nullable +03247_object_column_copy +03248_with_insert +03250_ephemeral_comment +03252_fill_missed_arrays +03254_project_lwd_respects_row_exists +03254_uniq_exact_two_level_negative_zero +03255_fix_sbstrings_logical_error +03257_reverse_sorting_key_simple +03258_multiple_array_joins +03258_old_analyzer_const_expr_bug +03258_quantile_exact_weighted_issue +03259_join_condition_executed_block_bug +03259_negate_key_overflow +03259_orc_date_out_of_range +03261_any_respect_camelCase_aliases +03261_minmax_indices_by_default_table_copy +03262_analyzer_materialized_view_in_with_cte +03262_const_adaptive_index_granularity +03263_analyzer_materialized_view_cte_nested +03267_join_swap_bug +03268_system_parts_index_granularity +03269_partition_key_not_in_set +03270_fix_column_modifier_write_order +03271_decimal_monotonic_day_of_week +03273_dynamic_pretty_json_serialization +03274_grace_hash_max_joined_block_size_rows_bug +03274_philipzucker +03274_squashing_transform_sparse_bug +03274_with_fill_dup_sort_bug +03275_matview_with_union +03275_subcolumns_in_primary_key_bug +03276_functions_to_subcolumns_lc +03276_index_empty_part +03276_merge_tree_index_lazy_load +03276_parquet_output_compression_level +03278_dateTime64_in_dateTime64_bug +03286_collation_locale_with_modifier +03286_format_datetime_timezones +03286_serialization_hint_system_columns +03287_format_datetime_mysqlfraction +03289_tuple_element_to_subcolumn +03290_final_collapsing +03290_final_replacing +03290_final_sample +03290_mix_engine_and_query_settings +03290_partial_arrayROCAUC_and_arrayAUCPR +03290_pr_non_replicated_in_subquery +03292_format_tty_friendly +03298_server_client_native_settings +03299_map_named_tuple +03299_pretty_squash +03301_is_ipv4_string +03301_subcolumns_in_mv +03302_any_enum_aggregation +03303_alias_inverse_order +03303_dynamic_in_not_xor +03304_pretty_fallback_to_vertical +03305_compressed_memory_eng_crash_reading_subcolumn +03307_parallel_hash_max_joined_rows +03310_aggregate_projection_count_nullable +03312_analyzer_unused_projection_fix +03312_sparse_column_tuple +03313_h3togeo_result_order +03314_analyzer_resolve_in_parent_scope_5 +03314_has_column_in_table_alias_column +03314_summing_merge_tree_final_not_found_column_in_block +03314_variant_rowbinary_file +03315_analyzer_correlated_subqueries +03315_array_join_scalar +03315_join_on_optimize_pass_alias +03317_index_hint_prewhere +03321_functions_to_subcolumns_skip_index +03323_bfloat16_least_supertype +03323_union_all_constants_bug +03328_formatting_assignment_expression +03352_distinct_sorted_bug +03352_lazy_column_filter_by_uint8 +03354_translate_crap +03356_analyzer_unused_scalar_subquery +03358_block_structure_match +03359_point_in_polygon_index +03363_constant_nullable_key +03363_function_keccak256 +03364_pretty_json_bool +03365_time_implicit_conversion +03365_time_time64_best_effort_parsing +03365_time_to_time64_conv_bug +03365_time64_casts +03366_with_fill_dag +03368_bfloat16_merge_join +03369_bfloat16_map +03370_join_identifiers +03371_nullable_tuple_string_comparison +03374_date_trunc_with_negatives +03375_bool_partition +03393_smallest_index_floating_point +03395_global_join_supported_kind +03399_divide_zero_or_null +03399_mapContains_functions +03399_sparse_grams +03401_remote_bool +03402_zero_streams_after_max_streams_to_max_threads_ratio +03403_read_in_order_streams_memory_usage +03404_bfloat16_insert_values +03404_dynamic_in_interval_bug +03404_geotoh3_input_order +03405_bool_array_to_fixed_strings +03406_reservoir_sample_self_merging +03407_parse_date_time_best_effort_unix_timestamp_with_fraction +03411_analyzer_scalar_correlated_subquery +03411_iceberg_bucket +03414_formatDateTime_compound_formatter_after_varsize_formatter +03415_dont_highlight_probable_hashes +03443_index_match_alternatives +03444_lm_block_mismatch +03447_float_nan_order +03447_function_reverse_for_tuple +03447_grouping_sets_analyzer_const_columns +03447_window_functions_distinct +03448_topk_merging +03449_window_cannot_find_column +03450_parameterized_view_forward +03451_parameterized_views_without_alias +03454_parameterized_view_constant_identifier +03454_parameterized_views_null +03458_numeric_indexed_vector_operations_i8f64 +03458_numeric_indexed_vector_operations_u16f64 +03458_numeric_indexed_vector_operations_u32f64 +03458_numeric_indexed_vector_operations_u32i16 +03458_numeric_indexed_vector_operations_u32i32 +03458_numeric_indexed_vector_operations_u32i64 +03458_numeric_indexed_vector_operations_u32i8 +03458_numeric_indexed_vector_operations_u32u16 +03458_numeric_indexed_vector_operations_u32u32 +03458_numeric_indexed_vector_operations_u32u64 +03458_numeric_indexed_vector_operations_u32u8 +03459_join_cannot_add_column +03459_numeric_indexed_vector_decode +03459_socket_asynchronous_metrics +03460_numeric_indexed_vector_to_value_map +03461_numeric_indexed_vector_chain +03462_numeric_indexed_vector_serialization +03509_stripe_log_compatible_types +03511_formatDateTime_e_space_padding +03513_filter_push_down_rand_bug +03513_resize_pipeline_after_totals +03515_array_join_different_sizes +03516_int_exp2_join +03518_left_to_cross_incorrect +03519_fulter_push_down_duplicate_column_name_bug +03519_left_to_cross_incorrect +03520_left_to_cross_incorrect +03521_bitNot_String_NUL_terminated +03521_system_unicode +03521_tuple_of_dynamic_with_string_comparison +03532_divideOrNull_jit_crash +03532_dynamic_column_inside_map_rollback +03538_higher_order_functions_null_filter +03545_array_join_index_set_bug +03545_map_contains_bloom_index_bug +03547_analyzer_correlated_subqueries +03548_optimize_syntax_fuse_functions_clash +03549_conv_function +03549_system_dimensional_metrics +03551_cast_decimal_to_float +03555_inconsistent_formatting_ttl +03562_json_date_as_integer +03563_coarser_minmax_indexes_first +03565_clickhouse_smaller_indexes_first +03567_json_extract_case_insensitive_edge_cases +03568_ddsketch_merge +03568_json_extract_case_insensitive +03571_join_inequality_constants +03572_planner_merge_filter_into_join_bug +03573_concurrent_hash_scatter_bug +03573_planner_merge_filter_into_join_bug_2 +03574_parallel_replicas_last_right_join +03575_analyzer_merge_filter_into_join_bug_2 +03577_hash_output_format +03579_system_columns_column_alias +03580_join_runtime_filter_column_type +03581_bool_literal_column_name +03582_initcap_fixedstring +03582_normalize_utf8_empty +03595_alter_if_exists_runtime_check +03596_parquet_prewhere_page_skip_bug +03599_bad_date_and_datetimes_inference +03600_analyzer_setting_bool +03600_replace_fixed_string_bug +03601_histogram_quantile +03601_insert_squashing_remove_const +03601_replace_regex_fixedstring_empty_needle +03602_query_system_tables_definer +03604_functions_to_subcolumns_outer_join +03604_join_reorder_pinned_bug +03604_plan_step_description_limit +03611_cte_deterministic +03613_empty_tuple_permute_with_limit +03622_generic_aggregate_functions__state_compatibility +03623_datetime64_preepoch_fractional_precision +03623_parquet_bool +03624_parquet_row_number +03624_pr_lefl_right_joins_chain +03624_resource_exhaustion_window_function +03625_case_without_condition_non_constant_branches +03625_prewhere-and-default-bug +03625_upper_lower_utf8_different_number_of_code_points +03626_case_function_with_dynamic_argument +03627_non_constant_replacement_in_replace_regexp +03628_parse_date_time_short_circuit +03629_starts_endswith_caseinsensitive +03630_hash_join_max_block_size +03630_parquet_bool_bug +03631_select_replace_comprehensive +03632_join_logical_assert_85403 +03633_negative_limit_offset +03633_set_index_bulk_filtering +03635_in_function_different_types_many_columns +03636_index_analysis_with_session_tz +03639_hash_of_dynamic_column +03640_skip_indexes_with_or_and_not +03642_column_ttl_sparse +03643_paste_join_disable_filter_pushdown +03644_join_order_mixed_comma_and_left +03644_min_level_for_wide_part +03644_rows_before_aggregation_in_order +03651_positional_argument_agg_projection +03652_join_using_legacy_step +03653_fractional_limit_offset +03654_case_non_constant_null +03654_grouping_sets_any_min_max +03656_nan_comparison +03657_gby_overflow_any_sparse +03657_rollup_constant +03660_udf_subquery +03664_parameterized_view_restart +03666_count_matches_complexity +03671_dict_in_subquery_in_index_analysis_context_expired +03672_columns_same_as_subcolumns +03672_nested_array_nested_tuple +03699_reverse_utf8 +03700_vertical_format_pretty_print_json +03701_distinct_but_no_group_by_projection_table_use_check +03702_encode_decode_memory_usage +03703_function_dict_get_keys_large +03704_default_empty_order_by +03704_fractional_limit_with_ties +03704_function_dict_get_keys_cache_type +03705_count_if_asterisk +03705_fix_compression_T64_unaligned +03705_function_dict_get_keys_multiple_dict_and_no_caching +03708_exact_rows_before_limit_in +03708_flush_async_insert_queue_for_table +03708_join_or_to_right_any_bug +03709_coalescing_final +03710_midpoint_jit +03713_group_by_injective_function_old_analyzer +03714_base32_base58_short_string +03714_queries_escaping_1 +03714_queries_escaping_2 +03716_join_duplicate_columns_89411 +03716_join_right_side_sorting +03717_system_unicode_enums +03719_generic_hash_over_constant_and_non_constant +03719_ntile_no_partition_by_check +03720_datetime64_bad_inference +03720_insert_mem_no_self_deduplication +03721_join_residual_condition_bug_88635 +03721_right_join_logical_step +03723_incorrect_implicit_projection +03724_to_date_time_or_null_negative_arg_bug +03727_block_structure_mismatch_after_filter_push_down +03727_ipv4_parsing_bug +03727_named_tuples_pretty_format +03727_tolowcardinality_nullable_cast +03728_analyzer_identifier_resolution_join +03728_explain_column_structure +03733_base58_decode_bug +03740_alter_modify_query_dict_name_in_cse +03742_lazy_materialization_of_array_after_alter_add_column +03742_test_flattened_crash +03748_tuple_of_sparse_elements_bug +03749_cross_join_use_nulls_matcher +03749_implicit_index_ephemeral_alias +03751_join_empty_string_nullable +03752_fractional_limit_offset_small_blocks +03752_join_part +03754_fractional_limit_offset_multiple_streams +03754_h3_polygon_to_cells_const +03755_concurrent_hash_join_dispatch_bug +03756_sparse_serialization_nullable_in_tuple +03757_ast_not_formatting +03760_consistent-in-formatting +03760_join_reorder_outer_inner_nulls +03761_count_distinct_optimization_window +03761_log_with_string_size +03762_count_distinct_optimization_multiple_columns +03763_no_new +03764_join_nothing_type_column_crash +03771_tokens_crash +03773_join_on_formatting +03773_lightweight_update_index_subquery +03773_nullable_sparse_join +03773_parquet_roundtrip_bug +03777_join_runtime_filter_with_consts +03779_lexer_pointer_overflow_ubsan +03784_msan_token_iterator +03789_right_join_column_replicated +03791_decimal_string_zero +03793_logical_expressions_optimizer_group_by_use_nulls diff --git a/tests/clickhouse-test-runner/src/backends/client.ts b/tests/clickhouse-test-runner/src/backends/client.ts index d30941db6..da63c7593 100644 --- a/tests/clickhouse-test-runner/src/backends/client.ts +++ b/tests/clickhouse-test-runner/src/backends/client.ts @@ -1,72 +1,39 @@ -import { randomUUID } from "node:crypto"; -import { createClient } from "@clickhouse/client"; -import type { ParsedArgs } from "../args.js"; import { appendLog } from "../log.js"; +import { + type BackendOptions, + buildClickHouseSettings, + createSessionClient, + settleExpectedError, +} from "./shared.js"; -export interface BackendOptions { - args: ParsedArgs; - queries: string[]; - logPath: string; -} - -function buildClickHouseSettings( - args: ParsedArgs, -): Record { - const settings: Record = {}; - settings["default_format"] = "TabSeparated"; - if (args.logComment !== null && args.logComment.length > 0) { - settings["log_comment"] = args.logComment; - } - if (args.sendLogsLevel !== null && args.sendLogsLevel.length > 0) { - settings["send_logs_level"] = args.sendLogsLevel; - } - if (args.maxInsertThreads !== null && args.maxInsertThreads.length > 0) { - settings["max_insert_threads"] = args.maxInsertThreads; - } - for (const [k, v] of Object.entries(args.serverSettings)) { - settings[k] = v; - } - return settings; -} - +/** + * Passthrough backend (the default): stream ClickHouse's own `TabSeparated` + * output straight to stdout. The client only transports bytes; ClickHouse does + * all the formatting. This is what `default_format = TabSeparated` (in + * {@link buildClickHouseSettings}) selects. + */ export async function executeWithClient(opts: BackendOptions): Promise { - const { args, queries, logPath } = opts; - const proto = args.secure ? "https" : "http"; - const url = `${proto}://${args.host}:${args.port}`; - // Use a dedicated per-invocation session_id so that settings applied via - // `SET ...` in one statement persist for subsequent statements within the - // same .sql script. Without a session, every `client.exec(...)` call is an - // independent HTTP request and `SET` has no effect on later requests, which - // breaks upstream tests that rely on patterns like - // SET allow_deprecated_syntax_for_merge_tree = 1; - // CREATE TABLE ... ENGINE = MergeTree(d, k, 8192); - const sessionId = `clickhouse-js-test-runner-${randomUUID()}`; - appendLog(logPath, "session_id=" + sessionId); - const client = createClient({ - url, - username: args.user, - password: args.password, - database: args.database, - session_id: sessionId, - }); - + const { args, statements, logPath } = opts; + const client = createSessionClient(args, logPath); const clickhouse_settings = buildClickHouseSettings(args); try { - for (const q of queries) { - appendLog(logPath, "executing_query=" + q); - const result = await client.exec({ - query: q, - clickhouse_settings, - }); - for await (const chunk of result.stream) { - process.stdout.write(chunk); + for (const stmt of statements) { + appendLog(logPath, "executing_query=" + stmt.sql); + let execError: unknown = null; + try { + const result = await client.exec({ + query: stmt.sql, + clickhouse_settings, + }); + for await (const chunk of result.stream) { + process.stdout.write(chunk); + } + } catch (err) { + execError = err; } + settleExpectedError(stmt, execError, logPath); } - } catch (err) { - const msg = err instanceof Error ? err.message : String(err); - appendLog(logPath, "error=" + msg); - throw err; } finally { await client.close(); } diff --git a/tests/clickhouse-test-runner/src/backends/rowbinary.ts b/tests/clickhouse-test-runner/src/backends/rowbinary.ts new file mode 100644 index 000000000..bb143f4be --- /dev/null +++ b/tests/clickhouse-test-runner/src/backends/rowbinary.ts @@ -0,0 +1,152 @@ +import { compileRowBinaryWithNamesAndTypes } from "@clickhouse/rowbinary/rowBinaryWithNamesAndTypes"; +import { Cursor } from "@clickhouse/rowbinary/core"; +import { compileRowRenderers } from "../tsv-serialize.js"; +import { appendLog } from "../log.js"; +import { + type BackendOptions, + buildClickHouseSettings, + createSessionClient, + settleExpectedError, +} from "./shared.js"; + +/** + * RowBinary backend: instead of letting ClickHouse format the result text, ask + * for `RowBinaryWithNamesAndTypes`, decode it with `@clickhouse/rowbinary`'s + * dynamic header→reader path, and re-render the rows as `TabSeparated` so the + * upstream `clickhouse-test` diff against the static `.reference` still applies. + * That round-trip is what actually exercises (and proves) the parser end-to-end + * on real upstream queries. + * + * Only statements that return a result set AND have no explicit `FORMAT` clause + * take the decode path; DDL / INSERT / `SET` / explicit-`FORMAT` statements fall + * through to plain passthrough (ClickHouse formats them, exactly as the default + * backend does). A decode or render error is NOT swallowed — it surfaces as a + * failed statement so the test stays off the rowbinary allowlist rather than + * silently passing through an unexercised path. Error-hinted statements + * (`-- { serverError ... }`) are reconciled with {@link settleExpectedError}, + * exactly like the passthrough backend. + */ +export async function executeWithRowBinary( + opts: BackendOptions, +): Promise { + const { args, statements, logPath } = opts; + const client = createSessionClient(args, logPath); + const clickhouse_settings = buildClickHouseSettings(args); + + try { + for (const stmt of statements) { + const decode = shouldDecode(stmt.sql); + appendLog( + logPath, + (decode ? "rowbinary_query=" : "passthrough_query=") + stmt.sql, + ); + let execError: unknown = null; + try { + if (decode) { + await runDecoded(stmt.sql); + } else { + const result = await client.exec({ + query: stmt.sql, + clickhouse_settings, + }); + for await (const chunk of result.stream) { + process.stdout.write(chunk); + } + } + } catch (err) { + execError = err; + } + settleExpectedError(stmt, execError, logPath); + } + } finally { + await client.close(); + } + + /** Issue `q` as RowBinaryWithNamesAndTypes, decode, render TSV to stdout. */ + async function runDecoded(q: string): Promise { + const result = await client.exec({ + query: `${q}\nFORMAT RowBinaryWithNamesAndTypes`, + clickhouse_settings, + }); + // `result.stream` yields Uint8Array chunks; Buffer.concat accepts them and + // returns a Buffer (what Cursor needs), so no per-chunk cast is required. + const chunks: Uint8Array[] = []; + for await (const chunk of result.stream) { + chunks.push(chunk); + } + const buf = Buffer.concat(chunks); + // A statement that produced no result set at all (e.g. a misclassified + // no-output statement) yields zero bytes — nothing to decode or print. + if (buf.length === 0) return; + + const cursor = new Cursor(buf); + const { types, columnReaders } = compileRowBinaryWithNamesAndTypes(cursor); + const renderers = compileRowRenderers(types); + // Decode POSITIONALLY, column reader by column reader, rather than via the + // name-keyed row objects from `readRows`: a Record collapses duplicate + // column names (`SELECT 1 AS x, 2 AS x`) and reorders integer-like names + // (`SELECT 1 AS \`0\``), either of which would misalign cells against the + // header. The cursor sits at the first row after the header; the response is + // fully buffered, so we read complete rows until the bytes are exhausted. + const n = columnReaders.length; + const lines: string[] = []; + while (cursor.pos < buf.length) { + const cells = new Array(n); + for (let i = 0; i < n; i++) { + cells[i] = renderers[i]!(columnReaders[i]!(cursor)); + } + lines.push(cells.join("\t")); + } + // Join once (a row is terminated by \n, so the block ends with one too) + // rather than growing a string per row. + if (lines.length > 0) process.stdout.write(lines.join("\n") + "\n"); + } +} + +/** Statements that return a result set whose default text format is TabSeparated. */ +const RESULT_KEYWORDS = new Set([ + "SELECT", + "WITH", + "SHOW", + "DESC", + "DESCRIBE", + "EXISTS", + "EXPLAIN", + "VALUES", +]); + +/** + * True when `stmt` should go through the RowBinary decode path: it returns rows + * and carries no explicit `FORMAT` clause (which would fix its own output text, + * and whose `.reference` is in that other format). + */ +export function shouldDecode(stmt: string): boolean { + if (hasExplicitFormat(stmt)) return false; + const kw = leadingKeyword(stmt); + return kw !== null && RESULT_KEYWORDS.has(kw); +} + +/** A trailing/explicit `FORMAT ` clause (also matches INSERT ... FORMAT). */ +function hasExplicitFormat(stmt: string): boolean { + return /\bFORMAT\s+[A-Za-z0-9_]+/i.test(stmt); +} + +/** + * The leading SQL keyword, upper-cased — skipping leading line/block comments, + * whitespace and an optional opening parenthesis (`(SELECT ...) ...`). Returns + * null if no leading word can be found. + */ +function leadingKeyword(stmt: string): string | null { + let s = stmt; + // Strip leading comments and whitespace, possibly several in a row. + for (;;) { + const before = s; + s = s.replace(/^\s+/, ""); + s = s.replace(/^--[^\n]*\n?/, ""); + s = s.replace(/^\/\*[\s\S]*?\*\//, ""); + if (s === before) break; + } + s = s.replace(/^\(+\s*/, ""); + const m = /^([A-Za-z_]+)/.exec(s); + return m ? m[1]!.toUpperCase() : null; +} diff --git a/tests/clickhouse-test-runner/src/backends/shared.ts b/tests/clickhouse-test-runner/src/backends/shared.ts new file mode 100644 index 000000000..e92d2e390 --- /dev/null +++ b/tests/clickhouse-test-runner/src/backends/shared.ts @@ -0,0 +1,106 @@ +import { randomUUID } from "node:crypto"; +import { + ClickHouseLogLevel, + createClient, + type ClickHouseClient, +} from "@clickhouse/client"; +import type { ParsedArgs } from "../args.js"; +import { appendLog } from "../log.js"; +import { errorMatchesExpectation, type Statement } from "../test-hint.js"; + +export interface BackendOptions { + args: ParsedArgs; + statements: Statement[]; + logPath: string; +} + +/** Server settings to forward, assembled the same way for every backend. */ +export function buildClickHouseSettings( + args: ParsedArgs, +): Record { + const settings: Record = {}; + settings["default_format"] = "TabSeparated"; + if (args.logComment !== null && args.logComment.length > 0) { + settings["log_comment"] = args.logComment; + } + if (args.sendLogsLevel !== null && args.sendLogsLevel.length > 0) { + settings["send_logs_level"] = args.sendLogsLevel; + } + if (args.maxInsertThreads !== null && args.maxInsertThreads.length > 0) { + settings["max_insert_threads"] = args.maxInsertThreads; + } + for (const [k, v] of Object.entries(args.serverSettings)) { + settings[k] = v; + } + return settings; +} + +/** + * Create a client bound to a dedicated per-invocation `session_id` so that + * settings applied via `SET ...` in one statement persist for subsequent + * statements within the same .sql script. Without a session, every + * `client.exec(...)` is an independent HTTP request and `SET` has no effect on + * later requests, which breaks upstream tests that rely on patterns like + * SET allow_deprecated_syntax_for_merge_tree = 1; + * CREATE TABLE ... ENGINE = MergeTree(d, k, 8192); + */ +export function createSessionClient( + args: ParsedArgs, + logPath: string, +): ClickHouseClient { + const proto = args.secure ? "https" : "http"; + const url = `${proto}://${args.host}:${args.port}`; + const sessionId = `clickhouse-js-test-runner-${randomUUID()}`; + appendLog(logPath, "session_id=" + sessionId); + return createClient({ + url, + username: args.user, + password: args.password, + database: args.database, + session_id: sessionId, + // The client logs request errors to stderr by default. We surface failures + // ourselves (and deliberately swallow errors matched by a `-- { serverError + // ... }` hint), and upstream `clickhouse-test` fails any test that writes to + // stderr — so keep the client itself silent. + log: { level: ClickHouseLogLevel.OFF }, + }); +} + +/** + * Reconcile the outcome of a single statement against its expected-error hint, + * the same way for every backend: + * - hinted (`-- { serverError ... }`): an error is the success path, its + * absence (or a mismatching error) is a failure; + * - un-hinted: any error propagates. + * Throws on failure; returns normally when the statement is considered passed. + */ +export function settleExpectedError( + stmt: Statement, + execError: unknown, + logPath: string, +): void { + const expected = stmt.expectedError; + if (expected !== null) { + if (execError === null) { + throw new Error( + `Expected error (${expected.label}) but the query succeeded: ${stmt.sql}`, + ); + } + if (!errorMatchesExpectation(execError, expected)) { + const actual = + execError instanceof Error ? execError.message : String(execError); + throw new Error( + `Expected error (${expected.label}) but got a different error: ${actual}`, + ); + } + appendLog(logPath, "expected_error_matched=" + expected.label); + return; + } + + if (execError !== null) { + const msg = + execError instanceof Error ? execError.message : String(execError); + appendLog(logPath, "error=" + msg); + throw execError; + } +} diff --git a/tests/clickhouse-test-runner/src/main.ts b/tests/clickhouse-test-runner/src/main.ts index 7c208f49d..816a9474c 100644 --- a/tests/clickhouse-test-runner/src/main.ts +++ b/tests/clickhouse-test-runner/src/main.ts @@ -3,8 +3,10 @@ import { readFileSync } from "node:fs"; import { parseArgs, printUsage } from "./args.js"; import { appendLog, resolveLogPath, safeForLog } from "./log.js"; import { splitQueries } from "./split-queries.js"; +import { buildStatements, type Statement } from "./test-hint.js"; import { handleExtractFromConfig } from "./extract-from-config.js"; import { executeWithClient } from "./backends/client.js"; +import { executeWithRowBinary } from "./backends/rowbinary.js"; async function main(): Promise { const argv = process.argv.slice(2); @@ -51,12 +53,14 @@ async function main(): Promise { return; } - const queries = args.multiquery ? splitQueries(query) : [query.trim()]; - if (queries.length === 0) { - process.stderr.write( - "No query provided. Use --query or pipe SQL via stdin.\n", - ); - process.exitCode = 1; + const statements: Statement[] = args.multiquery + ? buildStatements(splitQueries(query)) + : [{ sql: query.trim(), expectedError: null }]; + if (statements.length === 0) { + // The input contained only comments/whitespace (e.g. a leftover error + // hint after the final statement). Nothing to run; exit cleanly like the + // native client would. + appendLog(logPath, "no_executable_statements=true"); return; } @@ -68,13 +72,35 @@ async function main(): Promise { appendLog(logPath, "send_logs_level=" + safeForLog(args.sendLogsLevel)); appendLog(logPath, "max_insert_threads=" + safeForLog(args.maxInsertThreads)); appendLog(logPath, "server_settings=" + JSON.stringify(args.serverSettings)); - appendLog(logPath, "queries_count=" + String(queries.length)); - for (const q of queries) { - appendLog(logPath, "query=" + q); + appendLog(logPath, "queries_count=" + String(statements.length)); + for (const stmt of statements) { + appendLog(logPath, "query=" + stmt.sql); + if (stmt.expectedError !== null) { + appendLog(logPath, "expected_error=" + stmt.expectedError.label); + } + } + + // Backend selection is via env (the upstream runner controls argv): the + // RowBinary backend exercises the @clickhouse/rowbinary decode path; the + // default passthrough backend streams ClickHouse's own TabSeparated text. + // Reject an unknown value rather than silently falling back to passthrough, + // which would hide a typo'd TEST_RUNNER_BACKEND in CI. + const backend = process.env["TEST_RUNNER_BACKEND"] ?? "passthrough"; + if (backend !== "passthrough" && backend !== "rowbinary") { + process.stderr.write( + `Error: unknown TEST_RUNNER_BACKEND "${backend}" (expected "passthrough" or "rowbinary")\n`, + ); + process.exitCode = 1; + return; } + appendLog(logPath, "backend=" + backend); try { - await executeWithClient({ args, queries, logPath }); + if (backend === "rowbinary") { + await executeWithRowBinary({ args, statements, logPath }); + } else { + await executeWithClient({ args, statements, logPath }); + } } catch (err) { const msg = err instanceof Error ? err.message : String(err); appendLog(logPath, "error=" + msg); diff --git a/tests/clickhouse-test-runner/src/test-hint.ts b/tests/clickhouse-test-runner/src/test-hint.ts new file mode 100644 index 000000000..f4fee2ebe --- /dev/null +++ b/tests/clickhouse-test-runner/src/test-hint.ts @@ -0,0 +1,177 @@ +// Support for upstream `clickhouse-test` error hints. +// +// Many `.sql` tests in ClickHouse/ClickHouse annotate a statement that is +// *expected to fail* with a trailing comment, e.g.: +// +// SELECT throwIf(1); -- { serverError FUNCTION_THROW_IF_VALUE_IS_NON_ZERO } +// SELECT * FROM does_not_parse(; -- { clientError SYNTAX_ERROR } +// SELECT 1 / 0; -- { serverError 153, 6 } +// +// The real `clickhouse-client` parses these hints (see upstream +// `src/Client/TestHint.cpp`) and, when the query raises the expected error, +// treats it as a pass and emits no output. Our Node shim must do the same, +// otherwise the propagated `ClickHouseError` aborts the script and the test is +// reported as a failure even though the server behaved exactly as expected. +// +// Upstream rule we mirror: an error hint is only honored when it *trails* the +// statement it refers to — hints in a leading comment are ignored. Because our +// statement splitter cuts on `;`, the trailing comment of statement N ends up +// as the leading comment of statement N+1 (or as a standalone comment-only +// final element). We therefore attach a leading hint to the *previous* +// statement, which reproduces the upstream "trailing comment" semantics. + +/** A set of error codes/names a query is expected to fail with. */ +export interface ExpectedError { + /** Human-readable hint, e.g. `serverError SIZES_OF_ARRAYS_DONT_MATCH`. */ + label: string; + /** Numeric error codes (as strings), matched against `ClickHouseError.code`. */ + codes: Set; + /** Named error codes, matched against `ClickHouseError.type`. */ + names: Set; +} + +/** A single statement to run, with an optional expected-error annotation. */ +export interface Statement { + sql: string; + expectedError: ExpectedError | null; +} + +const ERROR_COMMANDS = new Set(["serverError", "clientError", "error"]); + +/** + * Parse the contents of a single hint comment (the text *including* the comment + * markers). Returns the expected error if the comment carries a + * `serverError` / `clientError` / `error` directive, otherwise `null`. + * + * We deliberately merge `serverError` and `clientError` codes into one + * acceptance set: over HTTP every failure surfaces as a server error, so a + * `clientError` hint (which upstream attributes to the native client's local + * parsing) is still satisfied when the matching code comes back from the + * server. + */ +export function parseHintComment(comment: string): ExpectedError | null { + const open = comment.indexOf("{"); + if (open === -1) return null; + const close = comment.indexOf("}", open + 1); + if (close === -1) return null; + + const inner = comment.slice(open + 1, close); + const tokens = inner.split(/[\s,]+/).filter((t) => t.length > 0); + const cmdIndex = tokens.findIndex((t) => ERROR_COMMANDS.has(t)); + if (cmdIndex === -1) return null; + + const command = tokens[cmdIndex]; + const codes = new Set(); + const names = new Set(); + // Everything after the command keyword is a comma-separated list of codes, + // each either a numeric error code or a named one (e.g. SYNTAX_ERROR). + for (const token of tokens.slice(cmdIndex + 1)) { + if (/^\d+$/.test(token)) { + codes.add(token); + } else { + names.add(token); + } + } + if (codes.size === 0 && names.size === 0) return null; + + const label = `${command} ${tokens.slice(cmdIndex + 1).join(", ")}`; + return { label, codes, names }; +} + +/** + * Walk the leading run of whitespace and comments of a statement and return the + * first error hint found, or `null`. Scanning stops at the first non-comment + * token (where the SQL body begins), so genuine trailing hints — which live in + * the *next* split element — are not picked up here. + */ +function extractLeadingHint(statement: string): ExpectedError | null { + let i = 0; + const n = statement.length; + while (i < n) { + const ch = statement.charAt(i); + if (ch === " " || ch === "\t" || ch === "\n" || ch === "\r") { + i++; + continue; + } + if (ch === "-" && statement.charAt(i + 1) === "-") { + const newlineIdx = statement.indexOf("\n", i + 2); + const end = newlineIdx === -1 ? n : newlineIdx; + const hint = parseHintComment(statement.slice(i, end)); + if (hint) return hint; + i = end; + continue; + } + if (ch === "/" && statement.charAt(i + 1) === "*") { + const closeIdx = statement.indexOf("*/", i + 2); + const end = closeIdx === -1 ? n : closeIdx + 2; + const hint = parseHintComment(statement.slice(i, end)); + if (hint) return hint; + i = end; + continue; + } + break; // SQL body starts here + } + return null; +} + +/** Whether a split element contains any SQL beyond comments/whitespace. */ +function hasSqlContent(statement: string): boolean { + let i = 0; + const n = statement.length; + while (i < n) { + const ch = statement.charAt(i); + if (ch === " " || ch === "\t" || ch === "\n" || ch === "\r") { + i++; + continue; + } + if (ch === "-" && statement.charAt(i + 1) === "-") { + const newlineIdx = statement.indexOf("\n", i + 2); + i = newlineIdx === -1 ? n : newlineIdx; + continue; + } + if (ch === "/" && statement.charAt(i + 1) === "*") { + const closeIdx = statement.indexOf("*/", i + 2); + i = closeIdx === -1 ? n : closeIdx + 2; + continue; + } + return true; + } + return false; +} + +/** + * Turn the raw output of `splitQueries` into executable statements, attaching + * each trailing error hint to the statement it annotates. Comment-only + * elements (e.g. a hint left dangling after the final statement) carry their + * hint to the preceding statement and are otherwise dropped. + */ +export function buildStatements(rawQueries: string[]): Statement[] { + const statements: Statement[] = []; + for (const raw of rawQueries) { + const hint = extractLeadingHint(raw); + const previous = statements[statements.length - 1]; + if (hint && previous !== undefined) { + // The trailing hint of the previous statement (upstream semantics). + previous.expectedError = hint; + } + // A leading hint with no preceding statement is ignored, matching upstream. + if (hasSqlContent(raw)) { + statements.push({ sql: raw, expectedError: null }); + } + } + return statements; +} + +/** Whether an error raised by the client matches the expected-error hint. */ +export function errorMatchesExpectation( + err: unknown, + expected: ExpectedError, +): boolean { + if (typeof err !== "object" || err === null) return false; + const code = (err as { code?: unknown }).code; + const type = (err as { type?: unknown }).type; + if (typeof code === "string" && expected.codes.has(code)) return true; + if (typeof code === "number" && expected.codes.has(String(code))) return true; + if (typeof type === "string" && expected.names.has(type)) return true; + return false; +} diff --git a/tests/clickhouse-test-runner/src/tsv-serialize.ts b/tests/clickhouse-test-runner/src/tsv-serialize.ts new file mode 100644 index 000000000..eba457295 --- /dev/null +++ b/tests/clickhouse-test-runner/src/tsv-serialize.ts @@ -0,0 +1,337 @@ +/** + * Render decoded `RowBinaryWithNamesAndTypes` values back into the exact text + * ClickHouse produces for the `TabSeparated` format, so the upstream + * `clickhouse-test` runner can diff our output against the checked-in + * `.reference` files. + * + * This is the value-oracle half of the RowBinary backend: ClickHouse is the + * byte oracle (it produces the RowBinary), our parser decodes it, and this + * module re-serializes the decoded JS values. A faithful round-trip + * (RowBinary-decode → TSV-render == server's TSV) is what proves the dynamic + * header→reader path decoded every column correctly. + * + * The renderer is TYPE-DIRECTED: it walks the column's parsed data-type AST + * (from `@clickhouse/datatype-parser`, the same AST the parser folds into + * readers) alongside the decoded value, because the JS value alone is + * insufficient to reproduce ClickHouse's text — e.g. an `Enum8` decodes to its + * underlying integer but TSV prints the NAME, which lives only in the type. + * + * Two text contexts, mirroring ClickHouse's `serializeTextEscaped` (top level) + * vs `serializeTextQuoted` (inside Array/Tuple/Map): + * - top level: strings/dates/etc. are escaped but UNQUOTED; NULL is `\N`. + * - nested: the same values are wrapped in single quotes; NULL is `NULL`. + * Numbers, decimals and booleans are bare in both contexts. + * + * Types this v1 does not yet render (Variant, JSON, Dynamic, the geo types, + * Nested) throw {@link TSVRenderError}; the RowBinary backend treats that as a + * decode failure for the statement, so such tests stay off the rowbinary + * allowlist rather than silently passing through an unexercised path. + */ + +import { + parseDataType, + NodeKind, + type Node, +} from "@clickhouse/datatype-parser"; +import { formatDecimal } from "@clickhouse/rowbinary/decimals"; +import { formatTime, formatTime64 } from "@clickhouse/rowbinary/time"; +import { formatUUID } from "@clickhouse/rowbinary/uuid"; +import { formatIPv4, formatIPv6 } from "@clickhouse/rowbinary/ip"; + +/** Thrown when a column type has no TSV renderer yet (see module note). */ +export class TSVRenderError extends Error { + constructor(message: string) { + super(message); + this.name = "TSVRenderError"; + } +} + +/** Types whose text is unquoted+escaped at top level but single-quoted when nested. */ +function renderStringish(s: string, nested: boolean): string { + return nested ? `'${escapeQuoted(s)}'` : escapeRaw(s); +} + +// C-style escapes for the control characters ClickHouse escapes in text +// formats. Backslash itself is included so a SINGLE-pass `replace` is complete: +// escaping in one pass (rather than chaining `.replace`s) avoids re-processing +// the backslashes we introduce, and lets static analysis see that `\` is +// handled. +const RAW_ESCAPES: Record = { + "\\": "\\\\", + "\t": "\\t", + "\n": "\\n", + "\r": "\\r", + "\0": "\\0", +}; +const QUOTED_ESCAPES: Record = { ...RAW_ESCAPES, "'": "\\'" }; + +/** TabSeparated top-level escaping (`serializeTextEscaped`): backslash + delimiters. */ +function escapeRaw(s: string): string { + return s.replace(/[\\\t\n\r\0]/g, (c) => RAW_ESCAPES[c]!); +} + +/** Quoted escaping (`serializeTextQuoted`): as {@link escapeRaw} plus the single quote. */ +function escapeQuoted(s: string): string { + return s.replace(/[\\\t\n\r\0']/g, (c) => QUOTED_ESCAPES[c]!); +} + +/** ClickHouse's lower-case words for the non-finite floats and signed zero, else null. */ +function floatSpecial(n: number): string | null { + if (Number.isNaN(n)) return "nan"; + if (n === Infinity) return "inf"; + if (n === -Infinity) return "-inf"; + if (n === 0) return Object.is(n, -0) ? "-0" : "0"; + return null; +} + +/** + * Reconcile JS's number text with ClickHouse's: ClickHouse writes a positive + * exponent without the `+` (`3.4028235e38`, not JS's `3.4028235e+38`); negative + * exponents (`1e-10`) already match. + */ +function chFloatText(s: string): string { + return s.replace("e+", "e"); +} + +/** + * ClickHouse `Float64` text: the shortest decimal that round-trips to the + * double, which is exactly what `String(number)` produces in V8 (and what + * ClickHouse emits). + */ +function formatFloat(n: number): string { + return floatSpecial(n) ?? chFloatText(String(n)); +} + +/** + * ClickHouse `Float32` text. The parser widens a Float32 to a JS double, so + * `String(n)` would print the double's full precision (e.g. `0.2689400017261505` + * for a value ClickHouse prints as `0.26894`). ClickHouse instead emits the + * shortest decimal that round-trips to the *single*-precision value, so search + * increasing precisions for the shortest string whose `Math.fround` is `n`. + */ +function formatFloat32(n: number): string { + const special = floatSpecial(n); + if (special !== null) return special; + for (let p = 1; p < 9; p++) { + const candidate = Number(n.toPrecision(p)); + if (Math.fround(candidate) === n) return chFloatText(String(candidate)); + } + return chFloatText(String(n)); +} + +const pad2 = (n: number): string => String(n).padStart(2, "0"); + +/** `YYYY-MM-DD` from the UTC components of the parser's `Date` (days since epoch). */ +function formatDate(d: Date): string { + return `${d.getUTCFullYear()}-${pad2(d.getUTCMonth() + 1)}-${pad2(d.getUTCDate())}`; +} + +/** `YYYY-MM-DD HH:MM:SS` in UTC. Server tz is UTC, so tz-less columns match. */ +function formatDateTime(d: Date): string { + return `${formatDate(d)} ${pad2(d.getUTCHours())}:${pad2(d.getUTCMinutes())}:${pad2(d.getUTCSeconds())}`; +} + +/** + * `YYYY-MM-DD HH:MM:SS[.fff...]` for `DateTime64(P)`. The parser hands back + * `[Date (whole seconds), nanoseconds]`; the P-digit fraction is the + * nanoseconds scaled down to the column precision. + */ +function formatDateTime64(value: unknown, precision: number): string { + const [d, ns] = value as [Date, number]; + const base = formatDateTime(d); + if (precision <= 0) return base; + // TRUNCATE toward the precision, matching ClickHouse's text output — never + // round, which could nudge the fraction up to 10^precision and would then + // need to carry into the seconds field. + const frac = Math.trunc(ns / 10 ** (9 - precision)); + return `${base}.${String(frac).padStart(precision, "0")}`; +} + +/** A geo `Point` decodes to `[x, y]` (two Float64s) and renders as `(x,y)`. */ +function renderPoint(value: unknown): string { + const [x, y] = value as [number, number]; + return `(${formatFloat(x)},${formatFloat(y)})`; +} + +/** `[elem,elem,…]` for the array-of-Point geo nestings (Ring/Polygon/Multi*). */ +function renderPointArray( + value: unknown, + renderElem: (e: unknown) => string, +): string { + return `[${(value as unknown[]).map(renderElem).join(",")}]`; +} + +/** Map a decoded enum integer to its name via the explicit `'name' = value` pairs in the type. */ +function enumName(node: Node, value: unknown): string { + const v = BigInt(value as number); + for (const ev of node.values) { + if (ev.value === v) return ev.name; + } + throw new TSVRenderError( + `enum value ${String(value)} not found in ${node.name}`, + ); +} + +function requireArg(node: Node, index: number): Node { + const arg = node.arguments[index]; + if (arg === undefined) { + throw new TSVRenderError(`type ${node.name} is missing argument ${index}`); + } + return arg; +} + +function literalInt(node: Node): number { + if (node.kind !== NodeKind.Literal) { + throw new TSVRenderError(`expected a literal argument in ${node.name}`); + } + return Number(node.value); +} + +/** + * Render one decoded value as ClickHouse TSV text. `nested` selects the quoted + * (inside a composite) vs top-level serialization. Mirrors the type dispatch in + * the parser's `astToReader`, so the value shapes line up by construction. + */ +export function renderValue( + node: Node, + value: unknown, + nested: boolean, +): string { + // NULL is type-independent: only the surrounding context decides its text. + if (value === null || value === undefined) return nested ? "NULL" : "\\N"; + + if (node.kind === NodeKind.EnumDataType) { + return renderStringish(enumName(node, value), nested); + } + if (node.kind === NodeKind.TupleDataType) { + return renderTuple(node, value); + } + if (node.kind !== NodeKind.DataType) { + throw new TSVRenderError(`cannot render a ${node.kind} node`); + } + + switch (node.name) { + // --- transparent wrappers --- + case "Nullable": + case "LowCardinality": + return renderValue(requireArg(node, 0), value, nested); + + // --- composites (children always render in the nested/quoted context) --- + case "Array": + case "QBit": { + const elem = requireArg(node, 0); + return `[${(value as unknown[]).map((v) => renderValue(elem, v, true)).join(",")}]`; + } + case "Map": { + const keyT = requireArg(node, 0); + const valT = requireArg(node, 1); + const entries = [...(value as Map)].map( + ([k, v]) => + `${renderValue(keyT, k, true)}:${renderValue(valT, v, true)}`, + ); + return `{${entries.join(",")}}`; + } + + // --- geo: fixed nestings of Point (a Float64 pair). ClickHouse renders + // these the same in any context, so `nested` is irrelevant here. + case "Point": + return renderPoint(value); + case "Ring": + case "LineString": + return renderPointArray(value, renderPoint); + case "Polygon": + case "MultiLineString": + return renderPointArray(value, (r) => renderPointArray(r, renderPoint)); + case "MultiPolygon": + return renderPointArray(value, (poly) => + renderPointArray(poly, (r) => renderPointArray(r, renderPoint)), + ); + + // --- stringish (unquoted+escaped at top level, single-quoted when nested) --- + case "String": + case "FixedString": + return renderStringish(value as string, nested); + case "UUID": + return renderStringish(formatUUID(value as Buffer), nested); + case "IPv4": + return renderStringish(formatIPv4(value as number), nested); + case "IPv6": + return renderStringish(formatIPv6(value as Buffer), nested); + case "Date": + case "Date32": + return renderStringish(formatDate(value as Date), nested); + case "DateTime": + case "DateTime32": + return renderStringish(formatDateTime(value as Date), nested); + case "DateTime64": + return renderStringish( + formatDateTime64( + value, + node.arguments.length > 0 ? literalInt(node.arguments[0]!) : 3, + ), + nested, + ); + case "Time": + return renderStringish(formatTime(value as number), nested); + case "Time64": + return renderStringish( + formatTime64(value as readonly [bigint, number]), + nested, + ); + + // --- numeric / boolean (bare in both contexts) --- + case "Bool": + return value ? "true" : "false"; + case "Float32": + return formatFloat32(value as number); + case "Float64": + case "BFloat16": + return formatFloat(value as number); + case "Decimal": + case "Decimal32": + case "Decimal64": + case "Decimal128": + case "Decimal256": + return formatDecimal(value as readonly [bigint, number]); + + default: + // Integers (incl. 64/128/256-bit bigints) and Interval* are plain digits. + if ( + node.name.startsWith("Int") || + node.name.startsWith("UInt") || + node.name.startsWith("Interval") + ) { + return String(value); + } + throw new TSVRenderError(`no TSV renderer for type ${node.name}`); + } +} + +/** `(a,b,c)` — named (object) or positional (array) tuples both print positionally. */ +function renderTuple(node: Node, value: unknown): string { + const cells = node.arguments.map((field, i) => { + const name = node.element_names[i]; + const v = + name !== undefined && name.length > 0 && !Array.isArray(value) + ? (value as Record)[name] + : (value as unknown[])[i]; + return renderValue(field, v, true); + }); + return `(${cells.join(",")})`; +} + +/** Parse each column type string once and return a per-column top-level renderer. */ +export function compileRowRenderers( + types: string[], +): ((value: unknown) => string)[] { + return types.map((t) => { + const result = parseDataType(t); + if (!result.ok() || result.ast === null) { + throw new TSVRenderError( + `cannot parse column type ${JSON.stringify(t)}: ${result.error?.message ?? "unknown"}`, + ); + } + const node = result.ast; + return (value: unknown) => renderValue(node, value, false); + }); +} diff --git a/tests/clickhouse-test-runner/upstream-allowlist.txt b/tests/clickhouse-test-runner/upstream-allowlist.txt index c2210720f..43ba26aad 100644 --- a/tests/clickhouse-test-runner/upstream-allowlist.txt +++ b/tests/clickhouse-test-runner/upstream-allowlist.txt @@ -2534,7 +2534,10 @@ 03374_date_trunc_with_negatives 03375_bloom_filter_array_equals 03375_bool_partition -03380_accurate_cast_or_null_qbit +# 03380_accurate_cast_or_null_qbit — diverges on the `latest` released server: +# the experimental QBit `accurateCastOrNull` behavior differs from the upstream +# master `.reference` (passes on `head`). Pre-existing failure, unrelated to the +# RowBinary backend; re-enable once the released behavior matches the reference. 03389_regexp_rewrite_nullable_group_by 03392_crash_group_by_use_nulls 03393_smallest_index_floating_point diff --git a/tests/e2e/install/src/integration.ts b/tests/e2e/install/src/integration.ts index 905011c8e..7408cb427 100644 --- a/tests/e2e/install/src/integration.ts +++ b/tests/e2e/install/src/integration.ts @@ -7,33 +7,34 @@ // publish workflow's e2e job (which installs the package by its published // version and starts a single-node ClickHouse), so it validates the actual npm // tarball a consumer would receive, not the local build. +// +// NOTE: this file is run via `node src/integration.ts` across Node 20/22/24. +// Node 20 does not strip TypeScript types, so this must stay free of TS-only +// syntax (no type annotations, `as` casts, etc.) — plain JS in a .ts file, like +// src/index.ts. const assert = require("assert"); const { createClient, ClickHouseError } = require("@clickhouse/client"); -// The e2e job starts ClickHouse via docker-compose just before this runs; poll -// ping briefly so we don't race the container coming up. -async function waitForClickHouse(client: any) { - const maxAttempts = 30; - for (let attempt = 1; attempt <= maxAttempts; attempt++) { - try { - const res = await client.ping(); - if (res.success) return; - } catch { - // not ready yet - } - if (attempt === maxAttempts) { - throw new Error("ClickHouse did not become available in time"); - } - await new Promise((resolve) => setTimeout(resolve, 1000)); - } -} - async function main() { // Defaults target http://localhost:8123 with the default user, matching the // single-node `clickhouse` service from docker-compose.yml. const client = createClient(); try { - await waitForClickHouse(client); + // The e2e job starts ClickHouse via docker-compose just before this runs; + // poll ping briefly so we don't race the container coming up. + const maxAttempts = 30; + for (let attempt = 1; attempt <= maxAttempts; attempt++) { + try { + const res = await client.ping(); + if (res.success) break; + } catch { + // not ready yet + } + if (attempt === maxAttempts) { + throw new Error("ClickHouse did not become available in time"); + } + await new Promise((resolve) => setTimeout(resolve, 1000)); + } const table = `e2e_install_${Date.now()}`; await client.command({ @@ -71,7 +72,7 @@ async function main() { // A bad query must surface as a ClickHouseError instance from the SAME // installed package (a single bundle => one class identity). - let caught: unknown; + let caught; try { await client.query({ query: "SELECT * FROM table_that_does_not_exist_e2e", diff --git a/tests/e2e/web-browser/.gitignore b/tests/e2e/web-browser/.gitignore new file mode 100644 index 000000000..d5f19d89b --- /dev/null +++ b/tests/e2e/web-browser/.gitignore @@ -0,0 +1,2 @@ +node_modules +package-lock.json diff --git a/tests/e2e/web-browser/package.json b/tests/e2e/web-browser/package.json new file mode 100644 index 000000000..8229c125f --- /dev/null +++ b/tests/e2e/web-browser/package.json @@ -0,0 +1,18 @@ +{ + "name": "e2e-web-browser", + "version": "1.0.0", + "description": "Post-publish browser e2e for @clickhouse/client-web", + "private": true, + "type": "module", + "scripts": { + "test": "vitest run" + }, + "license": "Apache-2.0", + "devDependencies": { + "@types/node": "25.9.3", + "@vitest/browser-playwright": "^4.1.9", + "playwright": "^1.61.0", + "typescript": "^6.0.3", + "vitest": "^4.0.16" + } +} diff --git a/tests/e2e/web-browser/test/web.browser.test.ts b/tests/e2e/web-browser/test/web.browser.test.ts new file mode 100644 index 000000000..b622fd2ba --- /dev/null +++ b/tests/e2e/web-browser/test/web.browser.test.ts @@ -0,0 +1,60 @@ +import { describe, it, expect, beforeAll, afterAll } from "vitest"; +import { + createClient, + ClickHouseError, + type ClickHouseClient, +} from "@clickhouse/client-web"; + +// The publish workflow starts the single-node ClickHouse compose service before +// this runs. Its HTTP interface sends CORS headers (see +// .docker/clickhouse/single_node/config.xml -> ), so the +// browser can reach it cross-origin from the vitest page. +const url = "http://127.0.0.1:8123"; + +describe("[Web e2e] published @clickhouse/client-web in a real browser", () => { + let client: ClickHouseClient; + + beforeAll(() => { + client = createClient({ url }); + }); + + afterAll(async () => { + await client.close(); + }); + + it("pings the server", async () => { + const res = await client.ping(); + expect(res.success).toBe(true); + }); + + it("runs a simple query and reads JSON", async () => { + const rs = await client.query({ + // toUInt8 keeps the value an unquoted JSON number regardless of the + // server's 64-bit-integer quoting default. + query: "SELECT toUInt8(number) AS n FROM system.numbers LIMIT 3", + format: "JSONEachRow", + }); + expect(await rs.json()).toEqual([{ n: 0 }, { n: 1 }, { n: 2 }]); + }); + + it("streams rows", async () => { + const rs = await client.query({ + query: "SELECT number FROM system.numbers LIMIT 5", + format: "JSONEachRow", + }); + let streamed = 0; + for await (const rows of rs.stream()) { + streamed += rows.length; + } + expect(streamed).toBe(5); + }); + + it("surfaces a bad query as a ClickHouseError", async () => { + await expect( + client.query({ + query: "SELECT * FROM table_that_does_not_exist_e2e_web", + format: "JSONEachRow", + }), + ).rejects.toBeInstanceOf(ClickHouseError); + }); +}); diff --git a/tests/e2e/web-browser/vitest.config.ts b/tests/e2e/web-browser/vitest.config.ts new file mode 100644 index 000000000..f2f55f449 --- /dev/null +++ b/tests/e2e/web-browser/vitest.config.ts @@ -0,0 +1,28 @@ +import { defineConfig } from "vitest/config"; +import { playwright } from "@vitest/browser-playwright"; + +// Real-browser post-publish e2e for @clickhouse/client-web. The published +// package is installed into this project (see the publish workflow), and +// vitest's own Vite bundler serves it to a Playwright-driven browser — exactly +// how a bundler-based web consumer would load it. This is the right runtime to +// validate the Web client: a Node host (which has both `fetch` and `node:http`) +// would not catch a browser-incompatible regression. +const browser = process.env.BROWSER ?? "chromium"; + +export default defineConfig({ + test: { + include: ["test/**/*.test.ts"], + testTimeout: 60_000, + hookTimeout: 60_000, + browser: { + enabled: true, + provider: playwright(), + headless: true, + instances: [{ browser }], + }, + }, + // The published @clickhouse/client-web ships a CJS bundle; force Vite to + // pre-bundle it so its named exports are exposed to the browser ESM import + // (mirrors the main web suite's `dist` mode). + optimizeDeps: { include: ["@clickhouse/client-web"] }, +});