Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
176 changes: 176 additions & 0 deletions .github/workflows/publish-datatype-parser.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
name: "publish: datatype parser"

# Independent publish + release for the standalone @clickhouse/datatype-parser
# package (the data-type string parser). It is NOT part of the npm workspace
# lockstep release driven by publish.yml — it carries its own version in
# packages/datatype-parser/package.json and ships on its own cadence. Triggered
# manually, and — like publish.yml — must be dispatched from the `release`
# branch: the npm-publish environment is protected so only that branch may
# deploy (the repo's human-in-the-loop release gate). Dispatches from any other
# ref are skipped by the job-level `if` guard below.
#
# The release branch is itself protected, so the unit suite is not re-run here.
# The `publish` job instead builds, packs the tarball, installs that exact
# tarball into a throwaway project and smoke-tests its imports, and only then
# publishes the same tarball with the "latest" tag (npm OIDC + provenance) and
# pushes a matching git tag. The `e2e` job then repeats the smoke test against
# the freshly published version on the registry across the supported Node
# versions.

permissions:
contents: read
id-token: write # Required for npm OIDC authentication and provenance

concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false

on:
workflow_dispatch:

jobs:
publish:
# The npm-publish environment only permits the release branch to deploy;
# skip cleanly on any other ref instead of failing the protection check.
if: github.ref == 'refs/heads/release'
runs-on: ubuntu-latest
timeout-minutes: 10
environment: npm-publish
permissions:
contents: write # Required to push the release git tag
id-token: write # Required for npm OIDC authentication and provenance
defaults:
run:
working-directory: packages/datatype-parser
outputs:
version: ${{ steps.version.outputs.version }}
steps:
- name: Checkout repository
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3

- name: Setup Node.js
uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
with:
node-version: 24
registry-url: "https://registry.npmjs.org"

- name: Install dependencies
run: npm ci

- name: Build
run: npm run build

- name: Get the release version
id: version
run: |
VERSION=$(node -p "require('./package.json').version")
echo "Publishing @clickhouse/datatype-parser@$VERSION"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"

- name: Pack the tarball
id: pack
# prepack rebuilds dist before packing.
run: |
set -euo pipefail
TARBALL=$(npm pack --pack-destination "$RUNNER_TEMP" | tail -1)
echo "Packed: $TARBALL"
echo "tarball=$RUNNER_TEMP/$TARBALL" >> "$GITHUB_OUTPUT"

- name: Pre-publish smoke test (install the packed tarball)
env:
TARBALL: ${{ steps.pack.outputs.tarball }}
run: |
set -euo pipefail
work="$(mktemp -d)"
cd "$work"
npm init -y >/dev/null 2>&1
npm install "$TARBALL"
# Verify the main barrel entry resolves and exposes the parser, from
# the exact artifact we are about to publish.
node --input-type=module -e "
import * as dt from '@clickhouse/datatype-parser';
if (typeof dt.parseDataType !== 'function') throw new Error('parseDataType missing from main export');
console.log('OK: packed tarball imports cleanly');
"

- name: Publish to npm
# Publish the exact tarball that passed the pre-publish smoke test.
env:
TARBALL: ${{ steps.pack.outputs.tarball }}
run: npm publish "$TARBALL" --access public --provenance

- name: Create and push release git tag
env:
RELEASE_TAG: datatype-parser-v${{ steps.version.outputs.version }}
RELEASE_VERSION: ${{ steps.version.outputs.version }}
run: |
if git ls-remote --exit-code --tags origin "refs/tags/${RELEASE_TAG}" >/dev/null 2>&1; then
echo "Tag ${RELEASE_TAG} already exists on origin; skipping."
exit 0
fi
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git tag -a "${RELEASE_TAG}" -m "Release @clickhouse/datatype-parser ${RELEASE_VERSION}"
git push origin "refs/tags/${RELEASE_TAG}"

e2e:
name: e2e (node ${{ matrix.node }})
needs: publish
if: needs.publish.result == 'success'
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: true
matrix:
node: [20, 22, 24]
env:
PUBLISHED_VERSION: ${{ needs.publish.outputs.version }}
steps:
- name: Setup NodeJS ${{ matrix.node }}
uses: actions/setup-node@53b83947a5a98c8d113130e565377fae1a50d02f # v6.3.0
with:
node-version: ${{ matrix.node }}
registry-url: "https://registry.npmjs.org"

- name: Wait for @clickhouse/datatype-parser@${{ needs.publish.outputs.version }} on npm
run: |
set -euo pipefail
if [ -z "${PUBLISHED_VERSION}" ]; then
echo "PUBLISHED_VERSION is empty; cannot wait for npm publication." >&2
exit 1
fi
pkg="@clickhouse/datatype-parser"
# Poll the registry for up to ~5 minutes. New versions usually surface
# in seconds, but the registry CDN can lag.
max_attempts=60
sleep_seconds=5
attempt=1
echo "Waiting for ${pkg}@${PUBLISHED_VERSION} to be available on npm..."
while true; do
if npm view "${pkg}@${PUBLISHED_VERSION}" version >/dev/null 2>&1; then
echo " ${pkg}@${PUBLISHED_VERSION} is available."
break
fi
if [ "$attempt" -ge "$max_attempts" ]; then
echo "Timed out waiting for ${pkg}@${PUBLISHED_VERSION} on npm" >&2
exit 1
fi
echo " attempt ${attempt}/${max_attempts}: not available yet, sleeping ${sleep_seconds}s..."
attempt=$((attempt + 1))
sleep "$sleep_seconds"
done

- name: Install and import the published package
run: |
set -euo pipefail
work="$(mktemp -d)"
cd "$work"
npm init -y >/dev/null 2>&1
npm install "@clickhouse/datatype-parser@${PUBLISHED_VERSION}"
# Verify the main barrel entry resolves and exposes the parser to a
# downstream consumer.
node --input-type=module -e "
import * as dt from '@clickhouse/datatype-parser';
if (typeof dt.parseDataType !== 'function') throw new Error('parseDataType missing from main export');
console.log('OK: @clickhouse/datatype-parser@${PUBLISHED_VERSION} imports cleanly');
"
30 changes: 30 additions & 0 deletions .github/workflows/tests-skill-rowbinary-parser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,14 @@ on:
paths:
- .github/workflows/tests-skill-rowbinary-parser.yml
- skills/clickhouse-js-node-rowbinary-parser/**
# The skill depends on @clickhouse/datatype-parser; rerun against local
# parser changes so a parser regression cannot pass this suite unnoticed.
- packages/datatype-parser/**
pull_request:
paths:
- .github/workflows/tests-skill-rowbinary-parser.yml
- skills/clickhouse-js-node-rowbinary-parser/**
- packages/datatype-parser/**

concurrency:
group: "${{ github.workflow }}-${{ github.ref }}"
Expand All @@ -33,9 +37,22 @@ jobs:
with:
node-version: 24

# Build + pack the in-repo @clickhouse/datatype-parser so the skill is
# exercised against THIS checkout, not the version published to npm. The
# committed package.json keeps a published range for end users; we only
# override node_modules below (`npm install <tarball> --no-save`).
- name: Pack local @clickhouse/datatype-parser
working-directory: ${{ github.workspace }}
run: |
npm ci
npm pack --workspace @clickhouse/datatype-parser --pack-destination "${{ runner.temp }}"

- name: Install dependencies
run: npm ci

- name: Use the local datatype-parser build
run: npm install "${{ runner.temp }}"/clickhouse-datatype-parser-*.tgz --no-save

- name: Typecheck
run: npm run typecheck

Expand Down Expand Up @@ -68,8 +85,21 @@ jobs:
with:
node-version: ${{ matrix.node }}

# Build + pack the in-repo @clickhouse/datatype-parser so the skill is
# exercised against THIS checkout, not the version published to npm. The
# committed package.json keeps a published range for end users; we only
# override node_modules below (`npm install <tarball> --no-save`).
- name: Pack local @clickhouse/datatype-parser
working-directory: ${{ github.workspace }}
run: |
npm ci
npm pack --workspace @clickhouse/datatype-parser --pack-destination "${{ runner.temp }}"

- name: Install dependencies
run: npm ci

- name: Use the local datatype-parser build
run: npm install "${{ runner.temp }}"/clickhouse-datatype-parser-*.tgz --no-save

- name: Run unit tests
run: npm test
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,14 @@

- The `@clickhouse/client-common` package is deprecated. `@clickhouse/client` (Node.js) and `@clickhouse/client-web` (Web) no longer depend on it; the shared code is now bundled into each client package. Everything previously importable from `@clickhouse/client-common` should be imported from `@clickhouse/client` or `@clickhouse/client-web` instead. The `@clickhouse/client-common` package itself will no longer receive updates. ([#845])

- The `parseColumnType` function and its `SimpleColumnTypes` companion (exported from `@clickhouse/client`, `@clickhouse/client-web`, and `@clickhouse/client-common`) are deprecated and slated for removal in a future major version. They are superseded by the new standalone [`@clickhouse/datatype-parser`](https://www.npmjs.com/package/@clickhouse/datatype-parser) package (`parseDataType` plus its `Node` AST), which parses the full ClickHouse data-type grammar and emits an AST that mirrors the server's. ([#893])

## New features

- (Node.js) Added a RowBinary reader library and agent skill under [`skills/clickhouse-js-node-rowbinary-parser`](./skills/clickhouse-js-node-rowbinary-parser). It ships type-specific, monomorphizable building blocks for decoding `RowBinary` / `RowBinaryWithNames` / `RowBinaryWithNamesAndTypes` streams (full-buffer and chunked), plus a skill that guides an agent to generate bespoke high-performance parsers from a query's column types. The skill is bundled into `@clickhouse/client` (registered in `agents.skills`) and is also published independently as the [`@clickhouse/rowbinary`](https://www.npmjs.com/package/@clickhouse/rowbinary) package. A matching RowBinary writer is planned. ([#864])

- Published the [`@clickhouse/datatype-parser`](https://www.npmjs.com/package/@clickhouse/datatype-parser) package: a small, dependency-free standalone parser for ClickHouse data-type strings (the kind sent in the types row of `RowBinaryWithNamesAndTypes`, e.g. `Array(Nullable(UInt64))`, `Tuple(a UInt8, b String)`, `Enum8('a' = 1)`). It is a faithful port of the server's `ParserDataType` and emits a JSON AST that is byte-identical to the server's `EXPLAIN AST json = 1` data-type subtree. It supersedes the deprecated `parseColumnType` (see Migration Notes). ([#893])

- (Node.js, `@experimental`) Added an additive `connection?: Connection<Stream.Readable>` option to `createClient` that lets a caller plug an externally-built backend `Connection`-like object in place of the default HTTP(S) factory. Only supposed to be used for testing the `chDB` integration. ([#879])

- Added `ClickHouseSettingsInterface`, a package-neutral structural counterpart to `ClickHouseSettings`, exported from `@clickhouse/client`, `@clickhouse/client-web`, and `@clickhouse/client-common`. It is identical to `ClickHouseSettings` except that its index signature omits `SettingsMap` (a class with a private member, which TypeScript compares nominally). Because each client package now bundles its own copy of the common module, their `ClickHouseSettings` types are mutually unassignable; `ClickHouseSettingsInterface` is structurally identical across all three packages and assignable into each package's `ClickHouseSettings`, so a consumer that shares a single settings-producing helper across both the Node.js and Web clients can type it against this one type without casts. Values typed as `SettingsMap` cannot be carried through it — use `ClickHouseSettings` if you need them. ([#889])
Expand Down Expand Up @@ -81,6 +85,7 @@ await client.query({
[#845]: https://github.com/ClickHouse/clickhouse-js/pull/845
[#864]: https://github.com/ClickHouse/clickhouse-js/pull/864
[#889]: https://github.com/ClickHouse/clickhouse-js/pull/889
[#893]: https://github.com/ClickHouse/clickhouse-js/pull/893

## Bug Fixes

Expand Down
Loading
Loading