fix(vquery): unify SQL dialects and make IndexedDB opt-in

.changeset/vquery-parameterized-dialects.md

@@ -0,0 +1,9 @@
+---
+'@visactor/vquery': patch
+---
+
+Expose parameterized `compileQuery` with MySQL as the default dialect and explicit `convertDSLToMysql` / `convertDSLToDuckDB` helpers, replacing `convertDSLToSQL`. Keep dataset queries on the DuckDB strategy.
+
+Align MySQL and DuckDB aggregation, date grouping, string matching, NULL ordering, and median/quantile behavior. Add a shared real-schema DSL catalog that snapshots both dialects without database test dependencies.
+
+Default browser and Node dataset storage to memory. Keep separate Node and browser strategy entrypoints with shared constructor option types, and enable IndexedDB persistence only with `new VQuery({ enableIndexedDB: true })`.

apps/website/docs/zh-CN/vquery/development/tests.md

@@ -3,25 +3,70 @@
 VQuery 使用 `rstest` 框架进行测试。**所有命令需在根目录执行。**
 
 ## 测试机制
-VQuery 的测试覆盖了：
-- **Unit**: 工具函数与编译器逻辑。
-- **examples**: 完整的 SQL 生成与数据查询流程。
+
+- **Unit**：工具函数、编译器逻辑与内存查询引擎。
+- **examples**：SQL 生成与内存数据查询流程。
+- **dialects**：共用一套 DSL，快照校验 MySQL 和 DuckDB 的 SQL 生成结果，不连接数据库。
 
 ## 常用命令
 
 ### 运行所有测试
+
 ```bash
-pnpm --filter=@visactor/vquery run test
+pnpm --filter @visactor/vquery run test
 ```
 
 ### 更新快照
-如果 SQL 生成逻辑变更符合预期，需更新快照：
+
+确认 SQL 变化符合预期后更新快照，并审阅生成的差异：
+
 ```bash
-pnpm --filter=@visactor/vquery run test:update
+pnpm --filter @visactor/vquery run test:update
 ```
 
 ### 覆盖率报告
-生成并查看测试覆盖率：
+
+```bash
+pnpm --filter @visactor/vquery run test:coverage
+```
+
+## 方言 SQL 快照
+
+方言测试只有一套共享用例和一份快照：
+
+- `tests/dialects/queries.json`：以 `supermarket` 为查询表，保存真实 CSV 的完整 20 字段逻辑 schema 和共享 DSL 用例。`source` 指向仓库已有的 `tests/unit/assets/supermarket.csv`；该文件与本次提供的 CSV 内容一致，测试无需访问用户的 Downloads 目录。每条用例用唯一且稳定的 `name` 标识，可附 `description` 解释业务场景。
+- `tests/dialects/compile.test.ts`：每条 DSL 分别调用 MySQL、DuckDB 的 `compileQuery`，将两种方言的 `{ sql, parameters }` 一起核对快照。
+- `tests/dialects/__snapshots__/compile.test.ts.snap`：按用例保存两种方言的 SQL 和参数；通过快照差异审阅编译逻辑变化。
+
+测试纳入现有的 `test`、`test:update`、`test:coverage`，无需专用配置、数据库服务、CLI 或连接凭据。新增方言时复用同一套 DSL，增加对应编译结果即可。
+
+CSV 中 `id`、`sales`、`amount`、`discount`、`profit` 使用 number，`order_date`、`delivery_date` 使用 date，其余列使用 string。日期样例按 SQL DATE 格式表示为 `YYYY-MM-DD`。用例涵盖销售利润、商品层级、省市地域、客户贡献、折扣亏损与配送分析，使用真实字段和业务值；不在业务用例中改写 schema 或虚构源字段。
+
+测试将 schema 字段与 CSV 表头逐一比较，并逐条检查 SELECT、WHERE、HAVING 的源字段；GROUP BY、ORDER BY 可以引用真实字段或 SELECT 输出别名。聚合查询中的维度必须参与分组，聚合结果别名不能作为分组键。别名是输出名称，不视为新的源字段。
+
+只运行方言快照测试：
+
 ```bash
-pnpm --filter=@visactor/vquery run test:coverage
+pnpm --filter @visactor/vquery exec rstest --include 'tests/dialects/**/*.test.ts' --coverage.enabled=false
 ```
+
+覆盖率沿用包现有的四项 100% 门槛，非法 DSL 的拒绝分支由 Unit 测试覆盖，不维护另一套方言覆盖率配置。
+
+## 功能与组合覆盖
+
+源码覆盖率 100% 不等于所有 DSL 组合都被验证。嵌套层数、条件数量和参数值没有有限的穷举集合；用例库持续积累实际场景和历史回归，不以数量代替覆盖范围。
+
+共享 JSON 用例库持续扩充，测试同时检查以下明确的组合要求，删除必要用例会失败。功能与操作符清单通过 DSL 类型约束；现有 `typecheck` 同时检查源码和测试，新增功能时需要同步补充清单与用例。
+
+| 范围 | 覆盖要求 |
+| --- | --- |
+| SELECT | 11 种基础聚合、8 种日期聚合，以及字段、别名与全部字段投影 |
+| HAVING | 11 种基础聚合 × 12 种比较操作符，共 132 种组合 |
+| 日期分组 | 8 种日期聚合 × 11 种基础聚合，共 88 种组合 |
+| 排序 | 11 种基础聚合 × 升降序 × 默认/first/last NULL 位置，共 66 种组合 |
+| WHERE | 16 种操作符分别位于 AND、OR 嵌套树中，共 32 种组合 |
+| 完整查询 | SELECT、WHERE、GROUP BY、HAVING、ORDER BY、limit 同时出现 |
+
+另包含商品名称 MIN/MAX 的 HAVING、下单日与配送日分组、多日期多字段分组、销售与利润跨字段统计、未投影 HAVING、别名冲突、2/4/8 层嵌套等回归场景。这些场景不代表任意组合均已穷尽。
+
+追加场景时，直接在 `queries.json` 的 `queries` 数组中增加基于这份 schema 的完整 DSL，用 `description` 说明触发条件或历史问题，再运行 `test:update` 并审阅两种方言的 SQL 差异。不要复制方言专属用例或加入随机生成步骤。无 schema、特殊标识符、JS `Date`、`Infinity` 等编译器边界由 Unit 测试维护。

apps/website/docs/zh-CN/vquery/index.md

@@ -1,5 +1,21 @@
 # VQuery
 
-:::info In Development
-Coming Soon
-:::
+VQuery 在 Node.js 和浏览器中默认使用内存查询与存储 adapter，无需 IndexedDB。每个实例独立管理自己的数据集。
+
+```ts
+import { VQuery } from '@visactor/vquery'
+
+const vquery = new VQuery()
+```
+
+默认存储的数据只在当前实例生命周期内有效，关闭实例或刷新页面后不保留。
+
+## 启用 IndexedDB
+
+需要跨会话保存数据集 schema 和数据源时，显式开启 `enableIndexedDB`：
+
+```ts
+const vquery = new VQuery({ enableIndexedDB: true })
+```
+
+开启后使用 IndexedDB 存储 adapter，运行环境需要支持 IndexedDB；查询仍通过内存查询 adapter 执行。省略此选项、传入空 options 或设置为 `false` 时，均使用内存存储，无需 IndexedDB API。

packages/vquery/README.md

@@ -1 +1,17 @@
 # VQuery
+
+Node.js and browser entrypoints use in-memory query and storage adapters by default. No IndexedDB API is required, and each VQuery instance owns its datasets.
+
+```ts
+import { VQuery } from '@visactor/vquery'
+
+const vquery = new VQuery()
+```
+
+Enable IndexedDB explicitly to persist dataset schemas and sources between sessions:
+
+```ts
+const vquery = new VQuery({ enableIndexedDB: true })
+```
+
+The runtime must support IndexedDB when this option is enabled. Queries still run in memory. With the default storage, closing the instance or reloading the page discards its datasets.

packages/vquery/rstest.config.ts

@@ -4,17 +4,16 @@ export default defineConfig({
   globals: true,
   testEnvironment: 'node',
   includeSource: ['src/**/*.{js,ts}'],
-  exclude: ['src/browser.ts', 'src/vquery-browser.ts'],
+  exclude: ['src/browser.ts'],
   setupFiles: ['./tests/setup.ts'],
   coverage: {
     enabled: true,
     provider: 'istanbul',
     exclude: [
       '**/node_modules/**',
       '**/dist/**',
-      // rstest can't run browser entrypoints.
+      // Export barrels have no runtime logic; both entrypoints are exercised by unit tests.
       'src/browser.ts',
-      'src/vquery-browser.ts',
     ],
     include: ['src/**'],
     reporters: ['html', ['text', { skipFull: true }], 'json', 'json-summary'],

packages/vquery/scripts/build-tests.js

@@ -34,7 +34,7 @@ function generateTest(jsonPath) {
     const testPath = path.join(path.dirname(jsonPath), testFileName)
 
     const template = `import type { DatasetColumn, VQueryDSL } from '@visactor/vquery'
-import { convertDSLToSQL, VQuery } from '@visactor/vquery'
+import { convertDSLToDuckDB, VQuery } from '@visactor/vquery'
 import vqueryConfig from './${jsonFileName}'
 
 describe('${description}', () => {
@@ -52,7 +52,7 @@ describe('${description}', () => {
 
     const dataset = await vquery.connectDataset(datasetId)
 
-    const sql = convertDSLToSQL(vqueryDSL as VQueryDSL<Record<string, string | number>>, datasetId)
+    const sql = convertDSLToDuckDB(vqueryDSL as VQueryDSL<Record<string, string | number>>, datasetId)
 
     expect(sql).toMatchInlineSnapshot()
 

packages/vquery/src/browser.ts

@@ -1,4 +1,5 @@
-export { convertDSLToSQL, Dataset } from './dataset'
+export { Dataset } from './dataset'
+export * from './sql-builder'
 export { DatasetSourceBuilder } from './data-source-builder/dataSourceBuilder'
 export * from './utils'
 export * from './types'

packages/vquery/src/dataset/dataset.ts

@@ -1,5 +1,5 @@
 import type { DatasetColumn, DatasetSource, QueryDSL, QueryAdapter, StorageAdapter, VQueryDSL } from '../types'
-import { convertDSLToSQL } from '../sql-builder'
+import { convertDSLToDuckDB } from '../sql-builder'
 
 export class Dataset {
   private queryAdapter: QueryAdapter
@@ -38,7 +38,7 @@ export class Dataset {
   }
 
   public async query<T extends Record<string, number | string>>(queryDSL: QueryDSL<T> | VQueryDSL<T>) {
-    const sql = convertDSLToSQL(queryDSL, this.datasetId)
+    const sql = convertDSLToDuckDB(queryDSL, this.datasetId)
     return this.queryBySQL(sql)
   }
 

packages/vquery/src/dataset/index.ts

@@ -1,2 +1 @@
 export { Dataset } from './dataset'
-export { convertDSLToSQL } from '../sql-builder'

packages/vquery/src/node.ts

@@ -1,4 +1,5 @@
-export { convertDSLToSQL, Dataset } from './dataset'
+export { Dataset } from './dataset'
+export * from './sql-builder'
 export { DatasetSourceBuilder } from './data-source-builder/dataSourceBuilder'
 export * from './utils'
 export * from './types'

packages/vquery/src/query-engine/sql.ts

@@ -29,7 +29,7 @@ type Predicate =
   | { type: 'truthy'; value: Expression }
 
 type SelectItem = { expression: Expression; alias?: string }
-type OrderItem = { expression: Expression; direction: 'asc' | 'desc' }
+type OrderItem = { expression: Expression; direction: 'asc' | 'desc'; nulls: 'first' | 'last' }
 
 export type SelectQuery = {
   table?: string
@@ -263,7 +263,12 @@ class Parser {
     if (this.match('like') || this.match('ilike')) {
       const matchedOperator = this.tokens[this.index - 1].value.toLowerCase() as 'like' | 'ilike'
       const operator: LikeOperator = negated ? `not ${matchedOperator}` : matchedOperator
-      return { type: 'comparison', operator, left, right: this.scalar() }
+      const right = this.scalar()
+      if (this.match('escape')) {
+        const escape = this.scalar()
+        if (escape.type !== 'literal' || escape.value !== '') throw new Error('Only empty LIKE escape is supported')
+      }
+      return { type: 'comparison', operator, left, right }
     }
     if (negated) throw new Error('Expected IN, BETWEEN, LIKE, or ILIKE after NOT')
     const token = this.current()
@@ -299,7 +304,12 @@ class Parser {
         do {
           const expression = this.scalar()
           const direction = this.match('desc') ? 'desc' : (this.match('asc'), 'asc')
-          result.orderBy.push({ expression, direction })
+          let nulls: 'first' | 'last' = 'last'
+          if (this.match('nulls')) {
+            if (this.match('first')) nulls = 'first'
+            else this.expect('last')
+          }
+          result.orderBy.push({ expression, direction, nulls })
         } while (this.match(','))
       } else if (this.match('limit')) {
         const token = this.current()
@@ -358,10 +368,8 @@ const compareText = (left: string, right: string): number => {
   return leftCodePoints.length - rightCodePoints.length
 }
 
-const compare = (left: unknown, right: unknown): number => {
+const compareNonNull = (left: unknown, right: unknown): number => {
   if (left === right) return 0
-  if (left === null || left === undefined) return 1
-  if (right === null || right === undefined) return -1
   if (typeof left === 'number' && typeof right === 'number') return left - right
   return compareText(String(left), String(right))
 }
@@ -411,8 +419,10 @@ const aggregate = (expression: Extract<Expression, { type: 'call' }>, rows: Data
     const numbers = numericValues(argument, rows)
     return numbers.length ? numbers.reduce((sum, value) => sum + value, 0) / numbers.length : null
   }
-  if (expression.name === 'min') return values.length ? values.reduce((a, b) => (compare(a, b) <= 0 ? a : b)) : null
-  if (expression.name === 'max') return values.length ? values.reduce((a, b) => (compare(a, b) >= 0 ? a : b)) : null
+  if (expression.name === 'min')
+    return values.length ? values.reduce((a, b) => (compareNonNull(a, b) <= 0 ? a : b)) : null
+  if (expression.name === 'max')
+    return values.length ? values.reduce((a, b) => (compareNonNull(a, b) >= 0 ? a : b)) : null
   const numbers = numericValues(argument, rows).sort((a, b) => a - b)
   if (!numbers.length) return null
   if (expression.name === 'median') {
@@ -426,10 +436,8 @@ const aggregate = (expression: Extract<Expression, { type: 'call' }>, rows: Data
     if (value === null || value === undefined || !Number.isFinite(quantile) || quantile < -1 || quantile > 1) {
       throw new Error('QUANTILE parameter must be a number in the range [-1, 1]')
     }
-    const index =
-      quantile < 0
-        ? Math.min(numbers.length + Math.floor(quantile * numbers.length), numbers.length - 1)
-        : Math.max(Math.ceil(quantile * numbers.length) - 1, 0)
+    const position = Math.max(numbers.length - Math.floor(numbers.length - numbers.length * Math.abs(quantile)), 1)
+    const index = quantile < 0 ? numbers.length - position : position - 1
     return numbers[index]
   }
   const average = numbers.reduce((sum, value) => sum + value, 0) / numbers.length
@@ -513,7 +521,7 @@ const evaluatePredicate = (predicate: Predicate, row: DataRow, rows: DataRow[]):
   if (predicate.operator === 'in' || predicate.operator === 'not in') {
     const values = predicate.values.map((value) => evaluate(value, row, rows))
     if (left === null || left === undefined) return null
-    const found = values.some((value) => value !== null && value !== undefined && compare(left, value) === 0)
+    const found = values.some((value) => value !== null && value !== undefined && compareNonNull(left, value) === 0)
     if (found) return predicate.operator === 'in'
     if (values.some((value) => value === null || value === undefined)) return null
     return predicate.operator === 'not in'
@@ -522,7 +530,7 @@ const evaluatePredicate = (predicate: Predicate, row: DataRow, rows: DataRow[]):
     const values = predicate.values.map((value) => evaluate(value, row, rows))
     if (left === null || left === undefined || values.some((value) => value === null || value === undefined))
       return null
-    const found = compare(left, values[0]) >= 0 && compare(left, values[1]) <= 0
+    const found = compareNonNull(left, values[0]) >= 0 && compareNonNull(left, values[1]) <= 0
     return predicate.operator === 'between' ? found : !found
   }
   const operator = predicate.operator
@@ -533,7 +541,7 @@ const evaluatePredicate = (predicate: Predicate, row: DataRow, rows: DataRow[]):
     return operator.startsWith('not ') ? !result : result
   }
   if (left === null || left === undefined || right === null || right === undefined) return null
-  const comparison = compare(left, right)
+  const comparison = compareNonNull(left, right)
   if (operator === '=') return comparison === 0
   if (operator === '!=' || operator === '<>') return comparison !== 0
   if (operator === '>') return comparison > 0
@@ -607,7 +615,13 @@ export const executeSQL = (query: SelectQuery, tableRows: DataRow[]): DataRow[]
         fieldName && Object.hasOwn(right.output, fieldName)
           ? right.output[fieldName]
           : evaluate(order.expression, right.row, right.rows)
-      const result = compare(leftValue, rightValue)
+      const leftNull = leftValue === null || leftValue === undefined
+      const rightNull = rightValue === null || rightValue === undefined
+      if (leftNull || rightNull) {
+        if (leftNull !== rightNull) return (leftNull ? 1 : -1) * (order.nulls === 'first' ? -1 : 1)
+        continue
+      }
+      const result = compareNonNull(leftValue, rightValue)
       if (result !== 0) return order.direction === 'desc' ? -result : result
     }
     if (!query.groupBy.length || !query.orderBy.length) return 0