diff --git a/.changeset/filter-icontains-and-regex-retirement.md b/.changeset/filter-icontains-and-regex-retirement.md new file mode 100644 index 0000000000..432cf85a79 --- /dev/null +++ b/.changeset/filter-icontains-and-regex-retirement.md @@ -0,0 +1,79 @@ +--- +"@objectstack/spec": minor +--- + +feat(spec): `$icontains` 入算子词表(ASCII 折叠域)、`$contains` 族钉死为大小写敏感、`$regex` 退役指引表(#5701) + +#4706 维护者裁决 B 案的**契约半边**。本次只改声明,不改任何运行时行为: +五个后端今天怎么答,落地后还怎么答。驱动侧下译归 #5702,`$regex` 唯一活生产者的 +翻转归 #5710。 + +## 1. 新增 `$icontains` —— 折叠域是 ASCII,不是 Unicode + +`StringOperatorSchema` / `FieldOperatorsSchema` / `Filter` 新增 `$icontains`: +忽略大小写的子串包含,**只折叠 `A-Z` 与 `a-z`**。 + +```ts +{ name: { $contains: 'acme' } } // 大小写敏感:匹配 "acme corp",不匹配 "ACME Corp" +{ name: { $icontains: 'acme' } } // 折叠 ASCII 大小写:两者都匹配 +``` + +**边界必须说清楚:`café` 不匹配 `CAFÉ`。** ASCII 以外一律按字面比较。 +选 ASCII 而非全 Unicode,是因为它是五个后端唯一都能真兑现的折叠域 —— +无 ICU 的 SQLite(`driver-sqlite-wasm` / `driver-turso` 跑的就是它)的 +`LOWER()` 与 `LIKE` 只折叠 ASCII,承诺 Unicode 等于承诺三个后端做不到的事, +那正是 #4706 用来否决「五后端真正则」的同一条判据。 + +比较值一律**字面量**:`%` / `_` 不是 LIKE 通配符,`.` / `*` 不是正则元字符 —— +`{ name: { $icontains: 'a.b' } }` 匹配 `a.b`,不匹配 `axb`。 + +## 2. `$contains` / `$notContains` / `$startsWith` / `$endsWith` = 大小写敏感 + +这条**取代**了 `filter.zod.ts` 里那句已记录的声明(Prime Directive #13, +取代记录写在原处): + +> Note: Case sensitivity should be handled at backend level. + +那不是漏写,是写下来的「不保证」,实测代价是同一个算子三种答案: +`driver-memory` 的参考匹配器与 `formula` 大小写敏感,`driver-mongodb` 硬编码 +`$options: 'i'` 全 Unicode 不敏感,SQL 家族看方言(SQLite 折叠 ASCII、 +Postgres 不折叠、MySQL 看 collation)。作者无法从算子名判断自己拿到哪一种。 + +**迁移**:此前依赖某后端偶然大小写不敏感的 `$contains` 查询,应改写为 +`$icontains`。行为在 #5702 落地前不变,所以这是一次可以提前做的改写,不是断裂。 + +## 3. `$regex` / `$options` 退役 —— 指引表 `RETIRED_FILTER_OPERATORS` + +`$regex` 从来不在 `FILTER_OPERATORS` 里,却有一个生产者、四个消费者,而且各读各的: +`driver-sql` 编译成 LIKE 转义后的子串匹配(`a.b` 只匹配字面 `a.b`), +`driver-memory` 当真正则求值(`a.b` 还匹配 `axb`;模式非法则被 `catch` 成零行, +无声)。真正则在五后端不可实现 —— `driver-turso` 的 remote 线协议无法注册 +SQLite `REGEXP` 函数。 + +新增 `RETIRED_FILTER_OPERATORS`(**纯数据**,不引入任何拒收行为), +给出逐条处方,供五个既有拒收点引用同一句话: + +| 原写法 | 改写为 | +|:---|:---| +| `{ name: { $regex: 'acme' } }` | `{ name: { $icontains: 'acme' } }` | +| `{ name: { $regex: 'acme', $options: 'i' } }` | `{ name: { $icontains: 'acme' } }` | +| `{ name: { $regex: '^acme' } }` | `{ name: { $startsWith: 'acme' } }` | +| `{ name: { $regex: 'acme$' } }` | `{ name: { $endsWith: 'acme' } }` | + +真正需要正则的查询没有 filter 层替代物:用已声明算子收窄,再在应用代码里匹配。 + +## 4. 新姊妹 case-set `FILTER_TEXT_CASES` + +`filter-text-conformance.ts` —— 大小写折叠、字面比较值、`$regex` 拒收的共享标准, +带 `expectRejection` 判别式(`FILTER_LOGIC_CASES` 刻意没长出来的那个形状, +其表头三条章程原样保留)。五个 driver 各记一条**实测** DEBT 台账,指向 #5702。 + +## 什么**没有**变 + +`$icontains` 暂不进 `FILTER_OPERATORS`。那个数组不是词表而是运行时白名单 —— +`driver-memory` 的 `SUPPORTED_FIELD_OPERATORS` 由它派生。实测:提前把 +`$icontains` 放进去,该驱动的形状门禁就不再拒收它,而匹配器没有对应分支, +`match({name:'zzz'}, {name:{$icontains:'acme'}})` 返回 `true` —— 谓词被静默丢弃, +全表命中。谓词被丢不是收窄而是**放大**,在 RLS 读作用域上是越权读(#3948)。 +所以它随 #5702 的实现一起入列,`filter-operator-vocabulary.test.ts` 把这处差异 +钉死为恰好 `{ $icontains }`,清偿时该断言会红,提醒作者一并删掉过渡说明。 diff --git a/content/docs/protocol/objectql/query-syntax.mdx b/content/docs/protocol/objectql/query-syntax.mdx index a55aae4d8c..28a18e05ca 100644 --- a/content/docs/protocol/objectql/query-syntax.mdx +++ b/content/docs/protocol/objectql/query-syntax.mdx @@ -237,14 +237,82 @@ const query: QueryAST = { | `$lte` | Less or equal | `{ discount: { $lte: 20 } }` | | `$in` | In list | `{ stage: { $in: ['proposal', 'negotiation'] } }` | | `$nin` | Not in list | `{ status: { $nin: ['deleted', 'archived'] } }` | -| `$contains` | String contains | `{ name: { $contains: 'Inc' } }` | -| `$notContains` | String does not contain | `{ name: { $notContains: 'test' } }` | -| `$startsWith` | String starts with | `{ email: { $startsWith: 'admin' } }` | -| `$endsWith` | String ends with | `{ domain: { $endsWith: '.com' } }` | +| `$contains` | String contains, **case-sensitive** | `{ name: { $contains: 'Inc' } }` | +| `$icontains` | String contains, **ignoring ASCII case** | `{ name: { $icontains: 'inc' } }` | +| `$notContains` | String does not contain, **case-sensitive** | `{ name: { $notContains: 'test' } }` | +| `$startsWith` | String starts with, **case-sensitive** | `{ email: { $startsWith: 'admin' } }` | +| `$endsWith` | String ends with, **case-sensitive** | `{ domain: { $endsWith: '.com' } }` | | `$between` | Range (inclusive) | `{ close_date: { $between: ['2024-01-01', '2024-12-31'] } }` | | `$null` | Null check | `{ manager_id: { $null: true } }` / `{ phone: { $null: false } }` | | `$exists` | Field exists (NoSQL) | `{ metadata: { $exists: true } }` | +### Case Sensitivity + +The string operators compare **case-sensitively**. `$icontains` is the one that does +not, and the case it ignores is **ASCII case only** — `A-Z` against `a-z`, and nothing +else. + +```typescript +// Case-sensitive: matches "acme corp", NOT "ACME Corp" +{ name: { $contains: 'acme' } } + +// ASCII case-insensitive: matches BOTH "acme corp" and "ACME Corp" +{ name: { $icontains: 'acme' } } +``` + + + **`café` does not match `CAFÉ`.** Outside `A-Z`/`a-z`, `$icontains` compares + literally — accented Latin, Cyrillic, Greek and every other script are matched + exactly as written. If your users search non-ASCII text, `$icontains` is not an + accent- or case-blind search, and treating it as one will silently return fewer + rows than expected. + + The boundary is ASCII because that is the only fold every backend can actually + deliver. SQLite compiled without ICU — which is what `driver-sqlite-wasm` and + `driver-turso` run on — folds ASCII only in both `LOWER()` and `LIKE`, so a + Unicode promise here would be a guarantee three of the five backends could not + keep. See [#4706](https://github.com/objectstack-ai/objectstack/issues/4706). + + +The comparand is always matched **literally**. `%` and `_` are ordinary characters, +not `LIKE` wildcards, and `.` / `*` / `+` are ordinary characters, not regex +metacharacters — `{ name: { $icontains: 'a.b' } }` matches `a.b` and not `axb`. + + + **Status:** the case rules above are the protocol's declaration as of + `@objectstack/spec` 18. The backend lowerings that deliver them — making SQLite's + and turso's `LIKE` case-exact, dropping MongoDB's hardcoded `$options: 'i'`, and + implementing `$icontains` everywhere — are tracked by + [#5702](https://github.com/objectstack-ai/objectstack/issues/5702). Until it + lands, a backend that has not been aligned refuses `$icontains` outright rather + than answering it approximately, and `$contains` still follows its dialect. The + shared standard both halves are measured against is `FILTER_TEXT_CASES` + (`@objectstack/spec/data`). + + +### `$regex` — removed + +`$regex` (and its `$options` companion) was never a declared operator and is +**retired** ([#4706](https://github.com/objectstack-ai/objectstack/issues/4706)). It +could not mean one thing across the backends: `driver-sql` compiled it to a +LIKE-escaped substring match, so `a.b` matched only the literal `a.b`, while +`driver-memory` evaluated it as a real `RegExp`, so the same filter also matched +`axb` — and an invalid pattern was caught and answered zero rows, in silence. A real +regex is not implementable on all five backends: `driver-turso`'s remote transport +speaks a wire protocol with no way to register a SQLite `REGEXP` function. + +| Instead of | Write | +|:---|:---| +| `{ name: { $regex: 'acme' } }` | `{ name: { $icontains: 'acme' } }` | +| `{ name: { $regex: 'acme', $options: 'i' } }` | `{ name: { $icontains: 'acme' } }` | +| `{ name: { $regex: '^acme' } }` | `{ name: { $startsWith: 'acme' } }` | +| `{ name: { $regex: 'acme$' } }` | `{ name: { $endsWith: 'acme' } }` | + +A pattern that genuinely needs a regular expression has no filter-level +replacement — narrow the query with the declared operators and match in application +code. The prescriptions above are declared as data in `RETIRED_FILTER_OPERATORS` +(`@objectstack/spec/data`), so every backend's refusal quotes the same sentence. + ### Multiple Conditions (Implicit AND) Multiple keys in `where` are combined with **AND** logic: @@ -783,11 +851,24 @@ search — and over the REST/protocol ingress it is `400 INVALID_FIELD` outright because the engine-side intersection alone used to drop the unknown name and fall back to scanning the full searchable set. Internal callers reaching `engine.find()` directly keep the tolerant intersection. Multiple whitespace-separated terms are AND-ed and -fields are OR-ed. Case sensitivity is the **driver's**, not the expansion's: the -expansion emits a plain `$contains`, which `SqlDriver` compiles to a parameterised -`LIKE '%…%'` with no case folding — so the dialect's own `LIKE`/collation rules decide — -while the in-memory driver matches with a case-insensitive regex. Only `select` / -`status` option *labels* are matched case-insensitively by the expansion itself. +fields are OR-ed. Case sensitivity comes from the operator the expansion emits, not +from the expansion: it emits a plain `$contains`, which is **case-sensitive** by the +rule in [Case Sensitivity](#case-sensitivity) above. Note what that means for search — +a user typing `acme` does not find `ACME Corp`. Only `select` / `status` option +*labels* are matched case-insensitively by the expansion itself. + + + **Measured today, and it does not match that rule yet.** The `$contains` alignment + is [#5702](https://github.com/objectstack-ai/objectstack/issues/5702), so until it + lands the answer is still the driver's: `SqlDriver` compiles a parameterised + `LIKE '%…%'` and the dialect decides (SQLite folds ASCII, Postgres does not), + `driver-mongodb` folds the full Unicode range through a hardcoded `$options: 'i'`, + and `driver-memory`'s query path matches with a case-insensitive regex. Which + driver you run therefore still changes which rows a search returns. Whether the + expansion should emit `$icontains` instead of `$contains` — i.e. whether search is + case-insensitive by definition — is a separate question that rides with that issue, + because it can only be answered once both operators mean one thing everywhere. + `fuzzy`, `boost`, `operator`, `minScore`, `language`, and `highlight` carry `[EXPERIMENTAL — not enforced]` markers (#4286): the schema accepts them, the expansion ignores them. diff --git a/content/docs/references/data/filter.mdx b/content/docs/references/data/filter.mdx index e490023a01..03fe6cd302 100644 --- a/content/docs/references/data/filter.mdx +++ b/content/docs/references/data/filter.mdx @@ -157,6 +157,7 @@ Type: `[FilterArray](#filterarray)[]` | **$notContains** | `string` | optional | | | **$startsWith** | `string` | optional | | | **$endsWith** | `string` | optional | | +| **$icontains** | `string` | optional | Contains substring, ignoring case — but ONLY ASCII case (A-Z against a-z). Every other character compares literally, so "café" does NOT match "CAFÉ" and "москва" does not match "МОСКВА". The domain is ASCII because that is the one fold all five backends can deliver: SQLite (and therefore turso and sqlite-wasm) folds ASCII only, so a Unicode promise here would be a guarantee three of the five could not keep. The comparand is matched LITERALLY — "%", "_" and regex metacharacters are ordinary characters, not wildcards. Case-SENSITIVE containment is $contains. [#5701: declared by the protocol; the driver lowerings land with #5702.] | --- diff --git a/packages/spec/api-surface.json b/packages/spec/api-surface.json index 33c4daba7a..5691868707 100644 --- a/packages/spec/api-surface.json +++ b/packages/spec/api-surface.json @@ -365,6 +365,8 @@ "FILTER_LOGIC_CASES (const)", "FILTER_LOGIC_ROWS (const)", "FILTER_OPERATORS (const)", + "FILTER_TEXT_CASES (const)", + "FILTER_TEXT_ROWS (const)", "FeedFilterMode (type)", "FeedItemType (type)", "Field (type)", @@ -398,6 +400,10 @@ "FilterLogicCase (interface)", "FilterLogicRow (interface)", "FilterOperatorKey (type)", + "FilterTextCase (type)", + "FilterTextRejectionCase (interface)", + "FilterTextRow (interface)", + "FilterTextRowsCase (interface)", "FormatValidation (type)", "FormatValidationSchema (const)", "FullTextSearch (type)", @@ -532,6 +538,7 @@ "READ_ONLY_BELONGS_ON_DATASOURCE (const)", "RECORD_SURFACE_PAGE_THRESHOLD (const)", "REFERENCE_VALUE_TYPES (const)", + "RETIRED_FILTER_OPERATORS (const)", "RPC_QUERY_ALIAS_SLOTS (const)", "RangeOperatorSchema (const)", "RecordFlow (type)", @@ -552,6 +559,7 @@ "ResolveApiOptions (interface)", "ResolveRecordDisplayNameOptions (interface)", "ResolvedHook (type)", + "RetiredFilterOperatorGuidance (interface)", "RowCrudActionOverride (type)", "RowCrudActionOverrideInput (type)", "RowCrudActionOverrideSchema (const)", diff --git a/packages/spec/authorable-surface.json b/packages/spec/authorable-surface.json index 98b3a4bca7..0cb7c4c06d 100644 --- a/packages/spec/authorable-surface.json +++ b/packages/spec/authorable-surface.json @@ -3815,6 +3815,7 @@ "data/StateMachineValidation:type", "data/StringOperator:$contains", "data/StringOperator:$endsWith", + "data/StringOperator:$icontains", "data/StringOperator:$notContains", "data/StringOperator:$startsWith", "data/TenancyConfig:enabled", diff --git a/packages/spec/src/data/filter-operator-vocabulary.test.ts b/packages/spec/src/data/filter-operator-vocabulary.test.ts new file mode 100644 index 0000000000..4393b517ad --- /dev/null +++ b/packages/spec/src/data/filter-operator-vocabulary.test.ts @@ -0,0 +1,129 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The filter operator vocabulary has TWO surfaces, and #5701 made them + * temporarily disagree on purpose. This file is what stops that from being + * silent. + * + * - **Declaration**: `FieldOperatorsSchema` / `StringOperatorSchema` / + * `Filter` — what an author may write and what `tsc` accepts. Nothing + * derives a runtime allowlist from these (verified: `NormalizedFilterSchema` + * is their only consumer and nothing parses a filter through it at runtime). + * - **Enforcement**: `FILTER_OPERATORS` — the array `driver-memory`'s shape + * gate and `service-analytics`' coverage test DERIVE from. An entry here is a + * claim that backends implement the operator. + * + * `$icontains` is declared and not yet enforced (#5701 is the contract half of + * the #4706 ruling; #5702 writes the lowerings). Measured on the branch that + * added it to `FILTER_OPERATORS` early: driver-memory's gate stopped refusing + * it and `match({ name: 'zzz' }, { name: { $icontains: 'acme' } })` returned + * `true` — the predicate silently dropped, every row matched. That is the + * widening #3948 is about, so the staging is not a stylistic choice. + * + * The pin below is deliberately an EQUALITY, not a subset check, so it fails in + * both directions: a second staged operator added without recording it fails + * here, and so does clearing `$icontains` in #5702 — which is the point. The + * failure message is the instruction. + */ + +import { describe, it, expect } from 'vitest'; +import { + FieldOperatorsSchema, + StringOperatorSchema, + FILTER_OPERATORS, + LOGICAL_OPERATORS, + RETIRED_FILTER_OPERATORS, +} from './filter.zod'; + +const declaredKeys = () => Object.keys(FieldOperatorsSchema.shape).sort(); + +describe('the declaration surface and the enforcement surface', () => { + it('differ by EXACTLY the operators staged ahead of their backends', () => { + const declared = new Set(declaredKeys()); + const enforced = new Set(FILTER_OPERATORS); + const stagedOnly = [...declared].filter((op) => !enforced.has(op)).sort(); + + expect( + stagedOnly, + 'FieldOperatorsSchema and FILTER_OPERATORS differ by something other than the recorded ' + + 'staging. If you are ADDING an operator: declare it in FieldOperatorsSchema only, and ' + + 'add it here plus a note on FILTER_OPERATORS saying which issue implements it — an ' + + 'operator in FILTER_OPERATORS with no backend arm makes driver-memory accept it and ' + + "silently DROP the predicate (measured, #5701). If you are CLEARING one because you " + + 'just implemented it (#5702): remove it from this list AND delete the staging paragraph ' + + 'on FILTER_OPERATORS, which is now describing something that is no longer true.', + ).toEqual(['$icontains']); + }); + + it('has no operator enforced that is not declared', () => { + const declared = new Set(declaredKeys()); + const undeclared = FILTER_OPERATORS.filter((op) => !declared.has(op)); + expect( + undeclared, + 'FILTER_OPERATORS demands backends implement an operator FieldOperatorsSchema does not ' + + 'declare, so an author cannot write it and `tsc` will reject it. This direction is ' + + 'never staging — it is a drift.', + ).toEqual([]); + }); + + it('declares $icontains on the string operator schema too', () => { + expect(Object.keys(StringOperatorSchema.shape)).toContain('$icontains'); + }); + + it('accepts a declared $icontains rather than stripping it', () => { + const parsed = FieldOperatorsSchema.parse({ $icontains: 'acme' }); + expect(parsed).toEqual({ $icontains: 'acme' }); + }); + + it('rejects a non-string $icontains comparand at the schema', () => { + expect(() => FieldOperatorsSchema.parse({ $icontains: 42 })).toThrow(); + }); +}); + +describe('RETIRED_FILTER_OPERATORS', () => { + const entries = Object.entries(RETIRED_FILTER_OPERATORS); + + it('covers the operators #4706 retired', () => { + expect(Object.keys(RETIRED_FILTER_OPERATORS).sort()).toEqual(['$options', '$regex']); + }); + + it('never points at an operator the protocol no longer has', () => { + // The `authoring-key-lint.test.ts` rule, applied to operators: a guidance + // table whose prescriptions name something undeclared is advice that sends + // an author into a second error. Note the check is against the DECLARATION + // surface, because `$icontains` is deliberately not in FILTER_OPERATORS yet. + const declared = new Set(declaredKeys()); + for (const [op, guidance] of entries) { + if (guidance.to === undefined) continue; + expect(declared.has(guidance.to), `${op} prescribes ${guidance.to}, which is not declared`).toBe(true); + } + }); + + it('states the replacement inside the prescription, not only in the `to` field', () => { + // A refusal prints `why`. If the replacement lives only in a sibling field + // the caller may not render, the error tells the author they are wrong + // without telling them what to write — which is the failure the tombstone + // convention exists to prevent (AGENTS.md, Post-Task Checklist step 3). + for (const [op, guidance] of entries) { + if (guidance.to === undefined) continue; + expect(guidance.why, `${op}'s prescription never names ${guidance.to}`).toContain(guidance.to); + } + }); + + it('names the retired operator itself, so a refusal can quote one string', () => { + for (const [op, guidance] of entries) { + expect(guidance.why, `${op}'s prescription never names ${op}`).toContain(op); + } + }); + + it('is not simultaneously declared anywhere — retired means gone', () => { + const declared = new Set([...declaredKeys(), ...FILTER_OPERATORS, ...LOGICAL_OPERATORS]); + for (const op of Object.keys(RETIRED_FILTER_OPERATORS)) { + expect(declared.has(op), `${op} is both retired and declared`).toBe(false); + } + }); + + it('is frozen — a consumer cannot mutate the shared prescriptions', () => { + expect(Object.isFrozen(RETIRED_FILTER_OPERATORS)).toBe(true); + }); +}); diff --git a/packages/spec/src/data/filter-text-conformance.test.ts b/packages/spec/src/data/filter-text-conformance.test.ts new file mode 100644 index 0000000000..52740c8312 --- /dev/null +++ b/packages/spec/src/data/filter-text-conformance.test.ts @@ -0,0 +1,158 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `FILTER_TEXT_CASES` is a standard no backend answers yet (#5701 is the + * contract half of the #4706 ruling; #5702 writes the lowerings). That makes it + * unusually easy for the table to be quietly WRONG — nothing executes it, so a + * miscounted `expected` list would sit there until a driver author trusted it + * and chased their own correct implementation. + * + * So this file executes it, against a reference evaluator written from the + * declared semantics: ASCII-only case folding, literal comparands. If the table + * and the declaration disagree, one of them is wrong TODAY rather than in six + * months. + * + * The second job is proving the table is not VACUOUS. Two cases exist solely to + * pin the ASCII-only boundary (#4706 Q1 = A), and a fixture that cannot tell an + * ASCII fold from a Unicode fold would pass them by accident. `the ASCII-only + * cases discriminate` runs both folds and requires them to DISAGREE exactly + * there — the reverse verification, made permanent, in the direction that + * matters: a Unicode-folding backend must FAIL those two rows. + */ + +import { describe, it, expect } from 'vitest'; +import { + FILTER_TEXT_CASES, + FILTER_TEXT_ROWS, + type FilterTextCase, + type FilterTextRowsCase, +} from './filter-text-conformance'; + +// ── A reference evaluator, from the declared semantics ─────────────────────── + +/** Fold `A-Z` and nothing else — the domain #4706 Q1 pinned. */ +const asciiFold = (s: string) => s.replace(/[A-Z]/g, (c) => c.toLowerCase()); +/** The fold the contract deliberately does NOT promise — for the discrimination test. */ +const unicodeFold = (s: string) => s.toLowerCase(); + +/** Evaluate one field constraint against one value, with the given fold. */ +function evaluate(value: string, ops: Record, fold: (s: string) => string): boolean { + return Object.entries(ops).every(([op, comparand]) => { + const c = String(comparand); + switch (op) { + // Case-insensitive containment folds BOTH sides. + case '$icontains': return fold(value).includes(fold(c)); + // The rest compare literally — no fold, no wildcards, no regex. + case '$contains': return value.includes(c); + case '$notContains': return !value.includes(c); + case '$startsWith': return value.startsWith(c); + case '$endsWith': return value.endsWith(c); + default: throw new Error(`reference evaluator has no arm for ${op}`); + } + }); +} + +/** Ids the reference evaluator returns for a rows-case, ascending. */ +function referenceIds(c: FilterTextRowsCase, fold: (s: string) => string): string[] { + const ops = (c.filter as Record>).name; + return FILTER_TEXT_ROWS.filter((r) => evaluate(r.name, ops, fold)).map((r) => r.id); +} + +const isRowsCase = (c: FilterTextCase): c is FilterTextRowsCase => c.expectRejection !== true; +const rowsCases = FILTER_TEXT_CASES.filter(isRowsCase); +const rejectionCases = FILTER_TEXT_CASES.filter((c) => !isRowsCase(c)); + +// ── The table agrees with the declaration ──────────────────────────────────── + +describe('FILTER_TEXT_CASES against the declared semantics', () => { + for (const c of rowsCases) { + it(`reference evaluator reproduces: ${c.name}`, () => { + expect(referenceIds(c, asciiFold)).toEqual([...c.expected]); + }); + } + + it('the ASCII-only cases discriminate — a Unicode fold FAILS exactly them', () => { + // The whole point of Q1 = A. If this ever reports zero disagreements, the + // fixture stopped being able to tell the two folds apart and the boundary + // is no longer pinned by anything. + const disagree = rowsCases + .filter((c) => referenceIds(c, unicodeFold).join() !== referenceIds(c, asciiFold).join()) + .map((c) => c.name); + + expect(disagree.sort()).toEqual([ + 'ASCII-only: a lower-case non-ASCII comparand does NOT match its upper-case row', + 'ASCII-only: an upper-case non-ASCII comparand does NOT match its lower-case row', + ]); + + // And name the wrong answer, so the failure explains itself: a Unicode + // folder returns BOTH café rows for a query that must return one. + const caseLower = rowsCases.find((c) => c.name.startsWith('ASCII-only: a lower-case'))!; + expect(referenceIds(caseLower, unicodeFold)).toEqual(['3', '4']); + expect(referenceIds(caseLower, asciiFold)).toEqual(['4']); + }); + + it('the case-SENSITIVITY cases discriminate — a folding $contains FAILS them', () => { + // The Q2 = A pin, proved non-vacuous the same way: today three of five + // backends fold `$contains`, so the rows that must exclude their + // case-variant twin have to actually exclude it. + const contains = rowsCases.find((c) => c.name.startsWith('$contains is case-SENSITIVE — a lower-case'))!; + expect(contains.expected).toEqual(['2']); + const folding = FILTER_TEXT_ROWS.filter((r) => asciiFold(r.name).includes(asciiFold('acme'))).map((r) => r.id); + expect(folding, 'a case-folding $contains returns both — that is the answer #4706 Q2 rejected').toEqual(['1', '2']); + }); +}); + +// ── The table is internally well-formed ────────────────────────────────────── + +describe('FILTER_TEXT_CASES shape', () => { + it('has unique case names — they are used as test names', () => { + const names = FILTER_TEXT_CASES.map((c) => c.name); + expect(new Set(names).size).toBe(names.length); + }); + + it('has unique row ids', () => { + const ids = FILTER_TEXT_ROWS.map((r) => r.id); + expect(new Set(ids).size).toBe(ids.length); + }); + + it('never expects an id the fixture does not contain', () => { + const ids = new Set(FILTER_TEXT_ROWS.map((r) => r.id)); + for (const c of rowsCases) { + for (const id of c.expected) { + expect(ids.has(id), `${c.name} expects row ${id}, which is not in FILTER_TEXT_ROWS`).toBe(true); + } + } + }); + + it('lists expected ids ascending, without duplicates', () => { + for (const c of rowsCases) { + const asked = [...c.expected]; + expect([...new Set(asked)].sort(), c.name).toEqual(asked); + } + }); + + it('covers both verdicts — a table with no rejection case would not need the discriminant', () => { + expect(rowsCases.length).toBeGreaterThan(0); + expect(rejectionCases.length).toBeGreaterThan(0); + }); +}); + +describe('the rejection cases carry a prescription, not just a refusal', () => { + for (const c of rejectionCases) { + it(c.name, () => { + if (isRowsCase(c)) throw new Error('unreachable — filtered above'); + // ADR-0112 envelope, so a suite cannot accept a bare 500-shaped throw. + expect(c.code).toBe('INVALID_FILTER'); + expect(c.mustMention.length).toBeGreaterThan(0); + // Every refused spelling must be named, so the author can find it in their filter… + const refused = Object.keys((c.filter as Record>).name); + for (const op of refused) { + expect(c.mustMention, `${c.name} refuses ${op} without requiring the message to name it`).toContain(op); + } + // …and the retired ones must name the replacement, or the error is a dead end. + if (refused.some((op) => op === '$regex' || op === '$options')) { + expect(c.mustMention).toContain('$icontains'); + } + }); + } +}); diff --git a/packages/spec/src/data/filter-text-conformance.ts b/packages/spec/src/data/filter-text-conformance.ts new file mode 100644 index 0000000000..2d20153985 --- /dev/null +++ b/packages/spec/src/data/filter-text-conformance.ts @@ -0,0 +1,298 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * Canonical conformance cases for the Filter Protocol's **text operators** — + * the single standard every filter backend is checked against for case + * folding, literal comparands, and the retired `$regex`. + * + * ## Why this is a SEPARATE table from `FILTER_LOGIC_CASES` + * + * `filter-logic-conformance.ts` writes down three rules about itself, and this + * table exists because #5701 needed all three left intact: + * + * 1. **Scope.** That table's header says, of its own contents: *"Nothing here + * exercises null handling, dates, numeric coercion, `LIKE` escaping, or case + * sensitivity — those legitimately differ between a SQL engine and a JS + * matcher, and folding them in would make the table unpassable rather than + * more useful. Keep it that way."* `LIKE` escaping and case sensitivity are + * precisely this table's subject. They stopped "legitimately differing" when + * #4706 ruled a single answer for them — but the logic table's rule is still + * right for the logic table. + * 2. **Timing.** *"A red row here does not enforce a ruling, it just turns + * another lane's unfinished work into this table's failure … Add the rows in + * the PR that closes the gap, not before."* The `check-driver-conformance` + * gate judges a case-set by IMPORT, so adding rows to a case-set five suites + * already drive turns those suites red immediately. A NEW case-set nobody + * imports yet is instead five DEBT ledger rows — measured, tracked, and + * `main` stays green. + * 3. **Shape.** *"`FilterLogicCase` has no way to spell 'this filter must be + * REJECTED' — `expected` is a row-id list, and an empty list means 'matched + * nothing', which is precisely the FALSE answer the ruling did NOT take. + * Enrolling this case needs the shape extended first (an `expectRejection` + * discriminant, or a sibling table) — deliberately not invented here."* + * This is that sibling table, and {@link FilterTextCase} is that + * discriminant. #5240's `{ field: {} }` family can adopt the shape from here + * without reopening the logic table. + * + * ## Status: NO backend answers this table yet — that is the design + * + * This is the contract half of the #4706 ruling (#5701). `$icontains` is + * declared by `StringOperatorSchema` (`filter.zod.ts`) and implemented by nobody; the + * `$contains` family's case-sensitivity is declared and honoured by two of five + * backends. Every driver therefore carries a measured DEBT row in + * `scripts/check-driver-conformance.mjs`, pointing at **#5702**, which is the + * issue that writes the lowerings and deletes the rows. A suite arriving here + * before then would be red about work nobody has been dispatched to do, which + * is the failure mode rule 2 above names. + * + * The `$regex` rejection cases carry a further ordering constraint: `$regex` + * still has one LIVE producer — `plugin-auth`'s ObjectQL adapter emits + * `{ field: { $regex: value } }` for better-auth's `contains` search, on the + * authentication path. **#5710 flips that producer first.** A backend that + * enrols these cases before #5710 lands breaks sign-in. + * + * ## What belongs here + * + * Text-operator semantics that every backend must agree on: which characters + * fold, which characters are literal, and which spellings are refused. Anything + * whose answer legitimately differs per backend does not belong — the same bar + * the logic table sets, applied to a different axis. + * + * @see FILTER_LOGIC_CASES — combinator semantics, the sibling standard. + * @see https://github.com/objectstack-ai/objectstack/issues/4706 (the ruling) + * @see https://github.com/objectstack-ai/objectstack/issues/5701 (this table) + * @see https://github.com/objectstack-ai/objectstack/issues/5702 (the backends) + */ + +import type { FilterCondition } from './filter.zod'; + +/** + * A row in the conformance fixture. One text column is enough: every case here + * is a predicate on one string, and adding columns would only invite cases that + * belong in the logic table. + */ +export interface FilterTextRow { + id: string; + name: string; +} + +/** + * The fixture. Every row exists to make one wrong answer VISIBLE, in pairs: + * + * | pair | rows | the mistake it catches | + * |---|---|---| + * | ASCII case | 1, 2 | a fold that does not happen (`$icontains`), or one that happens when it must not (`$contains`) | + * | non-ASCII case | 3, 4 | a fold WIDER than ASCII — the boundary #4706 Q1 pinned | + * | `%` | 5, 6 | a comparand `%` reaching SQL as a LIKE wildcard | + * | `_` / `.` | 7, 8, 9 | a comparand `_` reaching SQL as a wildcard, or `.` reaching a regex engine | + * + * Each pair is deliberately near-identical apart from the one character under + * test, so a case that matches the wrong member of a pair returns visibly wrong + * ids rather than the same count by luck. + */ +export const FILTER_TEXT_ROWS: readonly FilterTextRow[] = [ + { id: '1', name: 'ACME Corp' }, + { id: '2', name: 'acme corp' }, + { id: '3', name: 'CAFÉ' }, + { id: '4', name: 'café' }, + { id: '5', name: '100% match' }, + { id: '6', name: '100X match' }, + { id: '7', name: 'a_b' }, + { id: '8', name: 'axb' }, + { id: '9', name: 'a.b' }, +] as const; + +/** Fields every case carries, whatever its verdict. */ +interface FilterTextCaseBase { + /** Stable identifier, usable as a test name. */ + readonly name: string; + /** The filter under test. */ + readonly filter: FilterCondition; + /** Why the case is here — surfaced in failure output. */ + readonly note?: string; +} + +/** A case whose filter must be EVALUATED, matching exactly {@link expected}. */ +export interface FilterTextRowsCase extends FilterTextCaseBase { + readonly expectRejection?: false; + /** Ids of matching rows, ascending. */ + readonly expected: readonly string[]; +} + +/** + * A case whose filter must be REFUSED. + * + * This is the discriminant `filter-logic-conformance.ts` deliberately did not + * invent (its rule 3, quoted in this file's header). It exists because + * `expected: []` cannot express it: an empty row list means "evaluated, matched + * nothing", which is a FALSE answer — and for a retired operator, answering + * FALSE is exactly the silent wrong answer the retirement is meant to end. A + * suite must distinguish "returned no rows" from "refused to run". + */ +export interface FilterTextRejectionCase extends FilterTextCaseBase { + readonly expectRejection: true; + /** + * The ADR-0112 error code the refusal must carry. A refusal outside the + * envelope reaches the client as a 500-shaped body for a 400-class mistake, + * which is the half of #5324 that the refusal itself does not fix. + */ + readonly code: 'INVALID_FILTER'; + /** + * Substrings the refusal message must contain, checked case-sensitively. + * + * Always includes the rejected spelling and, where one exists, the + * REPLACEMENT — a rejection that does not name what to write instead sends + * the author to the docs, and the whole point of `RETIRED_FILTER_OPERATORS` + * is that the error carries the prescription (AGENTS.md, Post-Task Checklist + * step 3). + */ + readonly mustMention: readonly string[]; +} + +/** One conformance case: evaluated to a row list, or refused. */ +export type FilterTextCase = FilterTextRowsCase | FilterTextRejectionCase; + +/** + * The cases. Ordered by what they pin: the ASCII fold, its boundary, comparand + * literalness, the case-sensitive family, then the refusals. + */ +export const FILTER_TEXT_CASES: readonly FilterTextCase[] = [ + // ── `$icontains` folds ASCII case, in both directions ────────────────────── + { + name: '$icontains matches an upper-case row from a lower-case comparand', + filter: { name: { $icontains: 'acme' } }, + expected: ['1', '2'], + note: 'The fold has to run on BOTH sides — comparing a folded comparand against a raw column matches only row 2.', + }, + { + name: '$icontains matches a lower-case row from an upper-case comparand', + filter: { name: { $icontains: 'ACME' } }, + expected: ['1', '2'], + }, + + // ── The ASCII-ONLY boundary (#4706 Q1 = A) ───────────────────────────────── + // + // These two are the contract itself, not an edge case. A backend folding the + // whole Unicode range (JS `toLowerCase()`, mongo's `$options: 'i'`) answers + // ['3','4'] to BOTH, and is wrong on both — not because Unicode folding is + // worse, but because SQLite cannot do it, so a protocol that promised it + // would be promising what three of five backends cannot deliver. + { + name: 'ASCII-only: a lower-case non-ASCII comparand does NOT match its upper-case row', + filter: { name: { $icontains: 'café' } }, + expected: ['4'], + note: 'É does not fold to é. Row 3 (CAFÉ) must NOT match. A Unicode-folding backend returns [3,4] and fails here.', + }, + { + name: 'ASCII-only: an upper-case non-ASCII comparand does NOT match its lower-case row', + filter: { name: { $icontains: 'CAFÉ' } }, + expected: ['3'], + note: 'The mirror of the case above — the ASCII letters fold, É does not, so exactly one row survives each direction.', + }, + + // ── The comparand is LITERAL: no LIKE wildcards, no regex metacharacters ─── + // + // The escaping discipline is `applyLike`'s (driver-sql), and #5589's dialect + // matrix is why it is pinned per character rather than once. + { + name: '$icontains treats % as a literal character, not a LIKE wildcard', + filter: { name: { $icontains: '100%' } }, + expected: ['5'], + note: 'An unescaped comparand compiles to LIKE \'%100%%\', which also matches row 6 (100X match).', + }, + { + name: '$icontains treats _ as a literal character, not a single-character wildcard', + filter: { name: { $icontains: 'a_b' } }, + expected: ['7'], + note: 'An unescaped _ matches any one character, so LIKE \'%a_b%\' also returns rows 8 (axb) and 9 (a.b).', + }, + { + name: '$icontains treats . as a literal character, not a regex metacharacter', + filter: { name: { $icontains: 'a.b' } }, + expected: ['9'], + note: 'This is the `$regex` defect restated as a requirement: on a regex-evaluating backend "a.b" also matched rows 7 and 8.', + }, + { + name: '$contains treats _ as a literal character too', + filter: { name: { $contains: 'a_b' } }, + expected: ['7'], + note: 'The literal-comparand rule is the operator family\'s, not `$icontains`\'s alone.', + }, + + // ── The `$contains` family is CASE-SENSITIVE (#4706 Q2 = A) ──────────────── + // + // Supersedes `filter.zod.ts`\'s former "Case sensitivity should be handled at + // backend level". Two of five backends already answer this way + // (driver-memory, formula); the SQL family\'s LIKE and mongo\'s hardcoded + // `$options: 'i'` are #5702\'s work. + { + name: '$contains is case-SENSITIVE — a lower-case comparand misses the upper-case row', + filter: { name: { $contains: 'acme' } }, + expected: ['2'], + note: 'Row 1 (ACME Corp) must NOT match. On SQLite/turso today LIKE folds ASCII and returns both.', + }, + { + name: '$contains is case-SENSITIVE — an upper-case comparand misses the lower-case row', + filter: { name: { $contains: 'ACME' } }, + expected: ['1'], + }, + { + name: '$startsWith is case-SENSITIVE', + filter: { name: { $startsWith: 'ACME' } }, + expected: ['1'], + }, + { + name: '$endsWith is case-SENSITIVE', + filter: { name: { $endsWith: 'corp' } }, + expected: ['2'], + note: 'Row 1 ends with "Corp" — a folding backend returns both and cannot be told apart from a working one by count alone.', + }, + { + name: '$notContains is case-SENSITIVE, and negation does not widen it', + filter: { name: { $notContains: 'acme' } }, + expected: ['1', '3', '4', '5', '6', '7', '8', '9'], + note: 'Row 1 is EXCLUDED from the negation only if the positive form excluded it — the case rule has to hold under $not too.', + }, + + // ── Refusals ────────────────────────────────────────────────────────────── + { + name: '$regex is REFUSED, and the refusal names $icontains', + filter: { name: { $regex: 'ac.*' } }, + expectRejection: true, + code: 'INVALID_FILTER', + mustMention: ['$regex', '$icontains'], + note: 'Not `expected: []`. Answering zero rows is what driver-memory already did for an invalid pattern — the silent wrong answer #4706 retired the operator over.', + }, + { + name: '$regex with $options is REFUSED as one mistake, not two', + filter: { name: { $regex: '^acme', $options: 'i' } }, + expectRejection: true, + code: 'INVALID_FILTER', + mustMention: ['$regex', '$options', '$icontains'], + note: 'The exact shape plugin-auth\'s adapter can emit, and the one `$icontains` replaces one-for-one. #5710 flips that producer BEFORE any backend enrols this case. "One mistake" is about the AUTHOR\'s fix being single (write $icontains), not about the message naming one key: it must name BOTH retired spellings, or an author who fixes only $regex trips the dangling-$options refusal on the next attempt.', + }, + { + name: 'a dangling $options with no $regex is REFUSED', + filter: { name: { $options: 'i' } }, + expectRejection: true, + code: 'INVALID_FILTER', + mustMention: ['$options', '$icontains'], + note: 'It was a modifier, never a predicate: on its own it constrained nothing, so accepting it silently widens.', + }, + { + name: 'an empty $icontains comparand is REFUSED', + filter: { name: { $icontains: '' } }, + expectRejection: true, + code: 'INVALID_FILTER', + mustMention: ['$icontains'], + note: 'Every row contains the empty substring, so evaluating it is a predicate that constrains nothing — the widening #5240 refused `{ field: {} }` over, one level in.', + }, + { + name: 'a non-string $icontains comparand is REFUSED', + filter: { name: { $icontains: 42 } }, + expectRejection: true, + code: 'INVALID_FILTER', + mustMention: ['$icontains'], + note: 'Coercing 42 to "42" would answer a query nobody wrote; the declared comparand type is string.', + }, +] as const; diff --git a/packages/spec/src/data/filter.zod.ts b/packages/spec/src/data/filter.zod.ts index 7bd37b802d..fa8cc921bd 100644 --- a/packages/spec/src/data/filter.zod.ts +++ b/packages/spec/src/data/filter.zod.ts @@ -136,20 +136,103 @@ export const RangeOperatorSchema = lazySchema(() => z.object({ /** * String pattern matching operators. - * Note: Case sensitivity should be handled at backend level. + * + * ## Case sensitivity IS part of the contract (#5701, maintainer ruling 2026-08-06) + * + * **`$contains` / `$notContains` / `$startsWith` / `$endsWith` compare + * CASE-SENSITIVELY. `$icontains` is the case-INSENSITIVE twin, and its folding + * domain is ASCII (`A-Z` against `a-z`) and nothing else.** + * + * ### This SUPERSEDES a recorded decision (Prime Directive #13) + * + * Until 2026-08-06 this docblock read, in full: + * + * > Note: Case sensitivity should be handled at backend level. + * + * That sentence was not an omission — it was a written-down NON-guarantee, and + * the #4706 ruling (Q2 = A, transcribed on #5701) withdraws it. It is quoted + * here rather than deleted because reversing a recorded decision is itself a + * decision, and the next author needs to find the reversal from the sentence + * they remember. + * + * What that non-guarantee actually bought, measured surface by surface before + * the ruling: + * + * | surface | `$contains` case behaviour | mechanism | + * |---|---|---| + * | `formula` `matchesFilterCondition` | SENSITIVE | `actual.includes(v)` | + * | `driver-memory` — query path and analytics face | INSENSITIVE, full Unicode | `new RegExp(escapeRegex(v), 'i')` | + * | `driver-memory` — reference matcher (`memory-matcher`) | SENSITIVE | `value.includes(target)` | + * | `driver-mongodb` | INSENSITIVE, full Unicode | hardcoded `$options: 'i'` | + * | `driver-sql` family | the DIALECT's | `LIKE '%v%'` — ASCII-insensitive on SQLite (so also turso and sqlite-wasm), sensitive on Postgres, collation-dependent on MySQL | + * + * One declared operator, three different answers, selected by which backend ran + * the query — and `driver-memory` alone accounts for two of them, so the answer + * could change without changing driver. An author could not tell from the + * operator name which one they were getting, and neither could a generated + * filter. That is what a written-down "handled at backend level" costs once + * there is more than one backend. + * + * ### Why the folding domain is ASCII and not Unicode + * + * A Unicode fold cannot be delivered by every backend, so promising one would + * repeat the defect this retires rather than fix it. Measured on this repo's + * `better-sqlite3` (SQLite 3.53.4, no ICU): `LOWER(col) LIKE LOWER(?)` folds + * ASCII only, so `café` does not match `CAFÉ` and `москва` does not match + * `МОСКВА`, while the JS matchers' `toLowerCase()` folds both. Pinning the + * contract at ASCII is the one domain all five backends can actually deliver. + * + * **The boundary, stated plainly for authors: `café` does NOT match `CAFÉ`.** + * Outside `A-Z`/`a-z`, `$icontains` compares literally, exactly like + * `$contains`. An application whose users search non-ASCII text should not read + * `$icontains` as "accent- and case-blind search" — it is not one. + * + * ### Implementation status — declared here, NOT yet answered by any backend + * + * This PR is the contract half of the #4706 ruling and deliberately ships no + * runtime behaviour (#5701). No driver evaluates `$icontains` today; all five + * refuse it, loudly, as an operator they do not implement — which is the + * fail-closed direction and stays true until #5702 lands the lowerings. The + * same issue carries the `$contains`-family alignment the ruling above + * requires (SQLite/turso `LIKE` made case-exact, mongo's hardcoded `'i'` + * removed). Until then the sentence above is the DECLARATION and #5702 is the + * gap; `FILTER_TEXT_CASES` (`filter-text-conformance.ts`) is the standard that + * measures the gap, and the driver-conformance ledger carries one DEBT row per + * backend so the gap is counted rather than assumed. + * + * @see FILTER_TEXT_CASES — the conformance standard for every operator here. + * @see RETIRED_FILTER_OPERATORS — why `$regex` is not in this list. + * @see https://github.com/objectstack-ai/objectstack/issues/4706 (the ruling) + * @see https://github.com/objectstack-ai/objectstack/issues/5702 (the backends) */ export const StringOperatorSchema = lazySchema(() => z.object({ - /** Contains substring - SQL: LIKE %?% | MongoDB: $regex */ + /** Contains substring, CASE-SENSITIVELY - SQL: LIKE %?% (case-exact) */ $contains: z.string().optional(), - - /** Does not contain substring - SQL: NOT LIKE %?% | MongoDB: $not: $regex */ + + /** Does not contain substring, CASE-SENSITIVELY - SQL: NOT LIKE %?% (case-exact) */ $notContains: z.string().optional(), - - /** Starts with prefix - SQL: LIKE ?% | MongoDB: $regex */ + + /** Starts with prefix, CASE-SENSITIVELY - SQL: LIKE ?% (case-exact) */ $startsWith: z.string().optional(), - - /** Ends with suffix - SQL: LIKE %? | MongoDB: $regex */ + + /** Ends with suffix, CASE-SENSITIVELY - SQL: LIKE %? (case-exact) */ $endsWith: z.string().optional(), + + /** + * Contains substring, IGNORING ASCII case (#5701). The replacement for the + * retired `$regex` — see {@link RETIRED_FILTER_OPERATORS}. + */ + $icontains: z.string().optional().describe( + 'Contains substring, ignoring case — but ONLY ASCII case (A-Z against a-z). ' + + 'Every other character compares literally, so "café" does NOT match "CAFÉ" ' + + 'and "москва" does not match "МОСКВА". The domain is ASCII because that is ' + + 'the one fold all five backends can deliver: SQLite (and therefore turso and ' + + 'sqlite-wasm) folds ASCII only, so a Unicode promise here would be a ' + + 'guarantee three of the five could not keep. The comparand is matched ' + + 'LITERALLY — "%", "_" and regex metacharacters are ordinary characters, not ' + + 'wildcards. Case-SENSITIVE containment is $contains. [#5701: declared by the ' + + 'protocol; the driver lowerings land with #5702.]' + ), })); // ============================================================================ @@ -194,12 +277,14 @@ export const FieldOperatorsSchema = lazySchema(() => z.object({ z.union([z.number(), z.date(), FieldReferenceSchema]) ]).optional(), - // String-specific + // String-specific. Case-SENSITIVE, except `$icontains` which folds ASCII case + // only — see {@link StringOperatorSchema} for the contract and its boundary. $contains: z.string().optional(), $notContains: z.string().optional(), $startsWith: z.string().optional(), $endsWith: z.string().optional(), - + $icontains: z.string().optional(), + // Special $null: z.boolean().optional(), $exists: z.boolean().optional(), @@ -390,6 +475,8 @@ export type Filter = { $notContains?: T[K] extends string ? string : never; $startsWith?: T[K] extends string ? string : never; $endsWith?: T[K] extends string ? string : never; + /** Case-insensitive containment, ASCII fold only — see {@link StringOperatorSchema}. */ + $icontains?: T[K] extends string ? string : never; $null?: boolean; $exists?: boolean; } @@ -951,8 +1038,49 @@ export const FilterArraySchema: z.ZodType = z.lazy(() // ============================================================================ /** - * All supported operator keys. - * Useful for validation and parsing. + * The operator keys every backend is expected to EVALUATE. + * + * ## This list is a runtime allowlist, not a word list (#5701) + * + * It reads like documentation and is used like a gate. Two consumers derive + * enforcement from it rather than restating it — which is the right design, and + * is exactly why an entry here is a claim about implementations, not about + * vocabulary: + * + * - `driver-memory`'s `SUPPORTED_FIELD_OPERATORS` (`filter-refusal.ts`) is + * `new Set([...FILTER_OPERATORS, '$regex', '$options'])` — the set its shape + * gate ACCEPTS. Its matcher's `default:` arm then `break`s, on the documented + * assumption that the gate already refused anything unimplemented. + * - `service-analytics`' `objectql-echo-operator-coverage.test.ts` asserts its + * compiler renders a predicate for every member. + * + * So a name added here before any backend implements it does not merely + * document an intention. Measured on this branch by adding `$icontains` to this + * array and rebuilding: `SUPPORTED_FIELD_OPERATORS.has('$icontains')` became + * `true`, driver-memory's gate stopped refusing it, and + * `match({ name: 'zzz' }, { name: { $icontains: 'acme' } })` returned **`true`** + * — the predicate silently dropped, every row matched. A dropped predicate does + * not narrow a query, it WIDENS it, and on an RLS read scope that is a + * permission bypass rather than a degraded feature (#3948). + * + * ## `$icontains` is DECLARED but deliberately NOT here yet + * + * {@link StringOperatorSchema}, {@link FieldOperatorsSchema} and {@link Filter} + * declare `$icontains` (#5701, the contract half of the #4706 ruling). This + * array does not, and the difference is deliberate rather than an oversight: + * those three are declaration and TYPE surfaces with no runtime allowlist + * reader (verified — `NormalizedFilterSchema` is their only consumer, and + * nothing parses a filter through it at runtime), so declaring there is inert. + * Adding it HERE would flip driver-memory from a loud refusal to the silent + * widening measured above, before a single backend can answer the operator. + * + * **`$icontains` joins this array in the PR that implements it (#5702), not + * before.** `filter-operator-vocabulary.test.ts` pins the difference between + * the two surfaces at exactly `{ $icontains }`, so this staging cannot silently + * grow a second member, and clearing it is what makes that pin fail. + * + * Retired operators (`$regex`, `$options`) are not here either, and never were. + * Their prescriptions live in {@link RETIRED_FILTER_OPERATORS}. */ export const FILTER_OPERATORS = [ // Equality @@ -980,3 +1108,89 @@ export const ALL_OPERATORS = [...FILTER_OPERATORS, ...LOGICAL_OPERATORS] as cons export type FilterOperatorKey = typeof FILTER_OPERATORS[number]; export type LogicalOperatorKey = typeof LOGICAL_OPERATORS[number]; export type OperatorKey = typeof ALL_OPERATORS[number]; + +// ============================================================================ +// Retired Operators — the prescriptions a refusal prints (#5701) +// ============================================================================ + +/** The prescription for one retired filter operator. */ +export interface RetiredFilterOperatorGuidance { + /** + * The operator that replaces it, when one does. Absent when the retirement + * has no successor and the fix is to restructure the query. + */ + readonly to?: string; + /** + * The upgrade prescription, written as an instruction. This string IS the + * migration doc for whoever hits it — a refusal is expected to print it + * verbatim rather than paraphrase, so that five refusal sites say one thing. + */ + readonly why: string; +} + +/** + * Filter operators that were REMOVED from the protocol, and what to write + * instead (#5701, the #4706 ruling's contract half). + * + * ## What this is, and what it deliberately is not + * + * It is **data**: a lookup table, no behaviour (Prime Directive #2). It does + * not reject anything, and this file's schemas are unchanged by its existence — + * {@link FilterConditionSchema} still accepts any `$`-key, because narrowing it + * to a closed vocabulary is structurally impossible here (a nested relation + * constraint `{ profile: { verified: true } }` and an operator object are the + * same shape) and would reach every filter consumer at once. The #4706 ruling + * took that trade explicitly: the spec declares, the existing refusal sites + * enforce. + * + * Those sites are the five that already refuse unknown operators today — + * `driver-sql`'s `default:` arm, `driver-turso`'s remote transport, + * `driver-memory`'s `filter-refusal.ts`, `driver-mongodb`'s + * `translateFieldOperators`, and `objectql`'s `having` — and the point of one + * table is that they stop each writing their own sentence. Wiring them to it is + * **#5702**, deliberately not this PR: `$regex` still has one live producer + * (`plugin-auth`'s ObjectQL adapter, on the authentication path), so a refusal + * landing before #5710 flips that producer would break sign-in. Hard order: + * **#5710 flips the producer, then #5702 turns these strings into refusals.** + * + * ## Why `$regex` was retired rather than implemented (#4706) + * + * It was never in {@link FILTER_OPERATORS} — it was an undeclared operator that + * one producer emitted and four consumers grew arms for, each reading it + * differently. `driver-sql` compiled it to a substring `LIKE` with the value + * LIKE-escaped, so it was not a regex at all: `a.b` matched only the literal + * `a.b`. `driver-memory` ran it as a real `RegExp`, so the same filter matched + * `axb` too, and an invalid pattern was caught and answered `false` — zero rows, + * in silence. Implementing a real regex on all five was rejected as + * structurally impossible: turso's remote transport speaks a wire protocol with + * no way to register a SQLite `REGEXP` function. + * + * The replacement is the case-insensitive containment the one real producer + * actually wanted: {@link StringOperatorSchema}'s `$icontains`. + */ +export const RETIRED_FILTER_OPERATORS: Readonly< + Record +> = Object.freeze({ + $regex: { + to: '$icontains', + why: + '`$regex` was never declared by the Filter Protocol and is retired (#4706). It could not ' + + 'mean one thing across the backends: driver-sql compiled it to a LIKE-escaped substring ' + + 'match (so "a.b" matched only the literal "a.b"), driver-memory evaluated it as a real ' + + 'RegExp (so it also matched "axb", and an invalid pattern silently matched nothing), and ' + + 'a real regex is not implementable on all five — driver-turso\'s remote transport cannot ' + + 'register a SQLite REGEXP function over its wire protocol. Write `$icontains` for the ' + + 'case-insensitive substring search this was almost always used for (ASCII case fold; the ' + + 'comparand is matched literally, so "." and "%" are ordinary characters), or `$contains` ' + + 'for a case-sensitive one. A pattern that genuinely needs a regex has no filter-level ' + + 'replacement — narrow with the declared operators and match in application code.', + }, + $options: { + to: '$icontains', + why: + '`$options` was never a predicate — it was the regex-flags companion to `$regex`, which is ' + + 'retired (#4706). Its only real use was `$options: "i"` for a case-insensitive match: ' + + 'write `$icontains` instead, which says that in the operator name and folds ASCII case on ' + + 'every backend. On its own, with no `$regex` beside it, it never constrained anything.', + }, +}); diff --git a/packages/spec/src/data/index.ts b/packages/spec/src/data/index.ts index c468d8b9f7..a5538551c6 100644 --- a/packages/spec/src/data/index.ts +++ b/packages/spec/src/data/index.ts @@ -7,6 +7,13 @@ export * from './filter.zod'; // against, so they cannot drift apart again (#3774; the fifth — MongoDB's // `translateFilter` — was enrolled by #4405). export * from './filter-logic-conformance'; +// Canonical conformance cases for the filter TEXT operators — case folding +// (ASCII-only, #4706 Q1), literal comparands (no LIKE wildcards, no regex +// metacharacters), and the refusal of the retired `$regex`/`$options`. A +// sibling of the logic table rather than rows inside it: those two axes are +// explicitly out of that table's scope, and this one needs an `expectRejection` +// discriminant it deliberately never grew (#5701). +export * from './filter-text-conformance'; export * from './temporal-conformance'; // Canonical conformance cases for deterministic paged reads — the standard // every driver's `find()` is held to whenever `limit`/`offset` slice the result diff --git a/scripts/check-driver-conformance.mjs b/scripts/check-driver-conformance.mjs index fbb1867367..a8aa25540a 100644 --- a/scripts/check-driver-conformance.mjs +++ b/scripts/check-driver-conformance.mjs @@ -129,6 +129,11 @@ const CASE_SETS = [ marker: 'PAGINATION_UNORDERED_CASES', what: 'an UNSORTED paged read is a partition too — #4363', }, + { + file: 'filter-text-conformance.ts', + marker: 'FILTER_TEXT_CASES', + what: 'text operators: ASCII-only case folding, literal comparands, `$regex` refused — #4706/#5701', + }, ]; // ── The ledger ────────────────────────────────────────────────────────────── @@ -137,7 +142,14 @@ const CASE_SETS = [ // covered, is not yet) or EXEMPT (cannot meaningfully apply). Both are measured // claims; neither is a default. // -// EMPTY, as of #5590 — every cell of the matrix is covered by a suite. Five +// It was EMPTY between #5590 and #5701. It is not any more: #5701 added the +// FILTER_TEXT_CASES column ahead of its implementations, on purpose, and its +// five rows are documented under the historical notes below — read those first, +// they are the current live entries. The paragraph that follows is the record of +// how the ledger reached empty, kept because every clearing it describes was +// done the same way the five new rows must be. +// +// EMPTY, as of #5590 — every cell of the matrix was covered by a suite. Five // entries have passed through here across two generations, and every one of // them was cleared the same way: by writing the suite, never by the argument // that predicted the suite was unnecessary. @@ -199,7 +211,9 @@ const CASE_SETS = [ // // An empty ledger is the intended steady state, not a reason to delete the // mechanism: the next driver that arrives uncovered fails CONSUMED and lands -// its measured entry here. +// its measured entry here. #5701 is the other way that happens — a new +// CASE-SET, rather than a new driver, arriving ahead of the work — and it is +// what the five FILTER_TEXT_CASES rows are. // // ## What driver-mongodb's cells mean since #5517 — read this before trusting them // @@ -234,7 +248,121 @@ const CASE_SETS = [ // investment is frozen (#5499). Un-freezing it is what should re-run these cells // in CI; until then, this note is the honest state of the mongo column. -const LEDGER = []; +// ## FILTER_TEXT_CASES: five DEBT rows, opened by #5701 — read this first +// +// The ledger was EMPTY (see the note above) until `FILTER_TEXT_CASES` arrived. +// These five rows are not a regression in coverage: the case-set is the +// CONTRACT half of the #4706 ruling, landed deliberately ahead of every +// implementation, and one row per driver is what makes "ahead of" a counted +// fact instead of an assumption. They are cleared by #5702, one suite at a +// time, exactly the way the five before them were. +// +// What the case-set demands, and why nobody can answer it yet: +// +// 1. `$icontains` — a NEW operator (ASCII-only case fold). No backend has an +// arm for it. All five refuse it today, which is the fail-closed +// direction, so this is a missing capability rather than a live defect. +// 2. `$contains` / `$startsWith` / `$endsWith` / `$notContains` must be +// case-SENSITIVE (#4706 Q2 = A, superseding `filter.zod.ts`'s former +// "Case sensitivity should be handled at backend level"). NO driver +// delivers this on its live query path today: driver-memory and +// driver-mongodb fold the full Unicode range, and the SQL family follows +// its dialect (SQLite — so also turso and sqlite-wasm — folds ASCII; +// Postgres happens to be case-exact already; MySQL depends on collation). +// The one surface that does compare case-sensitively is driver-memory's +// REFERENCE matcher, which is not the path a query takes — see that row. +// 3. `$regex` / `$options` must be REFUSED, naming `$icontains`. Exactly one +// driver refuses `$regex` today (driver-mongodb, via its `default:` arm — +// though outside the ADR-0112 envelope the case-set requires; see its +// row). The other four accept it: driver-memory evaluates it as a real +// RegExp, and driver-sql / driver-sqlite-wasm / driver-turso compile it to +// a substring LIKE. That is deliberate, not neglect — `plugin-auth`'s +// ObjectQL adapter still emits it on the AUTHENTICATION path. **#5710 +// flips that producer before any of these four cells may be cleared**; a +// driver that refuses `$regex` first breaks sign-in. +// +// Each row's `why` is what that driver does TODAY, measured on this branch by +// reading the compiler and (for driver-memory) executing it. Nothing here is +// predicted. + +const LEDGER = [ + { + driver: 'driver-memory', + marker: 'FILTER_TEXT_CASES', + kind: 'DEBT', + why: + 'Measured, and the one row where "which face" changes the answer — do NOT take a single reading here. ' + + 'The QUERY path (`find()` -> `normalizeFieldOperators`, and the analytics face via ' + + '`filterSubstringPattern`) lowers `$contains` to `new RegExp(escapeRegex(v), "i")`: literal comparand ' + + '(requirement 2\'s escaping half holds) but case-INSENSITIVE over the whole Unicode range, which ' + + 'fails requirement 2 and overshoots requirement 1\'s ASCII boundary. The reference matcher ' + + '(`memory-matcher.ts` `match()`, the record-at-a-time evaluator `filter-logic-conformance.ts` counts ' + + 'as a backend) uses String.prototype.includes and is case-SENSITIVE — i.e. this package answers one ' + + '`$contains` two ways today, the divergence class #5374 fixed between the other two faces. Whichever ' + + 'suite clears this cell has to pick one and align both. `$icontains` is refused on both faces ' + + "(`SUPPORTED_FIELD_OPERATORS` derives from the spec's FILTER_OPERATORS, which deliberately does not " + + 'carry it yet) — unimplemented but fail-closed. `$regex`/`$options` are ACCEPTED and evaluated as a ' + + 'real RegExp, the opposite of requirement 3 and the only live regex evaluator in the repo; that arm ' + + "exists for plugin-auth's adapter and cannot be removed before #5710.", + issue: 'https://github.com/objectstack-ai/objectstack/issues/5702', + }, + { + driver: 'driver-sql', + marker: 'FILTER_TEXT_CASES', + kind: 'DEBT', + why: + 'Measured: `applyLike` escapes `\\`, `%` and `_` and binds ESCAPE, so the literal-comparand cases would ' + + 'pass today. Case sensitivity is the DIALECT\'s, not the driver\'s — SQLite\'s LIKE folds ASCII, ' + + 'Postgres does not, MySQL follows its collation — so requirement 2 fails on two of three dialects and ' + + 'needs a case-exact comparison (GLOB / instr() / a binary collation), not a flag. `$icontains` hits the ' + + '`default:` arm and is refused in the ADR-0112 envelope. `$regex` is COMPILED (to the same substring ' + + 'LIKE), not refused.', + issue: 'https://github.com/objectstack-ai/objectstack/issues/5702', + }, + { + driver: 'driver-sqlite-wasm', + marker: 'FILTER_TEXT_CASES', + kind: 'DEBT', + why: + 'Measured: `SqliteWasmDriver extends SqlDriver`, so every fact in the driver-sql row applies unchanged, ' + + 'with the dialect pinned to SQLite — i.e. requirement 2 fails here specifically because LIKE folds ' + + 'ASCII case. Tracked as DEBT rather than EXEMPT for the reason its FILTER_LOGIC row was: "inherits, ' + + 'therefore fine" is the assumption these suites exist to disprove, and what this one would add is the ' + + 'sql.js engine EXECUTING the compiled predicate, which is where a collation choice actually shows up.', + issue: 'https://github.com/objectstack-ai/objectstack/issues/5702', + }, + { + driver: 'driver-turso', + marker: 'FILTER_TEXT_CASES', + kind: 'DEBT', + why: + 'Measured: DUAL-TRANSPORT, so this cell needs TWO suites like its three predecessors. Local/replica ' + + 'inherits SqlDriver (see the driver-sql row) on the SQLite dialect. Remote does not go through knex at ' + + 'all: `remote-transport.ts` carries its own hand-written SUPPORTED_FILTER_OPERATORS — which lists ' + + '`$regex` and not `$icontains` — and its own LIKE assembly. Both transports therefore fail requirement ' + + '2 (SQLite LIKE folds ASCII) and requirement 3, and refuse `$icontains` today.', + issue: 'https://github.com/objectstack-ai/objectstack/issues/5702', + }, + { + driver: 'driver-mongodb', + marker: 'FILTER_TEXT_CASES', + kind: 'DEBT', + why: + 'Measured: the FURTHEST from the ruling. `translateFieldOperators` lowers `$contains`/`$startsWith`/' + + '`$endsWith`/`$notContains` to `$regex` with a HARDCODED `$options: "i"`, i.e. case-insensitive over ' + + 'the whole Unicode range — requirement 2 inverted, and requirement 1\'s ASCII-only boundary violated in ' + + 'the same expression. `escapeRegex` does escape metacharacters, so the literal-comparand cases hold. ' + + 'An incoming `$regex` reaches the `default:` arm and IS refused (mongo is the only backend that ' + + 'already satisfies requirement 3), and `$icontains` is refused there too — but that arm throws a bare ' + + '`new Error("[mongodb] unsupported filter operator ...")`, NOT the ADR-0112 envelope its own ' + + '`unsupportedFilterError` helper (same file, used by three other refusals here) produces. The ' + + "case-set requires `code: 'INVALID_FILTER'`, so clearing this cell means routing that arm through the " + + 'helper as well. Note this package is in the ' + + '#5499 frozen family: its real-mongod suites are opt-in, so whatever clears this cell needs a ' + + 'server-free half like `mongodb-filter-logic-translation.test.ts` has.', + issue: 'https://github.com/objectstack-ai/objectstack/issues/5702', + }, +]; // ── Discovery ───────────────────────────────────────────────────────────────