Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions .changeset/aggregation-array-string-agg-removed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
---
"@objectstack/spec": major
"@objectstack/objectql": minor
"@objectstack/service-analytics": patch
---

refactor(spec)!: retire `array_agg` / `string_agg` from `AggregationFunction` — `count_distinct` deliberately kept (#6188, ADR-0049)

`AggregationFunction` declared eight functions; the SQL family compiles five.
`SqlDriver.mapAggregateFunc` and the Turso `RemoteTransport.aggregate` each lower
`count`/`sum`/`avg`/`min`/`max` and route everything else to one refusal, so
three of the eight were declared-but-unenforced against the backends this
platform targets — and, worse, the *set* each backend implemented was different,
so "which aggregations can I use" had no answer an author could read off the
schema.

What makes these two sharper than an ordinary inert declaration is that another
package had to carry a denylist for them. `service-analytics` subtracted
`array_agg` and `string_agg` by name in `UNSUPPORTED_AGGREGATES`, because
without that subtraction they reached the Cube strategy's `default` and came
back as `COUNT(*)` — **a row count in place of the value the author asked for**,
with no error and no log (objectui#2945).

**The three unlowered functions were SPLIT, not retired as a block** (maintainer
ruling, 2026-08-07):

- **`count_distinct` STAYS** and takes ADR-0049's *enforce* leg. It is a
dashboard staple with one portable lowering (`COUNT(DISTINCT x)`), and
`service-analytics` lowers it already; the SQL-driver implementation follows
on its own card. Its declaration leads its implementation here by decision,
not by drift.
- **`array_agg` / `string_agg` take the *remove* leg.** Display conveniences
with no measured pull, and `string_agg` never had one shape to lower to at
all: the delimiter is a second argument in PostgreSQL, a `SEPARATOR` clause in
MySQL and a differently named function in SQL Server.

FROM → TO, both authoring surfaces:

| Was | Now |
|:--|:--|
| `aggregations: [{ function: 'array_agg', field: 'tag', alias: 'tags' }]` | no replacement — read the rows with an ordinary `fields` query and shape them in the caller, or materialise the roll-up as a stored field |
| `aggregations: [{ function: 'string_agg', field: 'name', alias: 'names' }]` | as above |
| `measures: [{ name: 'tags', aggregate: 'array_agg', field: 'tag' }]` | delete the measure — `compileDataset` already refused it by name, so it never produced a number |

The retirement kit:

- This is an enum **VALUE** retirement, so there is no `retiredKey()` tombstone:
the enum's own error map carries the prescription, keyed on the received value
so that only the two spellings which used to be legal are told they "were
removed" (the `crypto.hash` / `HookBodyCapability` precedent, #4391). A
mis-spelling still gets zod's list of the legal functions. For the same reason
nothing lands in `RETIRED_KEYS_BY_MAJOR` and the four surface ratchets are
byte-identical — no def and no authorable key changed.
- **ADR-0087 D2 conversion + D3 chain step**
(`dataset-measure-array-string-agg-removed`): `os migrate meta --from 16`
drops any `dataset.measures[]` declaring a retired aggregate, plus any derived
measure the drop strands, with a notice each. The measure is dropped rather
than stripped down because one with neither `aggregate` nor `derived` fails
the dataset's own refinement — a conversion whose output cannot parse is worse
than none.
- **D3 semantic entry** (`query-array-string-agg-retired`) for
`QueryAST.aggregations[].function`: a request surface, never stored, so there
is no source for the chain to rewrite and callers move their own queries.
- The engine's in-memory fallback (`@objectstack/objectql`) drops its arms for
both functions — a `switch` case on a value the enum no longer has does not
type-check, and a dead arm is how a retired vocabulary returns by accident.
- `service-analytics`' `UNSUPPORTED_AGGREGATES` is now **empty and kept**: it is
half of an arithmetic the lockstep suite enforces (`SUPPORTED = spec
vocabulary − this`), which is what stops the next aggregate added to the spec
from silently reaching that `COUNT(*)` default.

**Behaviour that actually changes** — this is the rare narrowing that removes
reachable behaviour, and it is worth stating plainly: on `driver-mongodb` and on
the engine's in-memory fallback these two DID compute. A raw QueryAST
aggregation against those backends returned an array or a joined string and will
now be refused at parse. That unpredictability is precisely what the ruling
ended — an aggregation that worked on one backend and failed on another is not a
capability — and both of those backends are inside the #5499 freeze. Their code
is untouched; it is simply no longer reachable through a spec-valid request. On
the dataset path nothing changes: `compileDataset` refused both by name already.

<!-- adr-0087: registered query-array-string-agg-retired, dataset-measure-array-string-agg-removed -->
24 changes: 16 additions & 8 deletions content/docs/data-modeling/queries.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -369,16 +369,24 @@ querying the related object directly.
| `min` | Minimum value | `{ function: 'min', field: 'price', alias: 'min_price' }` |
| `max` | Maximum value | `{ function: 'max', field: 'price', alias: 'max_price' }` |
| `count_distinct` | Count unique values | `{ function: 'count_distinct', field: 'category', alias: 'categories' }` |
| `array_agg` | Collect into array | `{ function: 'array_agg', field: 'tag', alias: 'all_tags' }` |
| `string_agg` | Concatenate strings | `{ function: 'string_agg', field: 'name', alias: 'names' }` |

<Callout type="warn">
`count_distinct` / `array_agg` / `string_agg` are only fully supported on the MongoDB driver.
The SQL drivers map only `count`/`sum`/`avg`/`min`/`max` and refuse all three as a
**capability gap** — `501 NOT_IMPLEMENTED`, "declared but not implemented by this backend"
— rather than as a caller mistake, because the query is spelled correctly and the gap is
the backend's (#5907). The in-memory driver's aggregator silently returns `null` for them.
Avoid these three on SQL- or memory-backed objects.
`count_distinct` is not yet lowered by the SQL drivers. They map
`count`/`sum`/`avg`/`min`/`max` and refuse it as a **capability gap** —
`501 NOT_IMPLEMENTED`, "declared but not implemented by this backend" — rather than as a
caller mistake, because the query is spelled correctly and the gap is the backend's
(#5907). It works on the MongoDB driver and on the engine's in-memory fallback, and its
SQL lowering (`COUNT(DISTINCT field)`) is scheduled.
</Callout>

<Callout type="warn">
**Removed in 17.** `array_agg` and `string_agg` were declared here and compiled by no SQL
backend, which left "what can this backend actually compute" unpredictable to the author.
Both were retired (#6188, ADR-0049 enforce-or-remove): a query carrying either is now
refused at parse with a prescription. There is no replacement in the query vocabulary —
read the rows with an ordinary `fields` query and shape them in the caller, or materialise
the roll-up as a stored field. `os migrate meta --from 16` rewrites affected dataset
measures.
</Callout>

<Callout type="info">
Expand Down
3 changes: 1 addition & 2 deletions content/docs/kernel/contracts/data-engine.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -354,8 +354,7 @@ interface EngineAggregateOptions {
}

interface AggregationNode {
function: 'count' | 'sum' | 'avg' | 'min' | 'max'
| 'count_distinct' | 'array_agg' | 'string_agg';
function: 'count' | 'sum' | 'avg' | 'min' | 'max' | 'count_distinct';
field?: string; // Field to aggregate (optional for COUNT(*))
alias: string; // Result column alias
distinct?: boolean; // Apply DISTINCT before aggregation
Expand Down
26 changes: 18 additions & 8 deletions content/docs/protocol/objectql/query-syntax.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -117,8 +117,7 @@ interface SortNode {

// AggregationNode — aggregation definition
interface AggregationNode {
function: 'count' | 'sum' | 'avg' | 'min' | 'max'
| 'count_distinct' | 'array_agg' | 'string_agg';
function: 'count' | 'sum' | 'avg' | 'min' | 'max' | 'count_distinct';
field?: string; // optional for COUNT(*)
alias: string; // result column alias
distinct?: boolean; // DISTINCT before aggregation — in-memory path only
Expand Down Expand Up @@ -715,17 +714,28 @@ const query: QueryAST = {
// { count: 100, total: 5000000, average: 50000, min_amount: 10000, max_amount: 500000 }
```

**Schema enum:** `count`, `sum`, `avg`, `min`, `max`, `count_distinct`, `array_agg`, `string_agg`.
**Schema enum:** `count`, `sum`, `avg`, `min`, `max`, `count_distinct`.

<Callout type="warn">
Only `count`, `sum`, `avg`, `min`, and `max` are portable. `count_distinct`, `array_agg`
and `string_agg` are implemented by the MongoDB driver and by the engine's in-memory
aggregation fallback, but not by the SQL drivers — on `SqlDriver` (and on the Turso
driver, both transports) they are refused as a **capability gap**:
Only `count`, `sum`, `avg`, `min`, and `max` are portable today. `count_distinct` is
declared and is implemented by the MongoDB driver and by the engine's in-memory
aggregation fallback, but not yet by the SQL drivers — on `SqlDriver` (and on the Turso
driver, both transports) it is refused as a **capability gap**:
`501 NOT_IMPLEMENTED`, "declared but not implemented by this backend". That is
deliberately a different answer from a function the schema enum never declared
(`median`), which is `400 INVALID_QUERY` — the caller's mistake — so an author who
wrote `count_distinct` is never told they made a typo (#5907).
wrote `count_distinct` is never told they made a typo (#5907). Its SQL lowering
(`COUNT(DISTINCT field)`) is scheduled: the declaration leads the implementation here
by decision, not by drift.
</Callout>

<Callout type="warn">
**Removed in 17:** `array_agg` and `string_agg` were declared by this enum and compiled
by no SQL backend, so which backend could compute them was unpredictable to the author.
Both were retired (#6188, ADR-0049 enforce-or-remove) and are now refused at parse with
a prescription. There is no replacement in the query vocabulary — read the rows with an
ordinary `fields` query and shape them in the caller, or materialise the roll-up as a
stored field. `os migrate meta --from 16` rewrites affected dataset measures.
</Callout>

### Group By Multiple Fields
Expand Down
2 changes: 1 addition & 1 deletion content/docs/references/api/contract.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ const result = ApiErrorSchema.parse(data);
| **top** | `number` | optional | Alias for limit (OData compatibility) |
| **cursor** | `never` | optional | [REMOVED] `query.cursor` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no driver ever implemented keyset pagination, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). Delete the key; `QueryBuilder.cursor()` was removed with it. Express the keyset as an ordinary `where` predicate on your sort key — `where: { created_at: { $gt: last.created_at } }` with the matching `orderBy` — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record. |
| **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and a single related column is a dotted `fields` path (`fields: ['owner.name']`). |
| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct' \| 'array_agg' \| 'string_agg'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | Aggregation functions |
| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | Aggregation functions |
| **groupBy** | `(string \| { field: string; dateGranularity?: Enum<'day' \| 'week' \| 'month' \| 'quarter' \| 'year'>; alias?: string })[]` | optional | GROUP BY targets (strings or `{field, dateGranularity?}` objects for date bucketing) |
| **having** | `any` | optional | HAVING — filter over the AGGREGATED rows (aggregation aliases + groupBy projections); applied engine-side after aggregation |
| **windowFunctions** | `never` | optional | [REMOVED] `query.windowFunctions` was removed in @objectstack/spec 17 (#4286, ADR-0049) — `find()` never applied it: no engine or driver read the key on the query path, so every OVER clause it declared was silently dropped. Delete the key. Window functions are a SQL-driver capability behind `SqlDriver.findWithWindowFunctions(object, query)` (embedder-level; not on the `IDataDriver` contract or the REST surface); request-level analytics are `aggregations` + `groupBy`. |
Expand Down
6 changes: 3 additions & 3 deletions content/docs/references/data/data-engine.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Options for DataEngine.aggregate operations
| :--- | :--- | :--- | :--- |
| **method** | `'aggregate'` | ✅ | |
| **object** | `string` | ✅ | |
| **query** | `{ context?: object; where?: Record<string, any> \| any; groupBy?: string[]; aggregations?: { function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct' \| 'array_agg' \| 'string_agg'>; field?: string; alias: string; distinct?: boolean; … }[]; … }` | ✅ | |
| **query** | `{ context?: object; where?: Record<string, any> \| any; groupBy?: string[]; aggregations?: { function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]; … }` | ✅ | |


---
Expand Down Expand Up @@ -328,7 +328,7 @@ This schema accepts one of the following structures:
| :--- | :--- | :--- | :--- |
| **method** | `'aggregate'` | ✅ | |
| **object** | `string` | ✅ | |
| **query** | `{ context?: object; where?: Record<string, any> \| any; groupBy?: string[]; aggregations?: { function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct' \| 'array_agg' \| 'string_agg'>; field?: string; alias: string; distinct?: boolean; … }[]; … }` | ✅ | |
| **query** | `{ context?: object; where?: Record<string, any> \| any; groupBy?: string[]; aggregations?: { function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]; … }` | ✅ | |

---

Expand Down Expand Up @@ -467,7 +467,7 @@ QueryAST-aligned options for DataEngine.aggregate operations
| **context** | `{ userId?: string; actor?: string; attributedUserId?: string; email?: string; … }` | optional | |
| **where** | `Record<string, any> \| any` | optional | |
| **groupBy** | `string[]` | optional | |
| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct' \| 'array_agg' \| 'string_agg'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | |
| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | |
| **having** | `any` | optional | HAVING — filter over the aggregated rows (aggregation aliases + groupBy projections); applied engine-side after aggregation |
| **timezone** | `string` | optional | |

Expand Down
6 changes: 2 additions & 4 deletions content/docs/references/data/query.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@ const result = AggregationFunction.parse(data);
* `min`
* `max`
* `count_distinct`
* `array_agg`
* `string_agg`


---
Expand All @@ -47,7 +45,7 @@ const result = AggregationFunction.parse(data);

| Property | Type | Required | Description |
| :--- | :--- | :--- | :--- |
| **function** | `Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct' \| 'array_agg' \| 'string_agg'>` | ✅ | Aggregation function |
| **function** | `Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>` | ✅ | Aggregation function |
| **field** | `string` | optional | Field to aggregate (optional for COUNT(*)) |
| **alias** | `string` | ✅ | Result column alias |
| **distinct** | `boolean` | optional | Apply DISTINCT before aggregation |
Expand Down Expand Up @@ -134,7 +132,7 @@ Type: `string`
| **top** | `number` | optional | Alias for limit (OData compatibility) |
| **cursor** | `never` | optional | [REMOVED] `query.cursor` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no driver ever implemented keyset pagination, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). Delete the key; `QueryBuilder.cursor()` was removed with it. Express the keyset as an ordinary `where` predicate on your sort key — `where: { created_at: { $gt: last.created_at } }` with the matching `orderBy` — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record. |
| **joins** | `never` | optional | [REMOVED] `query.joins` was removed in @objectstack/spec 17 (#4286, ADR-0049) — no engine or driver ever read it: a query carrying `joins` behaved exactly as if the key were absent, while its name squatted on the reserved REST parameter set. Delete the key. Related records are read through `expand` — `expand: { owner: { object: 'user', fields: ['name'] } }` — which the engine resolves via batch $in queries, and a single related column is a dotted `fields` path (`fields: ['owner.name']`). |
| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct' \| 'array_agg' \| 'string_agg'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | Aggregation functions |
| **aggregations** | `{ function: Enum<'count' \| 'sum' \| 'avg' \| 'min' \| 'max' \| 'count_distinct'>; field?: string; alias: string; distinct?: boolean; … }[]` | optional | Aggregation functions |
| **groupBy** | `(string \| { field: string; dateGranularity?: Enum<'day' \| 'week' \| 'month' \| 'quarter' \| 'year'>; alias?: string })[]` | optional | GROUP BY targets (strings or `{field, dateGranularity?}` objects for date bucketing) |
| **having** | `any` | optional | HAVING — filter over the AGGREGATED rows (aggregation aliases + groupBy projections); applied engine-side after aggregation |
| **windowFunctions** | `never` | optional | [REMOVED] `query.windowFunctions` was removed in @objectstack/spec 17 (#4286, ADR-0049) — `find()` never applied it: no engine or driver read the key on the query path, so every OVER clause it declared was silently dropped. Delete the key. Window functions are a SQL-driver capability behind `SqlDriver.findWithWindowFunctions(object, query)` (embedder-level; not on the `IDataDriver` contract or the REST surface); request-level analytics are `aggregations` + `groupBy`. |
Expand Down
Loading
Loading