Implement json_agg

[fulghum]

Adds PostgreSQL-compatible json_agg(anyelement) support for scalar, array, composite, JSON, and JSONB values, including window aggregation and DISTINCT with SQL NULLs.

Part of #3099

Depends on: dolthub/go-mysql-server#3728

[github-actions]

	Main	PR
Total	42090	42090
Successful	19177	19176
Failures	22913	22914
Partial Successes1	5471	5471

	Main	PR
Successful	45.5619%	45.5595%
Failures	54.4381%	54.4405%

${\color{red}Regressions (1)}$

copyselect

QUERY:          drop table test3;
RECEIVED ERROR: COPY DATA message received without a COPY FROM STDIN operation in progress

Footnotes

1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct. ↩

[coffeegoddd]

@fulghum DOLT

read_tests	from_latency	to_latency	percent_change
covering_index_scan_postgres	2.43	2.43	0.0
groupby_scan_postgres	78.6	78.6	0.0
index_join_postgres	2.26	2.26	0.0
index_join_scan_postgres	1.61	1.64	1.86
index_scan_postgres	484.44	484.44	0.0
oltp_point_select	0.36	0.37	2.78
oltp_read_only	6.32	6.43	1.74
select_random_points	0.7	0.7	0.0
select_random_ranges	1.03	1.03	0.0
table_scan_postgres	493.24	484.44	-1.78
types_table_scan_postgres	1213.57	1213.57	0.0

write_tests	from_latency	to_latency	percent_change
oltp_delete_insert_postgres	6.67	6.67	0.0
oltp_insert	3.36	3.36	0.0
oltp_read_write	13.46	13.46	0.0
oltp_update_index	3.55	3.55	0.0
oltp_update_non_index	3.25	3.25	0.0
oltp_write_only	7.04	7.04	0.0
types_delete_insert_postgres	7.17	7.17	0.0

[itoqa]

Commit: 14645bc: 15 test cases ran, 14 passed ✅, 1 additional finding ⚠️.

Summary

Coverage spans normal and edge-case database behavior for JSON aggregation, including ordering, filtering, duplicate removal, null handling, nested and typed values, numeric boundaries, grouped results, and windowed calculations across partitions and empty frames. Existing aggregate behavior also remained intact, while one broader compatibility limitation affects filtered aggregate syntax.

Safe to merge — the only observed failure is a medium-severity, pre-existing SQL compatibility limitation unrelated to this PR, with no regressions or new PR-attributable failures. It is a flag for later rather than a merge blocker.

Tests run by Ito

Result	Severity	Type	Description
✅	—	Aggregate	The database returned [3, null, 7] in the same order as the input. An empty input returned SQL NULL instead of an empty array.
✅	—	Aggregates	Grouped sums, averages, variances, and boolean checks returned the expected values. Running totals and averages stayed separate for each group, and a system-table count query also worked.
✅	—	General	The database accepted plain and windowed JSON aggregation. Rolling values stayed in the right group and frame, including multiple expressions and an empty frame.
✅	—	General	Grouped and window queries kept nested arrays, null values, composite objects, and embedded JSON in the expected format. Each window partition accumulated only its own rows.
✅	—	General	Rows with the same sort key returned the exact values for each requested window frame. The empty first frame returned SQL NULL, and later frames did not include values from a neighboring frame.
✅	—	Distinct	The query returned [2, null, 5]. Duplicate numbers were removed, and two SQL NULL values became one JSON null.
✅	—	Domains	Grouped and window queries returned the expected JSON for domain, array, composite, and scalar values. Nested arrays, SQL nulls, composite fields, and window frames were preserved correctly.
✅	—	Frames	The first row returned SQL NULL because its preceding frame had no rows. Later rows returned [10] and [20], containing only their one preceding value.
✅	—	Groups	Grouped results kept DISTINCT values separate: group a returned [9, null], while group b returned [9].
✅	—	Numeric	High-precision finite values stayed JSON numbers, while NaN and infinity values became valid JSON strings.
✅	—	Rebuild	The named window query accepted the rewritten aggregate expression and returned 1, 3, and 6 for the three growing frames.
✅	—	Rev	The windowed JSON arrays stayed isolated for each group: group a returned [10] and [10, 20], while group b returned [100] and [100, 200].
✅	—	Serialize	The local database was unavailable, so the SQL checks could not run. Source review shows that the new aggregate sends scalar, date, array, record, JSON, and JSONB values through the shared type-aware serializer.
✅	—	Window	Each output row received the values from its ordered frame: [10], [10, 20], and [10, 20, 30].
⚠️		Rev	The database rejects the query before it builds the JSON array. The filtered row 20 should be removed, while 30, the retained NULL, and 10 should appear in descending sort order as [30, null, 10].

Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Filtered JSON aggregation is rejected

• Severity: Medium
• Description: The database rejects the query before it builds the JSON array. The filtered row 20 should be removed, while 30, the retained NULL, and 10 should appear in descending sort order as [30, null, 10].
• Impact: Queries that use aggregate FILTER with json_agg fail instead of returning the requested JSON array. Users can still run other queries, but this valid PostgreSQL-compatible form cannot be used.
• Steps to Reproduce:
  1. Connect to a local Doltgres server with a PostgreSQL client.
  2. Run SELECT json_agg(value ORDER BY sort_key DESC) FILTER (WHERE keep_it) FROM (VALUES (10, 3, true), (NULL::int, 2, true), (20, 1, false), (30, 4, true)) AS v(value, sort_key, keep_it);
  3. Check the result. The expected JSON array is [30, null, 10], but the server returns ERROR: function filters are not yet supported.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: The parser and semantic type checker explicitly support FILTER on aggregate calls. In postgres/parser/sem/tree/type_check.go:965-978, a non-nil expr.Filter is accepted when the resolved function is an AggregateClass function, checked as a boolean expression, and retained on the function expression. The production AST conversion then prevents that valid expression from executing: server/ast/func_expr.go:29-35 returns errors.Errorf("function filters are not yet supported") whenever node.Filter is non-nil, before function-name dispatch, ORDER BY conversion, or construction of the execution expression. The PR's new server/functions/aggregate/json_aggregates.go:33-48 registers json_agg as a non-strict aggregate, and jsonAggBuffer.Update at lines 75-96 can collect values and preserve SQL NULL as JSON null, but that code is unreachable for a filtered call because conversion fails first. Aggregate-local ORDER BY is also rejected by the generic fallback at server/ast/func_expr.go:138-140 for functions other than the existing special cases, so this test's combined ORDER BY and FILTER query has no supported conversion path. The smallest practical remediation is to carry the parsed aggregate Filter into the aggregate execution expression, then apply it before json_agg's buffer update; this should be implemented at the existing AST/framework boundary rather than by changing JSON serialization.

Evidence Package

• text/REV-1/REV-1-filtered-json-agg.txt — 2 KB

Tip

Reply with @itoqa to send us feedback on this test run.

[Hydrocharged]

LGTM!

[itoqa]

History reset (rebase or force-push detected). Starting test narrative over.

Commit: f0bc8d7: 19 test cases ran, 1 failed ❌, 17 passed ✅, 1 additional finding ⚠️.

Summary

Coverage spans JSON aggregation happy paths, ordering, null handling, duplicate removal, windowed results, nested and specialized data types, numeric and temporal conversion, error recovery, and compatibility with existing aggregate behavior. Overall, the exercised database behavior is broadly healthy, with a correctness gap in a windowed duplicate-handling edge case and a separate limitation around filtered or sorted aggregate syntax.

Merge with caution — the PR introduces a medium-severity correctness defect where windowed JSON aggregation can retain duplicates, producing incorrect query results. The filtered and sorted aggregate limitation is unrelated to this PR and is a flag for later rather than a merge blocker.

Tests run by Ito

Result	Severity	Type	Description
❌		Distinct	The third row in both partitions returns [1, 2, 2]. It should return [1, 2], because the repeated value 2 should appear only once in a distinct result.
✅	—	Aggregate	The database returned [2, null, 1] in the same order as the input values, including the SQL NULL as JSON null.
✅	—	Aggregate	A query with no input rows returned SQL NULL, so callers can tell the difference between no result and an array containing null.
✅	—	General	A bad JSON value shows a clear error, and the next aggregate query still returns the complete list of values.
✅	—	General	A malformed JSON value shows the expected conversion error, and the next window query returns complete results: [1, 2], [2, 3], and [3].
✅	—	General	Grouped numeric and boolean aggregates, running sums, explicit casts, NULL handling, and recovery after an error all returned the expected results.
✅	—	General	The database kept values in the right order when rows shared the same order key. Both query forms returned [a], [a, b], [b, c], and [c, d].
✅	—	General	The moving window returned SQL NULL for the first row of the second partition instead of reusing the previous partition's ["a"] value. The next row returned the correct independent frame ["c"].
✅	—	Conversion	The database service was unavailable during the mixed-value check, so the expected result could not be observed. The available source and test coverage support the requested conversion behavior, and the run found no application defect.
✅	—	Conversion	The database returned nested arrays with null members preserved, and composite rows kept their field names and values in structured JSON objects.
✅	—	Conversion	Large decimal values keep their exact digits in the JSON array, and NaN and infinity values are returned as quoted strings.
✅	—	Conversion	Date and time values appear as quoted JSON strings, including the UTC offset, while positive integer domain values appear as JSON numbers.
✅	—	Distinct	The aggregate returned one 2, one 1, and one null value, so repeated values and repeated nulls were correctly reduced to a single entry.
✅	—	Distinct	Each group kept its own values, including the value shared by both groups, while duplicates within each group were removed.
✅	—	Regression	The grouped query returned 3 for the first group and 30 for the second group.
✅	—	Regression	The running total query returned 1, 3, and 6 for input values 1, 2, and 3.
✅	—	Window	The database returned [1], [1, 2], and [2, 3] for the three ordered window frames.
✅	—	Window	The database returned SQL NULL for the empty first frame, [null] for a frame containing a NULL value, and [1] for the next frame.
⚠️		Rev	The aggregate query returns an error, so it does not produce the filtered and sorted JSON list that users requested.

Additional Findings Details

These findings are unrelated to the current changes but were observed during testing.

🟡 Aggregate queries cannot sort and filter JSON values

• Severity: Medium
• Description: The aggregate query returns an error, so it does not produce the filtered and sorted JSON list that users requested.
• Impact: Users cannot create a filtered or sorted JSON list with the advertised aggregate query syntax. They can work around this by filtering and sorting in a subquery, but the standard form still fails.
• Steps to Reproduce:
  1. Connect to the local Doltgres server with a PostgreSQL client.
  2. Create a temporary table with text values, sort keys, and a true/false include flag, then insert rows in a different order from their sort keys.
  3. Run SELECT json_agg(v ORDER BY sort_key DESC) FILTER (WHERE include_flag) FROM the temporary table.
  4. Observe the error instead of an aggregate containing only included rows in descending sort-key order.
• Stub / mock content: No stubs, mocks, or bypasses were applied for this test in the recorded run.
• Code Analysis: The runtime evidence is code-backed. In server/ast/func_expr.go:29-35, nodeFuncExpr immediately returns function filters are not yet supported whenever tree.FuncExpr.Filter is non-nil. The same function then checks generic function-level ordering at server/ast/func_expr.go:138-140 and returns function ORDER BY is not yet supported for any function not handled by the special cases above. json_agg is not one of those special cases: the PR-added server/functions/aggregate/json_aggregates.go:28-47 registers json_agg and supplies its aggregate and window implementations, while jsonAggBuffer.Update at lines 75-96 only processes the rows it receives and has no opportunity to apply SQL FILTER or aggregate ORDER BY after the parser rejects the expression. The PR diff adds the json_agg registration/implementation and related value-type support, but does not modify server/ast/func_expr.go or provide an alternate path for these clauses. The smallest practical fix is to preserve aggregate FILTER and ORDER BY in the parsed expression and pass them through the aggregate execution path, with focused support for json_agg (or the shared aggregate framework) so filtering happens before buffer updates and ordering is applied before aggregation.

Evidence Package

• text/REV-2/REV-2-json-agg-order-filter.txt — 2 KB

Tip

Reply with @itoqa to send us feedback on this test run.