query: correlated count { } and exists { } blocks as return projections

Problem

A client that shows one node with its relationship counts (a node page, an inspector, a table's relationship columns) needs one count per edge type and direction. GQ accepts a correlated block only as a filter: subquery_predicate requires a comparison (query.pest#L183), and return takes expressions (#L189). So the client sends one query per group:

A Person with 92 relationship groups costs 92 requests to count, 1.0–3.4 s each.
The server saturates under that fan-out: 8 such reads at once take 1.6 s of wall time, 32 take 7.4 s.
A table page with N relationship columns needs N count queries per page of rows.

Measured against an OmniGraph 0.13.0 server reached over HTTPS from a laptop (/healthz 0.16–0.21 s on a reused connection), with the release CLI (10 ms start-up), 1–3 runs each. The graph: 101 node types, 552 edge types (149 empty), about 4,900 commits, an Artifact table of about 47k rows. Code references are at main d883c49e.

Proposed change

Allow count { … } and exists { … } (and the other block aggregates) as projections, correlated with the outer row:

match { $n: Person  $n.@id = $id }
return {
  $n,
  count { $n knows $m } as knows,
  count { $m mentions $n } as mentions,
  exists { $n worksAt $c } as employed
}

One row per outer row, one column per block. count is I64 non-null; exists is Bool.
A single-hop, filter-free block is answered from the graph index's degree, as count { } filters already are (bulk_row_count; docs: queries/index.md#L135). Its cost is then flat in the node's degree.
It also covers a page of nodes ($n.@id in $ids): all of a table page's relationship columns in one query.

Affected areas

GQ grammar (projection).
The typed IR: a block aggregate as a projected expression.
Lowering: reuse SubqueryAggregate and its bulk CSR degree path.
Result schema inference and explain.
The docs' "Correlated blocks" section.

Alternatives considered

A neighbourhood route returning per-edge-type counts. query: bounded node-and-neighborhood expansion with edge identities #649 asks for a generic expansion route. A route is a second query surface; a projection keeps the semantics in typed query structures (invariant 9) and serves tables too.
A wildcard traversal grouped by $e.@type. Members must join the same two node types. On 0.13.0 one node across three edge types took 58 s (performance: a wildcard traversal grouped by $e.@type expands in indexed-scan mode and is ~10× slower than its members #1057).
Optional traversals (GQ: optional traversals that preserve unmatched root rows #658) with grouping. These cross-multiply across groups.
One query per group (today).

Related: #649, #658, #764, #955.

Summary

A node page needs one count per edge type and direction, on the same row as the node. GQ can use count { … } and exists { … } only as match filters. A return projection cannot hold those blocks. The client therefore sends one query per edge group. Each query is its own snapshot, and a traversal-plus-count($m) drops every node whose degree is 0.

The gap is present on main 85a4613831e5047a0c988ea0ad37768fa7dff4bb. No open pull request adds this projection. This comment does not close the issue.

Reproduction

Checkout. origin/main is 85a4613831e5047a0c988ea0ad37768fa7dff4bb (2026-10-20, fix(server): delete a branch named like a static branch route (#1091)). The binary is the debug CLI from that tree: omnigraph 0.14.0. The graph is a local file store. The schema is Person, Company, Knows and Mentions (Person -> Person), and WorksAt (Person -> Company). Alice has three Knows edges and two distinct neighbors (a parallel edge to Bob). A later append adds hub with 3200 Knows edges and 31 distinct neighbors.

Code links in the issue body are at d883c49e. On this SHA the grammar moved. subquery_predicate is line 186, return_clause is line 192, and projection is line 194 in crates/omnigraph-compiler/src/query/query.pest. Lowering::bulk_row_count is still at crates/omnigraph/src/engine/lower.rs line 275. The degree sentence in docs/user/queries/index.md is still lines 134–135.

Claim: return rejects count { … } and exists { … }. Reproduced. The error is a parse error, not a type error.

omnigraph query --store graph.omni --format json -e 'query q($id: String) {
  match { $n: Person  $n.@id = $id }
  return { $n.slug, count { $n knows $m } as knows }
}'

parse error: expected not_expr, kw_and, kw_or, kw_is, kw_in, comp_op

Diagnostic Q001, on the { after count. The same error is on exists { $n worksAt $c } as employed, on sum($m.name) { $n knows $m }, and on a count { … } with no alias. In match, a count block with no comparison fails earlier and more specifically: parse error: expected comp_op.

Claim: subquery_predicate requires a comparison, and projection takes only an expression. Reproduced.

subquery_predicate = { agg_func ~ ("(" ~ expr ~ ")")? ~ subquery_block ~ comp_op ~ expr }
exists_block = { "exists" ~ subquery_block }
subquery_block = { "{" ~ clause+ ~ "}" }
return_clause = { "return" ~ kw_distinct? ~ "{" ~ projection+ ~ "}" }
projection = { expr ~ ("as" ~ (reserved_alias | expr_ident))? ~ ","? }

exists { … } is already a match clause and does not need a comparison. It is not an expression, so return still rejects it. Expression operands are calls, properties, variables, and literals. A block is not one of them (query.pest lines 216–218). Expr in crates/omnigraph-compiler/src/query/ast.rs has Aggregate for count($m) and no block form.

Claim: the same blocks work as filters. Reproduced. With $id = "alice":

Filter	Rows
`count { $n knows $m } > 0`	alice
`count { $n knows $m } = 2`	alice
`count { $n knows $m } = 3`	none
`count { $n $e:knows $m } = 3`	alice
`count { $m mentions $n } = 2`	alice
`exists { $n worksAt $c }`	alice
same `exists` for dave	none
`$n.@id in ["alice","bob","dave","p0","p1","p5"]` and `count { $n knows $m } > 0`	alice, bob, p1, p5

An unbound block counts distinct neighbors. Three Knows edges from alice become 2. A bound edge counts edge rows, so the same person is 3. That split is already the filter contract (docs/user/queries/index.md and the unbound-versus-bound note on issue #764). hub confirms it at a larger size: count { $n knows $m } = 31 keeps the row, and count($e.@id) over the bound edge returns 3200.

Claim: a single-hop, filter-free count { } uses the graph-index degree. Reproduced for the documented shape. explain does not run the query. The datafusion line is the marker lowering selects before execution.

Block	`AntiJoinMaskExec` detail
`count { $n knows $m } > 0` and `= 2`	`$n count …, bulk=Knows Out`
`count { $m mentions $n } > 0`	`bulk=Mentions In`
`exists { $n worksAt $c }`	`bulk=WorksAt Out`
`$n.@id in $ids` and `count { $n knows $m } > 0`	`bulk=Knows Out`, strategy `correlated`
`count { $n <knows> $m } > 0`	`bulk=Knows Both`
`count { $n <knows> $m } > 1`	no `bulk=`
`count { $n $e:knows $m } = 3`	no `bulk=`
`count { $n knows $m $m.name = "Bob" } > 0`	no `bulk=`
two hops `count { $n knows $m $m knows $p } > 0`	no `bulk=`
`count { $n (knows \| mentions) $m } > 0`	no `bulk=`, expand mode `indexed_scan`
`count { $m: Person $n * $m } > 0`	no `bulk=`, wildcard `[Knows out, Mentions out]`, mode `indexed_scan`

A wildcard with no node bindings fails typecheck: T5: wildcard traversal requires explicitly declared source and destination node bindings.

The user-facing physical AntiJoin JSON shows strategy, predicate, and typed_left type I64. It does not show bulk. bulk= is only on the datafusion operator text (AntiJoinMaskExec in crates/omnigraph/src/engine/operators/anti_join.rs). The docs name a directed edge (docs/user/queries/index.md lines 134–135). The code also selects the degree path for an undirected existence test (count > 0, bulk=Knows Both). An undirected count that is not an existence test does not. That part of the boundary is wider in code than in the docs. Call the documented directed case reproduced, and the undirected existence case extra.

Lowering::bulk_row_count (lower.rs lines 275–324) returns one named edge and a direction when the predicate is a row count, the expand is one hop, there is no filter, there is no edge binding, the input is an OuterReference, and the direction is not Both unless the predicate is an existence test. bulk_anti_join_mask (crates/omnigraph/src/engine/graph.rs line 1490) reads the CSR or CSC degree and returns a Boolean keep mask. For a real count it sorts neighbor ids and removes duplicates. SubqueryAggregate (crates/omnigraph/src/engine/operators/subquery_aggregate.rs) folds tagged inner rows only when that mask does not apply. The operator filters. It does not append a column. docs/dev/execution.md lines 79–82 and 243 describe the same mask.

Claim: the workaround is one query per group. Reproduced. For alice, three queries return knows = 2, mentions = 2, and employed = 1 from count($m) or count($c). count($e.@id) on Knows returns 3. A page query

match { $n: Person  $n.@id in $ids  $n knows $m }
return { $n.slug as slug, count($m) as knows }

with alice and dave returns only alice. Dave has degree 0, so the required traversal removes that outer row. return { count($m) > 0 as has_knows } works. That is the grouped comparison from #955. It is one Bool for the matched group. It still needs the traversal, so a zero-degree person still produces no row. explain types that grouped count($m) as I64?. The block row-count on an AntiJoin is I64 with no ? (docs/user/queries/explain.md line 265: row count is I64 non-null; column aggregates stay nullable).

Claim: 92 groups at 1.0–3.4 s, and 8-way / 32-way saturation. Not reproduced. This run did not use the 0.13.0 HTTPS server, the 552-edge graph, or a laptop round trip. Local debug CLI, file store, five runs after one warmup:

Query	Median
return `$n.slug` for hub	20.3 ms
`count { $n knows $m } = 31` (degree path, 3200 edges, 31 neighbors)	24.6 ms
`count { $n knows $m } > 0`	28.8 ms
`return { count($m) }` over the same hop	29.1 ms
`return { count($e.@id) }` over 3200 edge rows	31.7 ms
six degree-filter processes, sequential	133.4 ms
six traversal-count processes, sequential	140.4 ms

Process start is about 20 ms, so it dominates this graph. Six processes cost about six times one process. The degree filter and the edge-row count differ by about 7 ms here. These numbers do not confirm or deny the remote 1.0–3.4 s figures.

Root cause

A correlated block is a match clause. return typechecks expressions and infers the result schema from those expressions (infer_query_result_schema in crates/omnigraph-compiler/src/query/typecheck.rs). The planner lowers the block to an AntiJoin whose predicate keeps or drops the outer row. There is no IR node that places the aggregate value on the output row. The degree fast path computes that value and then throws it away after the comparison.

Design

The contract to add is a scalar block value on each outer row.

One output row per outer row, including a row whose block matches nothing.
One column per block. The block’s names stay inside the block. The block must read an outer variable, as a filter block must.
The number must be the number the same block already compares in match. Unbound count { $n knows $m } is distinct neighbors (alice is 2). Bound count { $n $e:knows $m } is edge rows (alice is 3). A projection that used the other number would make the filter and the column disagree.
Row count is I64, non-null, and 0 when there is no match. A count past the I64 range fails, as the filter already fails.
exists { … } is Bool, non-null, and true when that count is greater than 0.
sum, min, max, avg, and a column count keep the nullable block signatures they already have. With no matches, sum, min, max, and avg are null. They do not become 0.
Every block in the query reads the one snapshot the query already captured.

Make the block an atomic expression: count { … }, sum(expr) { … }, min, max, avg, and exists { … }. projection then accepts it with the existing as alias. subquery_predicate stays the match-clause rule and stays ahead of a filter, so count { … } > 0 in match stays a predicate. A missing comparison in match stays expected comp_op. Do not add not { … } as a column in this change. exists is the boolean form.

Two unaliased count columns already collide. Keep that error.

Fast path. Share the CSR degree read in bulk_anti_join_mask. Have it return the degree. The filter compares the degree and builds a mask. The projection writes an I64 column. One walk, two consumers. Select it only where bulk_row_count already returns an edge: one hop, no filter, no edge binding, one named edge, correlated strategy, direction out or in. For exists, direction Both may use the existence check the filter already uses (bulk=Knows Both on count > 0). A projected undirected count needs an explicit union of the two adjacency lists, and that union must equal count { $n <knows> $m } = k. Until then, leave the undirected count on the inner tree. The filter for > 1 already does that. Wildcard, alternation, a bound edge, a filter, and a second hop stay on the inner tree. They must stay correct. The missing fast path may change cost only. A page $n.@id in $ids is already the correlated bulk shape. The projecting operator walks each outer id, as the mask already does, and it keeps the zeros.

Grouping and order. A block value is a scalar on the outer row. count($m) in return groups rows and drops outer rows with no match. One return must not contain both until a rule names the cardinality. Refuse the mix. order may name the alias. distinct includes the column. A later cut can allow count { $n knows $m } > 0 as flag as a Bool expression. That is the per-row form of the grouped comparison #955 added. It is not required to close the example in this issue.

Explain and schema. The plan node must show the outer variable, the aggregate type, and bulk when lowering selects the degree path. Today an operator must read the datafusion text to see bulk=. Result-schema inference reads the block type: I64 non-null, Bool non-null, or the nullable column aggregate. Profile can keep the current signal: the inner OuterReference is skipped when the degree path runs (anti_join_inner_tree_is_skipped_under_the_bulk_check_and_executed_without_it).

Why this shape. An expression node is the typed structure the rest of the pipeline already uses. A projection-only grammar would need a second change for order and for a Bool comparison. A neighborhood route (#649) returns edge identities for a client that does not know the schema, with a page and a truncation signal. It does not return one integer per known column of a table. Optional match (#658) keeps a root and then multiplies rows by neighbors. A wildcard grouped by $e.@type (#1057) is one indexed_scan, and the members must share the two endpoint types.

What this cut leaves slow on purpose. Wildcard, bound-edge, filtered, and undirected counts stay on the inner tree. Those shapes do not have one CSR degree today. A second degree definition would make the filter and the column disagree. The mix with a grouped return aggregate is refused for the same reason: two cardinalities in one clause.

Liability. This removes N requests, N commits, and a client join. A zero-degree row becomes visible. The column and the filter are one number, so they cannot drift. This adds a public column type and a cardinality rule. Clients will store both. Explain must show a missing bulk, or a wide page can look cheap and still scan. The distinct-neighbor rule must stay visible, or a client that wants parallel-edge cardinality will read 2 where the edge table has 3.

docs/rfcs/README.md asks for an RFC when the change is a new query contract. This is that kind of change, because the cardinality and the I64 column are hard to take back. The RFC can be short. It records the row rule, the types, the identity with the filter count, the fast-path shapes, and the mixed-aggregate refusal. It does not need to wait for #658 or #649.

Tests. Extend the .gqt corpus. One case covers several edge types, both directions, degrees 0, 1, and N, one parallel edge, and $n.@id in $ids. Expect the rows, the column types, and the zero-degree row. Expect bulk= on the fast shapes and no bulk= on a bound edge, a filter, a wildcard, and two hops. Expect the projected number to equal count { … } = k on that seed (2 unbound, 3 bound). Extend the report test that already checks a skipped inner tree. Add a compiler test for the projection parse, the non-null I64 and Bool, and the mixed-aggregate refusal.

Lance

Lance does not cause this gap, and a Lance release will not add the syntax. OmniGraph pins unmodified Lance 13.0.0 (published 2026-10-07). No open pull request in this repository upgrades Lance. Upstream v14.0.0-beta.21 (2026-10-19) notes distributed index optimization and an FTS finalize fix: https://github.com/lance-format/lance/releases/tag/v14.0.0-beta.21. The degree structure is OmniGraph’s GraphIndex CSR/CSC, built from the edge tables. Lance stores those tables. The missing piece is the GQ expression and a projecting operator.

Related work

GQ: no way to filter by relationship COUNT — aggregates are rejected in a match block #764, closed by feat(query): extend AntiJoin for nested query predicates #771. Count filters exist. This issue is the projection of that count.
GQ: allow comparisons over aggregates in return #955, closed by feat(gq): project comparisons over aggregate results #974. count($p) > 0 is legal in a grouped return. It does not keep a zero-degree row, and it is not one column per edge type.
query: bounded node-and-neighborhood expansion with edge identities #649, open. Bounded neighborhood expansion. A different result. It does not replace this issue.
GQ: optional traversals that preserve unmatched root rows #658, open, needs-rfc. Optional traversals. A different cardinality.
performance: a projected count over one hop scans the far node table #1051, open. return { count($m) } over one hop scans the far node. A faster per-group query is still N requests, and it still drops zeros.
server: several named queries in one request, answered at one snapshot #1055, open. Several named queries in one request, one snapshot. That cuts round trips. It does not define these columns.
performance: a wildcard traversal grouped by $e.@type expands in indexed-scan mode and is ~10× slower than its members #1057, open. A wildcard grouped by $e.@type uses indexed_scan. The degree path does not apply to a wildcard. This projection does not depend on making that scan fast.
RFC 0056, PR rfc: add RFC 0056, one language for every graph operation #670, open draft. One language for every graph operation. It does not specify this projection. It does argue for a GQ form instead of a second route.

No merged pull request and no open pull request adds a block aggregate as a projection. This issue is not a duplicate. It stays open.

Limitations

The graph is a local file store: tens of nodes, three edge types, and one hub of 3200 parallel edges. It is not the reported 0.13.0 HTTPS server, 101 node types, 552 edge types, or 47k-row table. The concurrency figures were not rerun. explain does not execute, so this pass did not capture profile skipped rows. The existing report test covers that signal for filters. No code was changed.

Navigation Menu

Repository navigation

query: correlated `count { }` and `exists { }` blocks as `return` projections #1050

Problem

Proposed change

Affected areas

Alternatives considered

Summary

Reproduction

Root cause

Design

Lance

Related work

Limitations

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

query: correlated count { } and exists { } blocks as return projections #1050

Description

Problem

Proposed change

Affected areas

Alternatives considered

Activity

ragnorc commented on Oct 20, 2026

Summary

Reproduction

Root cause

Design

Lance

Related work

Limitations

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

query: correlated `count { }` and `exists { }` blocks as `return` projections #1050