Describe the bug
IndexRowCountFetcher seeds Calcite's row counts with real primary-shard doc counts for every index a
query touches. It collects those indices by walking the RelNode tree through RelNode.getInputs().
It runs BEFORE subquery removal / decorrelation, so at that point a subquery is still a RexSubQuery
hanging off a Filter condition or a Project expression — it is not an input in the RelNode tree.
Walking inputs alone therefore misses every table that appears ONLY inside a subquery, and each of those
scans falls back to Calcite's default row count (100).
The consequence is not a wrong answer — it is a plan chosen from a number with no relation to the data.
Estimates derived from a 100-row scan collapse: a filter and aggregate above it come out at about one row.
Anything downstream that consumes cardinality is then working from fiction, including join ordering and
cost-based operator choices.
The failure is silent. The query still returns correct results, so nothing surfaces the problem
except the plan quality itself.
Concrete instance
TPC-H Q4 on a 4-node cluster at sf=10. Its lineitem reference appears only inside the exists
subquery, while orders is scanned by the outer query:
OpenSearchBroadcastExchange est=1.0 cols=[l_orderkey]
OpenSearchAggregate est=1.0 groupSet={0}
OpenSearchExchangeReducer est=1.0
OpenSearchAggregate est=1.0 groupSet={0}
OpenSearchProject est=6.25
OpenSearchFilter est=6.25
OpenSearchTableScan est=100.0 <-- lineitem, actually ~60,000,000 rows
orders was seeded correctly in the same query (a sibling estimate resolves to 15,000,000 × 0.0625 = 937,500), which is what isolates this to the subquery path rather than to the fetch itself.
The chain is 100 → filter 6.25 → aggregate 1.0, so a build side that actually materialises
111,746,977 bytes is costed as a single 8-byte row. It is then admitted, computed in full, rejected by
the runtime size cap, and the query is re-planned — roughly doubling wall time.
Related component
Plugins
To Reproduce
Deterministic, no cluster required — the defect is visible directly in the collection step:
- Build a
ClusterState with two indices, e.g. outer_index and inner_index.
- Parse a query in which the second index is reachable ONLY through a subquery:
SELECT * FROM outer_index WHERE EXISTS (SELECT 1 FROM inner_index WHERE inner_index.status = outer_index.status)
- Call
IndexRowCountFetcher.referencedIndexNames(parsedPlan).
- Observe the returned set is
[outer_index] — inner_index is missing, so it is never seeded and keeps
Calcite's default row count.
The same holds for IN and scalar subqueries, which lower through different rules.
On a cluster, the observable symptom is a plan chosen on the wrong cardinality. For the Q4 shape above:
a broadcast build is admitted, overflows the runtime cap (BroadcastCaptureSink logs
Broadcast build-side payload exceeded the configured limit), and the query re-plans and completes in
about twice the time it needs.
Expected behavior
Every table a query reads should have its row count seeded, regardless of whether it is reached through
the input tree or through a RexSubQuery. A scan should not silently fall back to a default estimate
while its siblings are sized from real statistics — that makes the two sides of a join incomparable.
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
- OpenSearch
3.9.0-SNAPSHOT
- Component:
plugins/analytics-engine (sandbox)
Additional context
Add any other context about the problem here.
Describe the bug
IndexRowCountFetcherseeds Calcite's row counts with real primary-shard doc counts for every index aquery touches. It collects those indices by walking the RelNode tree through
RelNode.getInputs().It runs BEFORE subquery removal / decorrelation, so at that point a subquery is still a
RexSubQueryhanging off a
Filtercondition or aProjectexpression — it is not an input in the RelNode tree.Walking inputs alone therefore misses every table that appears ONLY inside a subquery, and each of those
scans falls back to Calcite's default row count (100).
The consequence is not a wrong answer — it is a plan chosen from a number with no relation to the data.
Estimates derived from a 100-row scan collapse: a filter and aggregate above it come out at about one row.
Anything downstream that consumes cardinality is then working from fiction, including join ordering and
cost-based operator choices.
The failure is silent. The query still returns correct results, so nothing surfaces the problem
except the plan quality itself.
Concrete instance
TPC-H Q4 on a 4-node cluster at sf=10. Its
lineitemreference appears only inside theexistssubquery, while
ordersis scanned by the outer query:orderswas seeded correctly in the same query (a sibling estimate resolves to15,000,000 × 0.0625 = 937,500), which is what isolates this to the subquery path rather than to the fetch itself.The chain is
100 → filter 6.25 → aggregate 1.0, so a build side that actually materialises111,746,977 bytes is costed as a single 8-byte row. It is then admitted, computed in full, rejected by
the runtime size cap, and the query is re-planned — roughly doubling wall time.
Related component
Plugins
To Reproduce
Deterministic, no cluster required — the defect is visible directly in the collection step:
ClusterStatewith two indices, e.g.outer_indexandinner_index.SELECT * FROM outer_index WHERE EXISTS (SELECT 1 FROM inner_index WHERE inner_index.status = outer_index.status)IndexRowCountFetcher.referencedIndexNames(parsedPlan).[outer_index]—inner_indexis missing, so it is never seeded and keepsCalcite's default row count.
The same holds for
INand scalar subqueries, which lower through different rules.On a cluster, the observable symptom is a plan chosen on the wrong cardinality. For the Q4 shape above:
a broadcast build is admitted, overflows the runtime cap (
BroadcastCaptureSinklogsBroadcast build-side payload exceeded the configured limit), and the query re-plans and completes inabout twice the time it needs.
Expected behavior
Every table a query reads should have its row count seeded, regardless of whether it is reached through
the input tree or through a
RexSubQuery. A scan should not silently fall back to a default estimatewhile its siblings are sized from real statistics — that makes the two sides of a join incomparable.
Additional Details
Plugins
Please list all plugins currently enabled.
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
3.9.0-SNAPSHOTplugins/analytics-engine(sandbox)Additional context
Add any other context about the problem here.