Skip to content

[BUG] Row counts are not seeded for tables referenced only inside a subquery, so those scans keep Calcite's default estimate #22877

Description

@LantaoJin

Describe the bug

IndexRowCountFetcher seeds Calcite's row counts with real primary-shard doc counts for every index a
query touches. It collects those indices by walking the RelNode tree through RelNode.getInputs().

It runs BEFORE subquery removal / decorrelation, so at that point a subquery is still a RexSubQuery
hanging off a Filter condition or a Project expression — it is not an input in the RelNode tree.
Walking inputs alone therefore misses every table that appears ONLY inside a subquery, and each of those
scans falls back to Calcite's default row count (100).

The consequence is not a wrong answer — it is a plan chosen from a number with no relation to the data.
Estimates derived from a 100-row scan collapse: a filter and aggregate above it come out at about one row.
Anything downstream that consumes cardinality is then working from fiction, including join ordering and
cost-based operator choices.

The failure is silent. The query still returns correct results, so nothing surfaces the problem
except the plan quality itself.

Concrete instance

TPC-H Q4 on a 4-node cluster at sf=10. Its lineitem reference appears only inside the exists
subquery, while orders is scanned by the outer query:

OpenSearchBroadcastExchange est=1.0   cols=[l_orderkey]
  OpenSearchAggregate      est=1.0   groupSet={0}
    OpenSearchExchangeReducer est=1.0
      OpenSearchAggregate  est=1.0   groupSet={0}
        OpenSearchProject  est=6.25
          OpenSearchFilter est=6.25
            OpenSearchTableScan est=100.0        <-- lineitem, actually ~60,000,000 rows

orders was seeded correctly in the same query (a sibling estimate resolves to 15,000,000 × 0.0625 = 937,500), which is what isolates this to the subquery path rather than to the fetch itself.

The chain is 100 → filter 6.25 → aggregate 1.0, so a build side that actually materialises
111,746,977 bytes is costed as a single 8-byte row. It is then admitted, computed in full, rejected by
the runtime size cap, and the query is re-planned — roughly doubling wall time.

Related component

Plugins

To Reproduce

Deterministic, no cluster required — the defect is visible directly in the collection step:

  1. Build a ClusterState with two indices, e.g. outer_index and inner_index.
  2. Parse a query in which the second index is reachable ONLY through a subquery:
    SELECT * FROM outer_index WHERE EXISTS (SELECT 1 FROM inner_index WHERE inner_index.status = outer_index.status)
  3. Call IndexRowCountFetcher.referencedIndexNames(parsedPlan).
  4. Observe the returned set is [outer_index]inner_index is missing, so it is never seeded and keeps
    Calcite's default row count.

The same holds for IN and scalar subqueries, which lower through different rules.

On a cluster, the observable symptom is a plan chosen on the wrong cardinality. For the Q4 shape above:
a broadcast build is admitted, overflows the runtime cap (BroadcastCaptureSink logs
Broadcast build-side payload exceeded the configured limit), and the query re-plans and completes in
about twice the time it needs.

Expected behavior

Every table a query reads should have its row count seeded, regardless of whether it is reached through
the input tree or through a RexSubQuery. A scan should not silently fall back to a default estimate
while its siblings are sized from real statistics — that makes the two sides of a join incomparable.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

  • OpenSearch 3.9.0-SNAPSHOT
  • Component: plugins/analytics-engine (sandbox)

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions