Skip to content

tp: allow pipelines as subqueries - #7644

Closed
LalitMaganti wants to merge 1 commit into
dev/lalitm/analysis-cte-scopesfrom
dev/lalitm/pipeline-subqueries-v2
Closed

LalitMaganti wants to merge 1 commit into
dev/lalitm/analysis-cte-scopesfrom
dev/lalitm/pipeline-subqueries-v2

Conversation

@LalitMaganti

@LalitMaganti LalitMaganti commented Sep 27, 2026 •

Copy link
Copy Markdown
Member

Pipelines can only be whole statements or the body of a CREATE PERFETTO TABLE, so nothing written in SQL can use one: not a join, a CTE, a view or a function. That is what stops users like wattson moving off the interval intersection macros.

A pipeline in parentheses now reads like any other subquery, in a FROM clause or as a CTE:

SELECT t.id, p.total
FROM tree t
JOIN (FROM tree |> TREE ACCUMULATE UP SUM(self) AS total) p USING (id)

How:

  • Once a statement is parsed, each such pipeline is compiled and replaced with SQL running its plan, recorded as a rewrite where it was written, like a macro call. So SQL and pipelines nest either way round, pipelines work in macros and macros in pipelines, and views, functions and triggers need nothing new.

  • The SQL a pipeline reads is evaluated there too, so it can read a function's arguments.

  • A pipeline's rewrite stands in for the macro calls written inside it, and errors trace back through macro calls.

  • SQL reading a pipeline of the same statement is described by the plan that pipeline compiled to.

  • A pipeline read by a pipeline is compiled into the same plan, in a scope of its own, so it can also be an interval intersection's operand.

  • A pipeline can read the CTEs of the statement it is written in (via tp: let semantic analysis see CTEs #7643).

@LalitMaganti
LalitMaganti added this pull request to stack #7645 September 27, 2026 20:08
@LalitMaganti
LalitMaganti removed this pull request from stack #7645 September 27, 2026 20:08
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries-v2 branch from 167c993 to 22e90ad Compare September 27, 2026 20:43
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/analysis-cte-scopes branch 2 times, most recently from 8ca182d to 9c25c8f Compare September 27, 2026 21:13
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries-v2 branch from 22e90ad to 8f84807 Compare September 27, 2026 21:13
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown

@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries-v2 branch from 8f84807 to a4caa76 Compare September 27, 2026 22:55
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/analysis-cte-scopes branch from 9c25c8f to 3124216 Compare September 27, 2026 22:55
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries-v2 branch from a4caa76 to 7fc87f4 Compare September 27, 2026 23:22
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/analysis-cte-scopes branch 2 times, most recently from 1673785 to 7ff5acd Compare September 27, 2026 23:45
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries-v2 branch 2 times, most recently from 37b398c to ad81123 Compare September 28, 2026 03:53
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/analysis-cte-scopes branch from 7ff5acd to 9e2c303 Compare September 28, 2026 03:53
Pipelines can only be whole statements or the body of a CREATE PERFETTO
TABLE, so nothing written in SQL can use one: not a join, a CTE, a view
or a function. That is what stops users like wattson moving off the
interval intersection macros.

A pipeline in parentheses now reads like any other subquery, in a FROM
clause or as a CTE:

  SELECT t.id, p.total
  FROM tree t
  JOIN (FROM tree |> TREE ACCUMULATE UP SUM(self) AS total) p USING (id)

Once a statement is parsed, the parser compiles each such pipeline and
replaces it with SQL running its plan, recorded as a rewrite where it
was written, as a macro call is. So SQL and pipelines nest either way
round, pipelines work in macros and macros in pipelines, and views,
functions and triggers need nothing more than they already do. As the
SQL a pipeline reads is evaluated there too, it can read a function's
arguments.

The rewrite tree now covers pipelines as well as macro calls. A
pipeline's rewrite stands in for the macro calls written inside it, and
a node written inside a macro's expansion is taken from there, so errors
trace back through the call. SQL reading a pipeline of the same
statement is described by the plan that pipeline compiled to.

A pipeline read by a pipeline is not expanded: the compiler reads it as
one more kind of relation, next to dataframes and SQL, and compiles it
into the same plan in a scope of its own. That also lets it be an
interval intersection's operand.

A pipeline can read the CTEs of the statement it is written in, as the
SQL it reads is described where it is written.
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/analysis-cte-scopes branch from 9e2c303 to 502bb9c Compare September 28, 2026 04:21
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries-v2 branch from ad81123 to 6933c43 Compare September 28, 2026 04:21
@LalitMaganti

Copy link
Copy Markdown
Member Author

Shelved for now: pipelines inside other SQL (subqueries, CTEs, joins) need proper predicate pushdown and cost estimation first. Pipelines reading SQL and CTEs as input continue in #7641 and #7643. Branch kept for when we pick this up again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant