tp: allow pipelines as subqueries - #7644
Closed
LalitMaganti wants to merge 1 commit into
Closed
LalitMaganti wants to merge 1 commit into
LalitMaganti wants to merge 1 commit into
Conversation
This was referenced Sep 27, 2026
LalitMaganti
added this pull request to stack #7645
September 27, 2026 20:08
LalitMaganti
removed this pull request from stack #7645
September 27, 2026 20:08
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries-v2
branch
from
September 27, 2026 20:43
167c993 to
22e90ad
Compare
LalitMaganti
force-pushed
the
dev/lalitm/analysis-cte-scopes
branch
2 times, most recently
from
September 27, 2026 21:13
8ca182d to
9c25c8f
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries-v2
branch
from
September 27, 2026 21:13
22e90ad to
8f84807
Compare
🎨 Perfetto UI Builds & Tests
|
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries-v2
branch
from
September 27, 2026 22:55
8f84807 to
a4caa76
Compare
LalitMaganti
force-pushed
the
dev/lalitm/analysis-cte-scopes
branch
from
September 27, 2026 22:55
9c25c8f to
3124216
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries-v2
branch
from
September 27, 2026 23:22
a4caa76 to
7fc87f4
Compare
LalitMaganti
force-pushed
the
dev/lalitm/analysis-cte-scopes
branch
2 times, most recently
from
September 27, 2026 23:45
1673785 to
7ff5acd
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries-v2
branch
2 times, most recently
from
September 28, 2026 03:53
37b398c to
ad81123
Compare
LalitMaganti
force-pushed
the
dev/lalitm/analysis-cte-scopes
branch
from
September 28, 2026 03:53
7ff5acd to
9e2c303
Compare
Pipelines can only be whole statements or the body of a CREATE PERFETTO TABLE, so nothing written in SQL can use one: not a join, a CTE, a view or a function. That is what stops users like wattson moving off the interval intersection macros. A pipeline in parentheses now reads like any other subquery, in a FROM clause or as a CTE: SELECT t.id, p.total FROM tree t JOIN (FROM tree |> TREE ACCUMULATE UP SUM(self) AS total) p USING (id) Once a statement is parsed, the parser compiles each such pipeline and replaces it with SQL running its plan, recorded as a rewrite where it was written, as a macro call is. So SQL and pipelines nest either way round, pipelines work in macros and macros in pipelines, and views, functions and triggers need nothing more than they already do. As the SQL a pipeline reads is evaluated there too, it can read a function's arguments. The rewrite tree now covers pipelines as well as macro calls. A pipeline's rewrite stands in for the macro calls written inside it, and a node written inside a macro's expansion is taken from there, so errors trace back through the call. SQL reading a pipeline of the same statement is described by the plan that pipeline compiled to. A pipeline read by a pipeline is not expanded: the compiler reads it as one more kind of relation, next to dataframes and SQL, and compiles it into the same plan in a scope of its own. That also lets it be an interval intersection's operand. A pipeline can read the CTEs of the statement it is written in, as the SQL it reads is described where it is written.
LalitMaganti
force-pushed
the
dev/lalitm/analysis-cte-scopes
branch
from
September 28, 2026 04:21
9e2c303 to
502bb9c
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries-v2
branch
from
September 28, 2026 04:21
ad81123 to
6933c43
Compare
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pipelines can only be whole statements or the body of a
CREATE PERFETTO TABLE, so nothing written in SQL can use one: not a join, a CTE, a view or a function. That is what stops users like wattson moving off the interval intersection macros.A pipeline in parentheses now reads like any other subquery, in a
FROMclause or as a CTE:How:
Once a statement is parsed, each such pipeline is compiled and replaced with SQL running its plan, recorded as a rewrite where it was written, like a macro call. So SQL and pipelines nest either way round, pipelines work in macros and macros in pipelines, and views, functions and triggers need nothing new.
The SQL a pipeline reads is evaluated there too, so it can read a function's arguments.
A pipeline's rewrite stands in for the macro calls written inside it, and errors trace back through macro calls.
SQL reading a pipeline of the same statement is described by the plan that pipeline compiled to.
A pipeline read by a pipeline is compiled into the same plan, in a scope of its own, so it can also be an interval intersection's operand.
A pipeline can read the CTEs of the statement it is written in (via tp: let semantic analysis see CTEs #7643).