tp: allow pipelines as subqueries - #7632
Closed
LalitMaganti wants to merge 1 commit into
Closed
LalitMaganti wants to merge 1 commit into
LalitMaganti wants to merge 1 commit into
Conversation
LalitMaganti
added this pull request to stack #7629
September 27, 2026 06:41
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries
branch
from
September 27, 2026 06:43
e452ef7 to
7bc6f63
Compare
🎨 Perfetto UI Builds & Tests
|
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 27, 2026 16:26
869482c to
e4cc368
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries
branch
from
September 27, 2026 16:26
7bc6f63 to
89446a9
Compare
LalitMaganti
removed this pull request from stack #7629
September 27, 2026 16:27
LalitMaganti
added this pull request to stack #7636
September 27, 2026 16:28
LalitMaganti
marked this pull request as ready for review
September 27, 2026 16:28
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries
branch
from
September 27, 2026 17:38
89446a9 to
b2c706b
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 27, 2026 17:38
e4cc368 to
53fc7ae
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries
branch
from
September 27, 2026 19:17
b2c706b to
fcc8471
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 27, 2026 19:17
53fc7ae to
ef42082
Compare
LalitMaganti
marked this pull request as draft
September 27, 2026 19:36
LalitMaganti
removed this pull request from stack #7636
September 27, 2026 19:36
LalitMaganti
added this pull request to stack #7639
September 27, 2026 19:37
Pipelines can only be whole statements or the body of a CREATE PERFETTO TABLE, so nothing written in SQL can use one: not a join, a CTE, a view or a function. That is what stops users like wattson moving off the interval intersection macros. A pipeline in parentheses now reads like any other subquery, in a FROM clause or as a CTE: SELECT t.id, p.total FROM tree t JOIN (FROM tree |> TREE ACCUMULATE UP SUM(self) AS total) p USING (id) The grammar hands each such pipeline to the parser's node expander once it is parsed. It is compiled there and replaced with SQL reading its serialized plan, recorded as a rewrite where it was written, as a macro call is. So SQL and pipelines nest either way round, pipelines work in macros and macros in pipelines, and views, functions and triggers need nothing more than they already do. The parser's rewrite tree now covers pipelines as well as macro calls. A pipeline's rewrite stands in for the macro calls written inside it, and a node written inside a macro's expansion is taken from there, so errors trace back through the call. A pipeline read by a pipeline is not expanded: the compiler reads it as one more kind of relation, next to dataframes and SQL, and compiles it into the same plan in a scope of its own. That also lets it be an interval intersection's operand. A pipeline's SQL source reading a function's arguments is refused when the pipeline is compiled: nothing binds them there, so they would otherwise silently read as NULL.
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 27, 2026 19:41
ef42082 to
f7d9206
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries
branch
from
September 27, 2026 19:41
fcc8471 to
53cf2d7
Compare
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pipelines can only be whole statements or the body of a
CREATE PERFETTO TABLE, so nothing written in SQL can use one: not a join, a CTE, a view or a function. That is what stops users like wattson moving off the interval intersection macros.A pipeline in parentheses now reads like any other subquery, in a FROM clause or as a CTE:
How it works
RewriteTreecovering pipelines as well as macro calls. It is built once per use and never changes. A pipeline's rewrite stands in for the macro calls written inside it (oneKeepOutermostrule). A node written inside a macro's expansion is taken as a slice of that expansion, so errors trace back through the call.CompileRelation), and compiles it into the same plan in a scope of its own. That also makes it usable as anINTERVAL INTERSECTION OFoperand.__intrinsic_pipeline, the column limit) moves topipeline_sql.h, out of serialization and the table function.Refused for now: a pipeline's SQL source reading a function's arguments (
$k), with an error when the pipeline is compiled. Nothing binds them there, so they would otherwise silently read as NULL. Supporting them later means passing them to the plan as parameters.Tests cover: