Skip to content

tp: allow pipelines as subqueries - #7632

Closed
LalitMaganti wants to merge 1 commit into
dev/lalitm/pipeline-table-functionfrom
dev/lalitm/pipeline-subqueries
Closed

LalitMaganti wants to merge 1 commit into
dev/lalitm/pipeline-table-functionfrom
dev/lalitm/pipeline-subqueries

Conversation

@LalitMaganti

@LalitMaganti LalitMaganti commented Sep 27, 2026 •

Copy link
Copy Markdown
Member

Pipelines can only be whole statements or the body of a CREATE PERFETTO TABLE, so nothing written in SQL can use one: not a join, a CTE, a view or a function. That is what stops users like wattson moving off the interval intersection macros.

A pipeline in parentheses now reads like any other subquery, in a FROM clause or as a CTE:

SELECT t.id, p.total
FROM tree t
JOIN (FROM tree |> TREE ACCUMULATE UP SUM(self) AS total) p USING (id)

How it works

  • Parser: the grammar hands each such pipeline to syntaqlite's node expander once it is parsed (syntaqlite 0.11, rolled in tp: roll syntaqlite to 0.12.1 #7635). It is compiled there and replaced with SQL reading its serialized plan (tp: run pipelines from their serialized plans #7631), recorded as a rewrite where it was written, just like a macro call. So SQL and pipelines nest either way round, pipelines work in macros and macros in pipelines, and views, functions and triggers need nothing more than they already do.
  • Rewrite tree: the parser's macro rewrite builder becomes a RewriteTree covering pipelines as well as macro calls. It is built once per use and never changes. A pipeline's rewrite stands in for the macro calls written inside it (one KeepOutermost rule). A node written inside a macro's expansion is taken as a slice of that expansion, so errors trace back through the call.
  • Compiler: a pipeline read by another pipeline is not expanded. The compiler reads it as one more kind of relation, next to dataframes and SQL (CompileRelation), and compiles it into the same plan in a scope of its own. That also makes it usable as an INTERVAL INTERSECTION OF operand.
  • Layering: the SQL that reads a plan (__intrinsic_pipeline, the column limit) moves to pipeline_sql.h, out of serialization and the table function.

Refused for now: a pipeline's SQL source reading a function's arguments ($k), with an error when the pipeline is compiled. Nothing binds them there, so they would otherwise silently read as NULL. Supporting them later means passing them to the plan as parameters.

Tests cover:

  • pipelines in subqueries, views and functions;
  • pipelines in macros and macros in pipelines, including an error tracing back through the macro call;
  • pipelines as pipeline sources and as intersection operands;
  • the function-argument error;
  • SQL → pipeline → SQL → pipeline nesting over several batches: streaming, re-reads from a join, errors from the bottom level, stopping early and interleaved runs.

@LalitMaganti
LalitMaganti added this pull request to stack #7629 September 27, 2026 06:41
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries branch from e452ef7 to 7bc6f63 Compare September 27, 2026 06:43
@LalitMaganti LalitMaganti changed the title tp: allow pipelines as subqueries [NOT FOR REVIEW] tp: allow pipelines as subqueries Sep 27, 2026
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from 869482c to e4cc368 Compare September 27, 2026 16:26
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries branch from 7bc6f63 to 89446a9 Compare September 27, 2026 16:26
@LalitMaganti LalitMaganti changed the title [NOT FOR REVIEW] tp: allow pipelines as subqueries tp: allow pipelines as subqueries Sep 27, 2026
@LalitMaganti
LalitMaganti removed this pull request from stack #7629 September 27, 2026 16:27
@LalitMaganti
LalitMaganti added this pull request to stack #7636 September 27, 2026 16:28
@LalitMaganti
LalitMaganti marked this pull request as ready for review September 27, 2026 16:28
@LalitMaganti
LalitMaganti requested a review from a team as a code owner September 27, 2026 16:28
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries branch from 89446a9 to b2c706b Compare September 27, 2026 17:38
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from e4cc368 to 53fc7ae Compare September 27, 2026 17:38
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries branch from b2c706b to fcc8471 Compare September 27, 2026 19:17
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from 53fc7ae to ef42082 Compare September 27, 2026 19:17
@LalitMaganti
LalitMaganti marked this pull request as draft September 27, 2026 19:36
@LalitMaganti
LalitMaganti removed this pull request from stack #7636 September 27, 2026 19:36
@LalitMaganti
LalitMaganti added this pull request to stack #7639 September 27, 2026 19:37
Pipelines can only be whole statements or the body of a CREATE PERFETTO
TABLE, so nothing written in SQL can use one: not a join, a CTE, a view
or a function. That is what stops users like wattson moving off the
interval intersection macros.

A pipeline in parentheses now reads like any other subquery, in a FROM
clause or as a CTE:

  SELECT t.id, p.total
  FROM tree t
  JOIN (FROM tree |> TREE ACCUMULATE UP SUM(self) AS total) p USING (id)

The grammar hands each such pipeline to the parser's node expander once
it is parsed. It is compiled there and replaced with SQL reading its
serialized plan, recorded as a rewrite where it was written, as a macro
call is. So SQL and pipelines nest either way round, pipelines work in
macros and macros in pipelines, and views, functions and triggers need
nothing more than they already do.

The parser's rewrite tree now covers pipelines as well as macro calls.
A pipeline's rewrite stands in for the macro calls written inside it,
and a node written inside a macro's expansion is taken from there, so
errors trace back through the call.

A pipeline read by a pipeline is not expanded: the compiler reads it as
one more kind of relation, next to dataframes and SQL, and compiles it
into the same plan in a scope of its own. That also lets it be an
interval intersection's operand.

A pipeline's SQL source reading a function's arguments is refused when
the pipeline is compiled: nothing binds them there, so they would
otherwise silently read as NULL.
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from ef42082 to f7d9206 Compare September 27, 2026 19:41
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries branch from fcc8471 to 53cf2d7 Compare September 27, 2026 19:41
@LalitMaganti

Copy link
Copy Markdown
Member Author

Superseded by #7644 (pipelines as subqueries), with its refactors in #7642 and CTE support in #7643.

@LalitMaganti
LalitMaganti deleted the dev/lalitm/pipeline-subqueries branch September 27, 2026 20:06
@LalitMaganti
LalitMaganti removed this pull request from stack #7639 September 27, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant