Skip to content

tp: run pipelines from their serialized plans - #7631

Draft
LalitMaganti wants to merge 1 commit into
dev/lalitm/pipe-source-namesfrom
dev/lalitm/pipeline-table-function
Draft

LalitMaganti wants to merge 1 commit into
dev/lalitm/pipe-source-namesfrom
dev/lalitm/pipeline-table-function

Conversation

@LalitMaganti

@LalitMaganti LalitMaganti commented Sep 27, 2026 •

Copy link
Copy Markdown
Member

A pipeline statement currently runs through a TEMP virtual table created for it, with its execution plan bound to the statement as a pointer. The plan only lives as long as that one prepared statement, so SQL holding a pipeline can't be stored and run later, as views, functions and triggers do. That blocks letting pipelines appear inside ordinary SQL.

Instead, this writes the pipeline's optimized logical plan into the SQL itself:

SELECT c0 AS ts, c1 AS dur FROM __intrinsic_pipeline(X'...')

__intrinsic_pipeline is a single built-in table function which loads the plan and lowers it when it runs. The plan is everything needed to run the pipeline, so the SQL works wherever it ends up, with nothing to keep alive or clean up alongside it. This removes the per-statement tables and the machinery that retired and dropped them.

The plan is stored after optimization, so loading it is a matter of reading it back and lowering it: nothing is parsed or compiled again. Dataframes are looked up again by name, and a plan whose dataframes have changed shape since is refused. Anyone can write such bytes into SQL, so reading a plan checks it is one the compiler could have produced before anything runs it: truncated and corrupted plans are tested to be refused, or to run safely.

Nothing changes for users yet: pipelines still appear only as whole statements or as the body of a CREATE PERFETTO TABLE.

@LalitMaganti
LalitMaganti added this pull request to stack #7629 September 27, 2026 05:12
@LalitMaganti LalitMaganti changed the title tp: run pipelines from their text [NOT FOR REVIEW] tp: run pipelines from their text Sep 27, 2026
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from d3b771d to 869482c Compare September 27, 2026 06:40
@LalitMaganti LalitMaganti changed the title [NOT FOR REVIEW] tp: run pipelines from their text tp: run pipelines from their serialized plans Sep 27, 2026
@LalitMaganti
LalitMaganti marked this pull request as ready for review September 27, 2026 14:06
@LalitMaganti
LalitMaganti requested a review from a team as a code owner September 27, 2026 14:06
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from 869482c to e4cc368 Compare September 27, 2026 16:26
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipe-source-names branch from 3d87dcd to 4c156d0 Compare September 27, 2026 16:26
@LalitMaganti
LalitMaganti removed this pull request from stack #7629 September 27, 2026 16:27
@LalitMaganti
LalitMaganti added this pull request to stack #7636 September 27, 2026 16:28
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipe-source-names branch from 4c156d0 to d6d5bd9 Compare September 27, 2026 17:38
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from e4cc368 to 53fc7ae Compare September 27, 2026 17:38
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipe-source-names branch from d6d5bd9 to cc61128 Compare September 27, 2026 19:17
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from 53fc7ae to ef42082 Compare September 27, 2026 19:17
@LalitMaganti
LalitMaganti marked this pull request as draft September 27, 2026 19:36
@LalitMaganti
LalitMaganti removed this pull request from stack #7636 September 27, 2026 19:36
@LalitMaganti
LalitMaganti added this pull request to stack #7639 September 27, 2026 19:37
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from ef42082 to f7d9206 Compare September 27, 2026 19:41
@LalitMaganti
LalitMaganti removed this pull request from stack #7639 September 27, 2026 20:08
@LalitMaganti
LalitMaganti changed the base branch from dev/lalitm/pipe-source-names to main September 27, 2026 20:08
@LalitMaganti
LalitMaganti added this pull request to stack #7645 September 27, 2026 20:08
@LalitMaganti
LalitMaganti removed this pull request from stack #7645 September 27, 2026 20:08
@LalitMaganti
LalitMaganti changed the base branch from main to dev/lalitm/pipe-source-names September 27, 2026 20:08
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch 2 times, most recently from bcde718 to 463a969 Compare September 27, 2026 23:22
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipe-source-names branch from cc61128 to d204174 Compare September 28, 2026 04:21
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from 463a969 to 0834102 Compare September 28, 2026 04:21
A pipeline statement currently runs through a TEMP virtual table created
for it, with its execution plan bound to the statement as a pointer. The
plan only lives as long as that one prepared statement, so SQL holding a
pipeline can't be stored and run later, as views, functions and triggers
do. That blocks letting pipelines appear inside ordinary SQL.

Instead, write the pipeline's optimized logical plan into the SQL itself:

  SELECT c0 AS ts, c1 AS dur FROM __intrinsic_pipeline(X'...')

`__intrinsic_pipeline` is a single built-in table function which loads
the plan and lowers it when it runs. The plan is everything needed to run
the pipeline, so the SQL works wherever it ends up, with nothing to keep
alive or clean up alongside it. This removes the per-statement tables and
the machinery that retired and dropped them.

The plan is stored after optimization, so loading it is a matter of
reading it back and lowering it: nothing is parsed or compiled again.
Dataframes are looked up again by name, and a plan whose dataframes have
changed shape since is refused. Anyone can write such bytes into SQL, so
reading a plan checks it is one the compiler could have produced before
anything runs it.

Nothing changes for users yet: pipelines still appear only as whole
statements or as the body of a CREATE PERFETTO TABLE.
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-table-function branch from 0834102 to 65f10ea Compare September 28, 2026 06:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant