tp: run pipelines from their serialized plans - #7631
Draft
LalitMaganti wants to merge 1 commit into
Draft
LalitMaganti wants to merge 1 commit into
LalitMaganti wants to merge 1 commit into
Conversation
LalitMaganti
added this pull request to stack #7629
September 27, 2026 05:12
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 27, 2026 06:40
d3b771d to
869482c
Compare
LalitMaganti
marked this pull request as ready for review
September 27, 2026 14:06
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 27, 2026 16:26
869482c to
e4cc368
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipe-source-names
branch
from
September 27, 2026 16:26
3d87dcd to
4c156d0
Compare
LalitMaganti
removed this pull request from stack #7629
September 27, 2026 16:27
LalitMaganti
added this pull request to stack #7636
September 27, 2026 16:28
LalitMaganti
force-pushed
the
dev/lalitm/pipe-source-names
branch
from
September 27, 2026 17:38
4c156d0 to
d6d5bd9
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 27, 2026 17:38
e4cc368 to
53fc7ae
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipe-source-names
branch
from
September 27, 2026 19:17
d6d5bd9 to
cc61128
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 27, 2026 19:17
53fc7ae to
ef42082
Compare
LalitMaganti
marked this pull request as draft
September 27, 2026 19:36
LalitMaganti
removed this pull request from stack #7636
September 27, 2026 19:36
LalitMaganti
added this pull request to stack #7639
September 27, 2026 19:37
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 27, 2026 19:41
ef42082 to
f7d9206
Compare
LalitMaganti
removed this pull request from stack #7639
September 27, 2026 20:08
LalitMaganti
changed the base branch from
dev/lalitm/pipe-source-names
to
main
September 27, 2026 20:08
LalitMaganti
added this pull request to stack #7645
September 27, 2026 20:08
LalitMaganti
removed this pull request from stack #7645
September 27, 2026 20:08
LalitMaganti
changed the base branch from
main
to
dev/lalitm/pipe-source-names
September 27, 2026 20:08
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
2 times, most recently
from
September 27, 2026 23:22
bcde718 to
463a969
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipe-source-names
branch
from
September 28, 2026 04:21
cc61128 to
d204174
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 28, 2026 04:21
463a969 to
0834102
Compare
A pipeline statement currently runs through a TEMP virtual table created for it, with its execution plan bound to the statement as a pointer. The plan only lives as long as that one prepared statement, so SQL holding a pipeline can't be stored and run later, as views, functions and triggers do. That blocks letting pipelines appear inside ordinary SQL. Instead, write the pipeline's optimized logical plan into the SQL itself: SELECT c0 AS ts, c1 AS dur FROM __intrinsic_pipeline(X'...') `__intrinsic_pipeline` is a single built-in table function which loads the plan and lowers it when it runs. The plan is everything needed to run the pipeline, so the SQL works wherever it ends up, with nothing to keep alive or clean up alongside it. This removes the per-statement tables and the machinery that retired and dropped them. The plan is stored after optimization, so loading it is a matter of reading it back and lowering it: nothing is parsed or compiled again. Dataframes are looked up again by name, and a plan whose dataframes have changed shape since is refused. Anyone can write such bytes into SQL, so reading a plan checks it is one the compiler could have produced before anything runs it. Nothing changes for users yet: pipelines still appear only as whole statements or as the body of a CREATE PERFETTO TABLE.
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-table-function
branch
from
September 28, 2026 06:05
0834102 to
65f10ea
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A pipeline statement currently runs through a TEMP virtual table created for it, with its execution plan bound to the statement as a pointer. The plan only lives as long as that one prepared statement, so SQL holding a pipeline can't be stored and run later, as views, functions and triggers do. That blocks letting pipelines appear inside ordinary SQL.
Instead, this writes the pipeline's optimized logical plan into the SQL itself:
__intrinsic_pipelineis a single built-in table function which loads the plan and lowers it when it runs. The plan is everything needed to run the pipeline, so the SQL works wherever it ends up, with nothing to keep alive or clean up alongside it. This removes the per-statement tables and the machinery that retired and dropped them.The plan is stored after optimization, so loading it is a matter of reading it back and lowering it: nothing is parsed or compiled again. Dataframes are looked up again by name, and a plan whose dataframes have changed shape since is refused. Anyone can write such bytes into SQL, so reading a plan checks it is one the compiler could have produced before anything runs it: truncated and corrupted plans are tested to be refused, or to run safely.
Nothing changes for users yet: pipelines still appear only as whole statements or as the body of a
CREATE PERFETTO TABLE.