Repository navigation
tp: keep a pipeline's rows when SQLite reads it again - #7633
Closed
LalitMaganti wants to merge 1 commit into
Closed
LalitMaganti wants to merge 1 commit into
LalitMaganti wants to merge 1 commit into
Conversation
LalitMaganti
added this pull request to stack #7629
September 27, 2026 06:41
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-reread
branch
from
September 27, 2026 06:43
eb3ac61 to
9221acc
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries
branch
from
September 27, 2026 06:43
e452ef7 to
7bc6f63
Compare
🎨 Perfetto UI Builds & Tests
|
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-reread
branch
from
September 27, 2026 16:26
9221acc to
c64d473
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries
branch
from
September 27, 2026 16:26
7bc6f63 to
89446a9
Compare
LalitMaganti
removed this pull request from stack #7629
September 27, 2026 16:27
LalitMaganti
added this pull request to stack #7636
September 27, 2026 16:28
LalitMaganti
marked this pull request as ready for review
September 27, 2026 16:28
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries
branch
from
September 27, 2026 17:38
89446a9 to
b2c706b
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-reread
branch
from
September 27, 2026 17:38
c64d473 to
7ad44f8
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries
branch
from
September 27, 2026 19:17
b2c706b to
fcc8471
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-reread
branch
from
September 27, 2026 19:17
7ad44f8 to
57efc33
Compare
LalitMaganti
marked this pull request as draft
September 27, 2026 19:36
LalitMaganti
removed this pull request from stack #7636
September 27, 2026 19:36
LalitMaganti
added this pull request to stack #7639
September 27, 2026 19:37
SQLite reads the inner side of a join once for every row of the outer side. When a pipeline is on the inner side, it runs again each time, including every pipeline it reads through SQL, which makes a join against a pipeline quadratic. Add Memoize, a source which passes on another source's batches while keeping them, and replays them on later reads once a read has reached the end. The pipeline table function puts one in front of a plan the second time the same cursor reads it, so a pipeline read once pays nothing extra and one read any number of times runs at most twice.
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-subqueries
branch
from
September 27, 2026 19:41
fcc8471 to
53cf2d7
Compare
LalitMaganti
force-pushed
the
dev/lalitm/pipeline-reread
branch
from
September 27, 2026 19:41
57efc33 to
a8afe5b
Compare
Member
Author
|
Closing for now. Inputs are now collected once, so this caching may not be worth it any more; if it is, it will come back on top of #7644. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SQLite reads the inner side of a join once for every row of the outer side. When a pipeline is on the inner side it runs again each time, including every pipeline it reads through SQL, which makes a join against a pipeline quadratic.
This adds
Memoize, a source which passes on another source's batches while keeping them, and replays them on later reads once a read has reached the end. A read stopped part way, or one whose batches can't be kept, keeps nothing.The pipeline table function puts one in front of a plan the second time the same cursor reads it, so a pipeline read once pays nothing extra and one read any number of times runs at most twice.