Skip to content

tp: keep a pipeline's rows when SQLite reads it again - #7633

Closed
LalitMaganti wants to merge 1 commit into
dev/lalitm/pipeline-subqueriesfrom
dev/lalitm/pipeline-reread
Closed

LalitMaganti wants to merge 1 commit into
dev/lalitm/pipeline-subqueriesfrom
dev/lalitm/pipeline-reread

Conversation

@LalitMaganti

Copy link
Copy Markdown
Member

SQLite reads the inner side of a join once for every row of the outer side. When a pipeline is on the inner side it runs again each time, including every pipeline it reads through SQL, which makes a join against a pipeline quadratic.

This adds Memoize, a source which passes on another source's batches while keeping them, and replays them on later reads once a read has reached the end. A read stopped part way, or one whose batches can't be kept, keeps nothing.

The pipeline table function puts one in front of a plan the second time the same cursor reads it, so a pipeline read once pays nothing extra and one read any number of times runs at most twice.

@LalitMaganti
LalitMaganti added this pull request to stack #7629 September 27, 2026 06:41
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-reread branch from eb3ac61 to 9221acc Compare September 27, 2026 06:43
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries branch from e452ef7 to 7bc6f63 Compare September 27, 2026 06:43
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-reread branch from 9221acc to c64d473 Compare September 27, 2026 16:26
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries branch from 7bc6f63 to 89446a9 Compare September 27, 2026 16:26
@LalitMaganti
LalitMaganti removed this pull request from stack #7629 September 27, 2026 16:27
@LalitMaganti
LalitMaganti added this pull request to stack #7636 September 27, 2026 16:28
@LalitMaganti
LalitMaganti marked this pull request as ready for review September 27, 2026 16:28
@LalitMaganti
LalitMaganti requested a review from a team as a code owner September 27, 2026 16:29
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries branch from 89446a9 to b2c706b Compare September 27, 2026 17:38
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-reread branch from c64d473 to 7ad44f8 Compare September 27, 2026 17:38
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries branch from b2c706b to fcc8471 Compare September 27, 2026 19:17
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-reread branch from 7ad44f8 to 57efc33 Compare September 27, 2026 19:17
@LalitMaganti
LalitMaganti marked this pull request as draft September 27, 2026 19:36
@LalitMaganti
LalitMaganti removed this pull request from stack #7636 September 27, 2026 19:36
@LalitMaganti
LalitMaganti added this pull request to stack #7639 September 27, 2026 19:37
SQLite reads the inner side of a join once for every row of the outer
side. When a pipeline is on the inner side, it runs again each time,
including every pipeline it reads through SQL, which makes a join
against a pipeline quadratic.

Add Memoize, a source which passes on another source's batches while
keeping them, and replays them on later reads once a read has reached
the end. The pipeline table function puts one in front of a plan the
second time the same cursor reads it, so a pipeline read once pays
nothing extra and one read any number of times runs at most twice.
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-subqueries branch from fcc8471 to 53cf2d7 Compare September 27, 2026 19:41
@LalitMaganti
LalitMaganti force-pushed the dev/lalitm/pipeline-reread branch from 57efc33 to a8afe5b Compare September 27, 2026 19:41
@LalitMaganti

Copy link
Copy Markdown
Member Author

Closing for now. Inputs are now collected once, so this caching may not be worth it any more; if it is, it will come back on top of #7644.

@LalitMaganti
LalitMaganti deleted the dev/lalitm/pipeline-reread branch September 27, 2026 20:06
@LalitMaganti
LalitMaganti removed this pull request from stack #7639 September 27, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant