[Coral-Incremental] Incremental plan generation for RelNode incremental rewrite#520
[Coral-Incremental] Incremental plan generation for RelNode incremental rewrite#520yyy1000 wants to merge 24 commits intolinkedin:masterfrom
Conversation
Subquery
| * - For other RelNodes: recursively processing their inputs to ensure uniformity. | ||
| * <p> |
There was a problem hiding this comment.
Not clear what this achieves.
There was a problem hiding this comment.
Does the example below show what it's doing?
There was a problem hiding this comment.
I am referring to "to ensure uniformity". The idea about the whole method is to ensure uniformity. Making it exclusive to "other RelNodes" does not make sense.
|
I think overall, the PR could benefit from more extensive documentation. Current documentation on different methods is a good start. Each documentation should be expanded with concrete running example with actual table names in the plan, input and output. Each method responsibility throughout the transformation process could be illustrated by this running example. If you add the table names to your current example, it will make the documentation much more clear and concrete by giving examples of the input and output of each method. |
What changes are proposed in this pull request, and why are they necessary?
Introduction: The goal of this PR is to introduce Incremental Plan Generation for Coral-Incremental. This feature will enable the generation of multiple incremental plans for a given RelNode (a logical plan) based on the number of sub-queries that will be computed incrementally. The number of generated plans will match the number of sub-queries in a SQL query. Each logical plan in these generated plans will update the materialized view with its results.
For example, in a multi-table join, we can choose between batch and incremental execution stages. Consider the following three-table join:
We could generate plans as follows:
Each generated plan will be represented as a List, where each RelNode indicates whether its corresponding sub-query will be executed incrementally or in batch mode.
Some important changes:
RelNodeGenerationTransformerto rewrite incremental format RelNodes by materializing the join subqueries, and combine them to generate different complete plans.RelNodeGenerationTransformer:findJoinNeedsProject: Converts allRelNodesinto a uniform format for future incremental plan generation.uniformFormat: Ensures consistent formatting ofRelNodes.convertRelPrevmethod which could convert the tables of aRelNodeinto '_prev' format, thus distinguishing from the original one.How was this patch tested?
Unit test