consumer: correct the order of ddl and dml in storage consumer#4328
consumer: correct the order of ddl and dml in storage consumer#4328
Conversation
ti-chi-bot
bot
added
do-not-merge/needs-triage-completed
release-note
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request resolves an issue in the storage consumer where the order of processing Data Definition Language (DDL) and Data Manipulation Language (DML) events was incorrect. The changes introduce a revised sorting mechanism to ensure that for any given table version, all DML operations are processed and applied before any corresponding DDL operations, thereby maintaining data integrity and consistency during replication. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
ti-chi-bot
bot
added
the
size/M
📝 WalkthroughWalkthroughThe change refactors DML path key sorting in the storage consumer by replacing field-by-field comparisons with a dedicated comparator function ( Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request corrects the processing order of DDL and DML events in the storage consumer, ensuring that for the same table version, DML events are processed before DDL events. This is achieved by refactoring the sorting logic into a new dmlPathKeyLess function with the corrected comparison logic. The change is clear and addresses the issue. My main feedback is to add unit tests for the new critical sorting function to ensure its correctness and prevent future regressions.
| func dmlPathKeyLess(lhs, rhs cloudstorage.DmlPathKey) bool { | ||
| if lhs.TableVersion != rhs.TableVersion { | ||
| return lhs.TableVersion < rhs.TableVersion | ||
| } | ||
|
|
||
| // For the same table version(commitTs), execute schema(DDL) after all DML files. | ||
| if lhs.PartitionNum != rhs.PartitionNum { | ||
| if lhs.PartitionNum == fakePartitionNumForSchemaFile { | ||
| return false | ||
| } | ||
| if rhs.PartitionNum == fakePartitionNumForSchemaFile { | ||
| return true | ||
| } | ||
| return lhs.PartitionNum < rhs.PartitionNum | ||
| } | ||
|
|
||
| if lhs.Date != rhs.Date { | ||
| return lhs.Date < rhs.Date | ||
| } | ||
| if lhs.Schema != rhs.Schema { | ||
| return lhs.Schema < rhs.Schema | ||
| } | ||
| return lhs.Table < rhs.Table | ||
| } |
There was a problem hiding this comment.
The new function dmlPathKeyLess contains critical sorting logic for ensuring DDLs are processed after DMLs. While the logic appears correct, it would be beneficial to add a unit test for this function to prevent future regressions. The function is pure and should be straightforward to test with various DmlPathKey combinations, including cases with schema files and DML files for the same and different table versions.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
cmd/storage-consumer/consumer.go (1)
547-547:⚠️ Potential issue | 🔴 CriticalHandle DML flush errors before continuing to DDL.
Line 547 drops the
flushDMLEventserror. If flush fails, processing can continue and execute later DDL, breaking ordering guarantees under failure paths.Suggested fix
- c.flushDMLEvents(ctx, tableID) + if err := c.flushDMLEvents(ctx, tableID); err != nil { + return errors.Trace(err) + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@cmd/storage-consumer/consumer.go` at line 547, The call to c.flushDMLEvents(ctx, tableID) currently ignores any returned error which can allow subsequent DDL to run and break ordering; update the call site to capture the error (err := c.flushDMLEvents(ctx, tableID)), check it, and if non-nil log/return/propagate the error (e.g., return fmt.Errorf or propagate the existing function's error) so processing stops and DDL is not executed when DML flush fails, preserving ordering guarantees; ensure you reference c.flushDMLEvents and the enclosing function's error flow when adding the early-return or error propagation.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@cmd/storage-consumer/consumer.go`:
- Line 547: The call to c.flushDMLEvents(ctx, tableID) currently ignores any
returned error which can allow subsequent DDL to run and break ordering; update
the call site to capture the error (err := c.flushDMLEvents(ctx, tableID)),
check it, and if non-nil log/return/propagate the error (e.g., return fmt.Errorf
or propagate the existing function's error) so processing stops and DDL is not
executed when DML flush fails, preserving ordering guarantees; ensure you
reference c.flushDMLEvents and the enclosing function's error flow when adding
the early-return or error propagation.
ti-chi-bot
bot
added
needs-1-more-lgtm
ti-chi-bot
bot
added
lgtm
and removed
needs-1-more-lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: 3AceShowHand, wlwilliamx The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
[LGTM Timeline notifier]Timeline:
|
|
/retest |
|
@wk989898: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
wk989898
added
the
do-not-merge/hold
3 participants
What problem does this PR solve?
Issue Number: close #4327
What is changed and how it works?
DML events should be executed before DDL event in the same commitTs
Check List
Tests
Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note
Summary by CodeRabbit