Repository navigation
fix(dlq): stop DLQ reconcilers failing Landing batches - #999
Closed
sbalabanov wants to merge 1 commit into
Closed
sbalabanov wants to merge 1 commit into
sbalabanov wants to merge 1 commit into
Conversation
A Landing batch has been handed to Runway, so only its merge result can say whether it landed. Every batch DLQ reconciler failed it anyway: a buildsignal, speculate, build, dependency-analysis or conclude dead letter could mark a merged change Error, and the landsignal DLQ failed the batch even when Runway reported the merge succeeded. failBatch now leaves a Landing batch alone unless the caller knows no merge happened: the land DLQ, whose merge was never requested, and the landsignal DLQ for a not-landed result. The landsignal DLQ records a succeeded merge as Succeeded with its requests Landed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
sergeyb: The problem is real (race condition between actual land and build signal), but the proposed fix is too complex to justify. Reliably simpler options are not found. Closing the diff without a fix; a broader discussion need to be initiated on how to solve merge-specific race conditions; i.e. when the change is merged atomically to source control while other components mark it as failed for other different reasons. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why?
A
Landingbatch has been handed to Runway, so only its merge result can say whether it landed. Every batch DLQ reconciler failed it anyway. A merged change could then be reported as Error, and its dependents re-planned as if it had failed, which can land combinations no build tested.Repro:
Landing; land publishes its merge to Runway.Failedand its requests toError.SUCCEEDED; landsignal finds B terminal and keepsFailed.The landsignal DLQ also fails B outright when a
SUCCEEDEDresult's batch write fails three times, without reading the outcome.What?
failBatchleaves aLandingbatch to its merge result unless the caller knows no merge happened: thelandDLQ (the merge was never requested) and thelandsignalDLQ for a not-landed result. For aSUCCEEDEDresult, thelandsignalDLQ moves the batch toSucceededand its requests toLanded.Test Plan
make fmt && make lint && make check-gazelle./tool/bazel test //submitqueue/orchestrator/... --test_output=errors🤖 Generated with Claude Code