Skip to content

fix(dlq): stop DLQ reconcilers failing Landing batches - #999

Closed
sbalabanov wants to merge 1 commit into
mainfrom
sergeyb/sq-29-dlq-landing
Closed

sbalabanov wants to merge 1 commit into
mainfrom
sergeyb/sq-29-dlq-landing

Conversation

@sbalabanov

Copy link
Copy Markdown
Contributor

Why?

A Landing batch has been handed to Runway, so only its merge result can say whether it landed. Every batch DLQ reconciler failed it anyway. A merged change could then be reported as Error, and its dependents re-planned as if it had failed, which can land combinations no build tested.

Repro:

  1. Batch B passes and goes Landing; land publishes its merge to Runway.
  2. One of B's other path builds is still stopping. Its buildsignal poll fails three times (about 3 s) and dead-letters.
  3. The buildsignal DLQ moves B to Failed and its requests to Error.
  4. Runway merges B and reports SUCCEEDED; landsignal finds B terminal and keeps Failed.

The landsignal DLQ also fails B outright when a SUCCEEDED result's batch write fails three times, without reading the outcome.

What?

failBatch leaves a Landing batch to its merge result unless the caller knows no merge happened: the land DLQ (the merge was never requested) and the landsignal DLQ for a not-landed result. For a SUCCEEDED result, the landsignal DLQ moves the batch to Succeeded and its requests to Landed.

Test Plan

make fmt && make lint && make check-gazelle
./tool/bazel test //submitqueue/orchestrator/... --test_output=errors

🤖 Generated with Claude Code

A Landing batch has been handed to Runway, so only its merge result can
say whether it landed. Every batch DLQ reconciler failed it anyway: a
buildsignal, speculate, build, dependency-analysis or conclude dead
letter could mark a merged change Error, and the landsignal DLQ failed
the batch even when Runway reported the merge succeeded.

failBatch now leaves a Landing batch alone unless the caller knows no
merge happened: the land DLQ, whose merge was never requested, and the
landsignal DLQ for a not-landed result. The landsignal DLQ records a
succeeded merge as Succeeded with its requests Landed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sbalabanov

Copy link
Copy Markdown
Contributor Author

sergeyb: The problem is real (race condition between actual land and build signal), but the proposed fix is too complex to justify. Reliably simpler options are not found. Closing the diff without a fix; a broader discussion need to be initiated on how to solve merge-specific race conditions; i.e. when the change is merged atomically to source control while other components mark it as failed for other different reasons.

@sbalabanov sbalabanov closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants