sbalabanov · Nov 6, 2026 · Nov 6, 2026
@@ -55,7 +55,7 @@ Every write is a compare-and-swap: a writer that loses re-reads on a later run.
 Verdicts are controller-owned facts: the Speculator can neither compute nor veto them.
 
 - **Land.** Each path carries an assumption about every dependency — *succeeds* (built on top of) or *fails* (built without). Normally, once a path's build has passed and every dependency has finished the way the path assumed — one assumed *succeeds* has landed, one assumed *fails* has failed or been cancelled — the speculate controller moves the head to Landing and hands it to Runway. A dependency that is merely *landing* has not finished, because a land can fail, so a single matching path still waits for the answer. Complete passed coverage is the exception described in Bypass large diff: it lets a head land before those answers arrive. If the hand-off is lost, the next run re-sends it. The same run sets the head's remaining in-flight paths *cancelling*: once the head can land they cannot help, and they hold CI slots until they stop. The landsignal controller records Runway's terminal result: success marks the head Succeeded, while failure marks it Failed. The result publishes a single dirty signal — no per-dependent fan-out — and the next run refutes paths whose assumption disagrees with the result: *fails* assumptions after success, *succeeds* assumptions after failure. The hand-off is idempotent, so Runway reports success without another land when the change is already present. A chain ordinarily lands one at a time, but a fully covered head can bypass its unsettled predecessors.
-- **Failure (no viable path).** A batch fails when every possible future has a failed build — no path can pass, so it can never land.
+- **Failure (no viable path).** A Speculating batch fails only after every dependency is terminal, at least one stored path remains unbroken, and every unbroken path is Failed or Cancelled. Cancellation consumes the execution attempt without rewriting the observed Cancelled fact as a test failure. Unfunded futures and unresolved dependencies still wait; a passed alternative still wins. Terminal paths cannot be retried automatically.
 - **Cancel.** A cancelled batch is driven terminal: its in-flight paths are set *cancelling*, then the batch is marked Cancelled once they stop (see Cancellation).
 
 ### Conflict relaxation
@@ -84,7 +84,7 @@ Cancellation is best-effort: a batch marked *cancelling* may still land if a lan
 
 Two kinds of cancel, split by owner:
 
-- **Preemption (extension).** The Speculator may propose `Cancel` on an in-flight path whose head is Speculating to free budget for a better candidate — its only cancel power. The controller validates it (never a passed path) and enacts it.
+- **Preemption (extension).** The Speculator may propose `Cancel` on an in-flight path whose head is Speculating to free budget for a better candidate — its only cancel power. The controller validates it (never a passed path) and enacts it. A viable path cancelled by a custom preempting Speculator consumes that attempt and can exhaust the head once dependencies settle; the shipped sticky allocator does not preempt.
 - **Correctness cancels (controller).** Refutation — a resolved dependency breaks a path's assumption, so the controller cancels that path — and batch cancellation (below). The Speculator is not consulted.
 
 Batch cancellation (user-initiated) marks the batch *cancelling*: a non-terminal intent that halts new work. The speculate run drives it terminal — it sets its in-flight paths *cancelling*, marks the batch *cancelled* once they stop, recomputes dependent paths against that outcome (those that assume it fails keep building; those that assume it succeeds are refuted), and concludes the contained requests.
@@ -121,3 +121,9 @@ Signatures live in code and are not copied here, so they cannot drift. This sect
 **Scorer** — [`submitqueue/extension/speculation/scorer`](../../../submitqueue/extension/speculation/scorer/README.md). `Score(ctx, batch, paths)` is how likely the batch is to reach Succeeded. The default impl is evidence wrapping a base (heuristic or composite): the base prices the change and ignores `paths`; evidence revises that price from the path set and batch state. See [outcome-scorer.md](outcome-scorer.md).
 
 **Generator and Allocator** — [`generator`](../../../submitqueue/extension/speculation/generator/README.md) and [`allocator`](../../../submitqueue/extension/speculation/allocator/README.md), the two composition points inside the default Speculator. The Generator opens a pull-based stream of candidate paths over the batches and path sets; the Allocator spends the build budget over that stream, reconciling it against the path sets. Both abort on a cancelled context.
+
+## Recovering quiet queues with cancelled paths
+
+After deploying the exhaustion policy and draining older speculation consumers, run `./tool/bazel run //tool/submitqueue/recovercancelledpaths -- --queue <queue> --recovery-id <operator-run-id> --dry-run` with the existing `MYSQL_DSN` configuration. Repeat `--queue` to explicitly bound the selection. The command reports the numerically earliest affected Speculating head per queue; a missing path set is unfunded, not affected. Dry-run is the default and never opens the queue database.
+
+Review the reported IDs, then repeat with `--apply` instead of `--dry-run` and configure `QUEUE_MYSQL_DSN`. Applying publishes one ordinary speculate nudge per affected queue; it never edits outcomes or invokes the planner directly. Retry an interrupted invocation with the same recovery ID so successful nudges deduplicate; use a new ID for a deliberately new recovery. Any inspection or publication error exits nonzero, while other selected queues are still attempted. Verify the ordinary conclude/request pipeline reaches terminal status. Do not apply against older consumers: they can acknowledge the wake-up without resolving the exhausted head. Rolling back the policy can reintroduce the stall but must not reverse terminal outcomes or replay submissions.
@@ -13,6 +13,7 @@ go_library(
         "//service/submitqueue:__subpackages__",
         "//submitqueue:__subpackages__",
         "//test:__subpackages__",
+        "//tool/submitqueue/recovercancelledpaths:__pkg__",
     ],
     deps = [
         "//api/base/change/protopb:go_default_library",

@@ -151,10 +151,8 @@ type SpeculationPathEntry struct {
 	Path SpeculationPath
 	// Status is the lifecycle status of the current build attempt.
 	Status SpeculationPathStatus
-	// Attempt is the build attempt number for this path, starting at 1. A path
-	// can be built more than once (e.g. after a prior build is cancelled to free
-	// budget, or fails and is retried), so ID alone does not identify an
-	// execution — (ID, Attempt) does. It increments with each new build.
+	// Attempt is the build attempt number for this path, starting at 1.
+	// The pair (ID, Attempt) uniquely identifies an execution.
 	Attempt int
 	// Version is the version of the object. It is used for optimistic locking.
 	// Versioning starts at 1 and is incremented for each change to the object.
@@ -192,7 +190,7 @@ type PathAction string
 const (
 	// PathActionUnknown is the zero-value sentinel; it is never a valid action.
 	PathActionUnknown PathAction = ""
-	// PathActionBuild proposes starting (or resurrecting) a build for the path.
+	// PathActionBuild proposes starting a build for the path.
 	PathActionBuild PathAction = "build"
 	// PathActionCancel proposes preempting an in-flight path to free build budget.
 	PathActionCancel PathAction = "cancel"

@@ -134,6 +134,13 @@ func TestFilterProposals_Rejects(t *testing.T) {
 				entryFor(valid, entity.SpeculationPathStatusPassed)),
 			want: rejectCancelPassed,
 		},
+		{
+			name:     "build would resurrect a cancelled path",
+			proposal: entity.Speculation{Path: valid, Action: entity.PathActionBuild},
+			snap: checkSnapshot(entity.BatchStateSpeculating,
+				entryFor(valid, entity.SpeculationPathStatusCancelled)),
+			want: rejectPathTerminal,
+		},
 		{
 			name:     "build would resurrect a finished path",
 			proposal: entity.Speculation{Path: valid, Action: entity.PathActionBuild},

@@ -54,7 +54,7 @@
 //     cancelled to free the slot.
 //   - A fails and P2 passed: B lands without A, again with no new build.
 //     P1's guess is broken.
-//   - A resolved either way, and every unbroken path failed: no future
+//   - A resolved either way, and every unbroken path failed or was cancelled: no future
 //     remains in which B passes, so B fails.
 //
 // # The life of a path
@@ -85,9 +85,9 @@
 // stops it, and only an observation of that stop (or proof nothing was ever
 // dispatched) moves the path to cancelled. The intent needs no dispatch of its
 // own — the poll loop reads it off the set and asks the runner to stop the
-// build. A terminal path can be resurrected by a new build proposal — status
-// returns to pending and Attempt increments, the one backwards step in the
-// diagram.
+// build. Terminal paths cannot be rebuilt. Failed and cancelled attempts are
+// exhausted once dependencies settle; preempting a viable path consumes its
+// attempt too. The sticky allocator does not preempt.
 //
 // # The life of a batch, as seen from here
 //

@@ -365,7 +365,7 @@ func (c *Controller) applyOutcome(ctx context.Context, store orchstorage.Storage
 		// Cancellation carries none — the cancel path owns that reason.
 		var concludeMeta map[string]string
 		if decision == outcomeFail {
-			concludeMeta = map[string]string{topickey.MetadataKeyFailureReason: "no speculation path could pass; every candidate build failed"}
+			concludeMeta = map[string]string{topickey.MetadataKeyFailureReason: "no speculation path could pass; every viable build failed or was cancelled"}
 		}
 		// Named for the run that decided it, so a redelivery re-deriving the
 		// same outcome does not conclude the batch twice, and so it stays

@@ -154,8 +154,8 @@ func allAssumptionsSettled(path entity.SpeculationPath, snap snapshot) bool {
 }
 
 // hasNoViableFuture reports whether the head can never pass: every dependency
-// has resolved, and every path consistent with how they resolved has a failed
-// build.
+// has resolved, and every path consistent with how they resolved has exhausted
+// its attempt with a failed or cancelled build.
 //
 // Deliberately conservative — failing a batch that could still have landed is
 // far worse than failing it a tick later, so this errs toward waiting:
@@ -178,7 +178,7 @@ func hasNoViableFuture(head entity.Batch, set entity.SpeculationPathSet, snap sn
 			continue
 		}
 		live++
-		if entry.Status != entity.SpeculationPathStatusFailed {
+		if entry.Status != entity.SpeculationPathStatusFailed && entry.Status != entity.SpeculationPathStatusCancelled {
 			return false
 		}
 	}

@@ -352,41 +352,38 @@ func TestHasNoViableFuture(t *testing.T) {
 		pathOver(entity.DependencyAssumptionSucceeds, entity.DependencyAssumptionSucceeds),
 		entity.SpeculationPathStatusFailed)
 
-	t.Run("waits while a dependency is unresolved", func(t *testing.T) {
-		// An unresolved dependency means futures the queue has not tried yet.
-		snap := snapWith(entity.BatchStateSucceeded, entity.BatchStateSpeculating)
-		assert.False(t, hasNoViableFuture(headBatch, setOf(failed), snap))
-	})
-
-	t.Run("fails once everything is resolved and every live path failed", func(t *testing.T) {
-		snap := snapWith(entity.BatchStateSucceeded, entity.BatchStateSucceeded)
-		assert.True(t, hasNoViableFuture(headBatch, setOf(failed), snap))
-	})
-
-	t.Run("does not fail while a path is still running", func(t *testing.T) {
-		running := entryFor(
-			pathOver(entity.DependencyAssumptionSucceeds, entity.DependencyAssumptionSucceeds),
-			entity.SpeculationPathStatusBuilding)
-		running.ID = "still-running"
-		snap := snapWith(entity.BatchStateSucceeded, entity.BatchStateSucceeded)
-		assert.False(t, hasNoViableFuture(headBatch, setOf(failed, running), snap))
-	})
-
-	t.Run("does not fail a head with nothing funded", func(t *testing.T) {
-		snap := snapWith(entity.BatchStateSucceeded, entity.BatchStateSucceeded)
-		assert.False(t, hasNoViableFuture(headBatch, setOf(), snap),
-			"an unfunded head is waiting for the Speculator, not out of options")
-	})
-
-	t.Run("ignores broken paths when judging", func(t *testing.T) {
-		// This path assumed dep1 would fail; it succeeded, so the failed build
-		// tells us nothing about a future that can still happen.
-		brokenFail := entryFor(
-			pathOver(entity.DependencyAssumptionFails, entity.DependencyAssumptionSucceeds),
-			entity.SpeculationPathStatusFailed)
-		snap := snapWith(entity.BatchStateSucceeded, entity.BatchStateSucceeded)
-		assert.False(t, hasNoViableFuture(headBatch, setOf(brokenFail), snap))
-	})
+	cancelled := failed
+	cancelled.Status = entity.SpeculationPathStatusCancelled
+	settled := snapWith(entity.BatchStateSucceeded, entity.BatchStateSucceeded)
+	for _, status := range []entity.SpeculationPathStatus{
+		entity.SpeculationPathStatusFailed, entity.SpeculationPathStatusCancelled,
+	} {
+		t.Run(string(status), func(t *testing.T) {
+			entry := failed
+			entry.Status = status
+			assert.True(t, hasNoViableFuture(headBatch, setOf(entry), settled))
+			assert.False(t, hasNoViableFuture(headBatch, setOf(entry),
+				snapWith(entity.BatchStateSucceeded, entity.BatchStateSpeculating)))
+			assert.True(t, hasNoViableFuture(entity.Batch{ID: head},
+				setOf(entryFor(entity.SpeculationPath{Head: head}, status)), snapshot{}))
+			broken := entry
+			broken.Path = pathOver(entity.DependencyAssumptionFails, entity.DependencyAssumptionSucceeds)
+			assert.False(t, hasNoViableFuture(headBatch, setOf(broken), settled))
+		})
+	}
+	assert.True(t, hasNoViableFuture(headBatch, setOf(failed, cancelled), settled))
+	assert.False(t, hasNoViableFuture(headBatch, setOf(), settled))
+	for _, status := range []entity.SpeculationPathStatus{
+		entity.SpeculationPathStatusUnknown, entity.SpeculationPathStatusPending,
+		entity.SpeculationPathStatusBuilding, entity.SpeculationPathStatusCancelling,
+		entity.SpeculationPathStatusPassed,
+	} {
+		t.Run("not exhausted "+string(status), func(t *testing.T) {
+			entry := failed
+			entry.Status = status
+			assert.False(t, hasNoViableFuture(headBatch, setOf(cancelled, entry), settled))
+		})
+	}
 }
 
 func TestDecide(t *testing.T) {
@@ -406,6 +403,10 @@ func TestDecide(t *testing.T) {
 
 	assert.Equal(t, outcomeLand, decide(headBatch, setOf(passed), allResolved))
 	assert.Equal(t, outcomeFail, decide(headBatch, setOf(failed), allResolved))
+	cancelled := failed
+	cancelled.Status = entity.SpeculationPathStatusCancelled
+	assert.Equal(t, outcomeFail, decide(headBatch, setOf(cancelled), allResolved))
+	assert.Equal(t, outcomeLand, decide(headBatch, setOf(cancelled, passed), allResolved))
 	assert.Equal(t, outcomeWait, decide(headBatch, setOf(building), allResolved))
 
 	// A passed path wins over a failed sibling: one way through is enough.

@@ -792,10 +792,41 @@ func TestRun_FailedHeadConcludesAfterTheStateWrite(t *testing.T) {
 	// The failure reason rides the conclude message so conclude can stamp it on
 	// the failed requests' terminal log, without persisting it as batch state.
 	require.Len(t, h.messages, 1)
-	assert.Equal(t, "no speculation path could pass; every candidate build failed",
+	assert.Equal(t, "no speculation path could pass; every viable build failed or was cancelled",
 		h.messages[0].Metadata[topickey.MetadataKeyFailureReason])
 }
 
+func TestRun_ProviderCancellationExhaustsHead(t *testing.T) {
+	ctrl := gomock.NewController(t)
+	batch := entity.Batch{ID: head, Queue: "q", State: entity.BatchStateSpeculating, Version: 1}
+	h := newRunHarness(t, ctrl, &scriptedSpeculator{}, []entity.Batch{batch})
+	path := entity.SpeculationPath{Head: head}
+	h.pathSets.EXPECT().Get(gomock.Any(), head).Return(entity.SpeculationPathSet{
+		Head: head, Paths: []entity.SpeculationPathEntry{entryFor(path, entity.SpeculationPathStatusBuilding)}, Version: 1,
+	}, nil)
+	h.pathBuilds.EXPECT().Get(gomock.Any(), path.ID(), 1).Return(entity.PathBuild{BuildID: "b1"}, nil)
+	h.builds.EXPECT().Get(gomock.Any(), "b1").Return(entity.Build{ID: "b1", Status: entity.BuildStatusCancelled}, nil)
+	gomock.InOrder(
+		h.pathSets.EXPECT().Update(gomock.Any(), gomock.Any(), int32(1), int32(2)).DoAndReturn(
+			func(_ context.Context, set entity.SpeculationPathSet, _, _ int32) error {
+				assert.Equal(t, entity.SpeculationPathStatusCancelled, set.Paths[0].Status)
+				assert.Equal(t, 1, set.Paths[0].Attempt)
+				assert.Empty(t, h.published)
+				return nil
+			}),
+		h.batches.EXPECT().Update(gomock.Any(), updateTo{id: head, state: entity.BatchStateFailed}, int32(1), int32(2)).DoAndReturn(
+			func(context.Context, entity.Batch, int32, int32) error {
+				assert.Empty(t, h.published)
+				return nil
+			}),
+	)
+
+	require.NoError(t, h.run(head))
+	assert.Equal(t, []string{"conclude"}, h.published)
+	assert.Equal(t, "no speculation path could pass; every viable build failed or was cancelled", h.messages[0].Metadata[topickey.MetadataKeyFailureReason])
+	assert.Zero(t, h.spec.calls)
+}
+
 // A failure resolves a dependency, which can fail everything stacked on it. The
 // whole cascade finalizes inside one run: a single pass would only catch the
 // dependents that happen to come later in queue order, and the rest would wait

@@ -0,0 +1,48 @@
+load("@rules_go//go:def.bzl", "go_binary", "go_library", "go_test")
+
+go_library(
+    name = "go_default_library",
+    srcs = ["main.go"],
+    importpath = "github.com/uber/submitqueue/tool/submitqueue/recovercancelledpaths",
+    visibility = ["//visibility:private"],
+    deps = [
+        "//platform/base/id:go_default_library",
+        "//platform/consumer:go_default_library",
+        "//platform/extension/messagequeue/mysql:go_default_library",
+        "//platform/publish:go_default_library",
+        "//submitqueue/core/messagequeue:go_default_library",
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/storage:go_default_library",
+        "//submitqueue/orchestrator/core/batch:go_default_library",
+        "//submitqueue/orchestrator/extension/storage:go_default_library",
+        "//submitqueue/orchestrator/extension/storage/mysql:go_default_library",
+        "@com_github_uber_go_tally//:go_default_library",
+        "@org_uber_go_zap//:go_default_library",
+    ],
+)
+
+go_binary(
+    name = "recovercancelledpaths",
+    embed = [":go_default_library"],
+    visibility = ["//visibility:public"],
+)
+
+go_test(
+    name = "go_default_test",
+    srcs = ["main_test.go"],
+    embed = [":go_default_library"],
+    deps = [
+        "//platform/base/messagequeue:go_default_library",
+        "//platform/consumer:go_default_library",
+        "//platform/extension/messagequeue/mock:go_default_library",
+        "//submitqueue/core/messagequeue:go_default_library",
+        "//submitqueue/entity:go_default_library",
+        "//submitqueue/extension/storage:go_default_library",
+        "//submitqueue/extension/storage/mock:go_default_library",
+        "//submitqueue/orchestrator/extension/storage:go_default_library",
+        "//submitqueue/orchestrator/extension/storage/mock:go_default_library",
+        "@com_github_stretchr_testify//assert:go_default_library",
+        "@com_github_stretchr_testify//require:go_default_library",
+        "@org_uber_go_mock//gomock:go_default_library",
+    ],
+)