Repository navigation
fix(mission): add bounded retry path so a stalled mission can recover - #228
Open
jimmyallinai wants to merge 1 commit into
Open
jimmyallinai wants to merge 1 commit into
jimmyallinai wants to merge 1 commit into
Conversation
Author
|
Hi — checking in on this PR, it has been open for a week. I re-verified just now: it is still mergeable with no conflicts, and this repo has no CI configured so there are no failing checks on my side. Happy to adjust the retry limit, the retry endpoint, or the test coverage if you would like anything changed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fixes #223. When a required task in the current phase fails (e.g. the per-dispatch backstop force-resolves a hung dispatch), the mission stalls permanently.
POST /api/mission/resumecannot recover it: the failed task is terminal, the next tick re-stalls with the same reason, and there was no task-retry API and no way to clear the failed task.Root cause
The stall itself is intentional and fail-closed (
tick()refuses to advance a phase past a failed required task), but the recovery transition the design calls for was missing:TaskQueuehadfail()/complete()/remove()and no way to return afailedtask topending, even thoughdocs/MISSION_RECOVERY.mdalready prescribes "an explicit bounded retry transition" for failed actions.What this PR does
TaskQueue.retry(taskId)(src/mission/index.ts): resets afailedtask topending— clears the failure result, assignee, and run timestamps, and counts the retry on the task (Task.retryCount, new optional field). Onlyfailedtasks can be retried, at mostMAX_TASK_RETRIES(3) times; anything else returnsfalseand changes nothing. Emits a newtask:retriedevent, forwarded byMissionControl(which also gains aretryTask()passthrough).TempestCommand.retryTask(taskId)(src/index.ts): re-queues the task and, only when the mission is paused because of the stall (stallReasonset), clears the stall so the next tick dispatches it. A mission the operator paused manually stays paused.POST /api/tasks/:id/retry(src/server.ts): operator-gated endpoint following the existing mission-control patterns — 404 with no active mission / unknown task, 409 when the task is not retryable, otherwise broadcaststask:retriedand returns the re-queued task. Also listed in the server startup banner.The stall-on-failed-required-task behavior is unchanged; this only adds the missing, explicit recovery path. The War Room UI button is left out to keep this PR to the engine/API layer.
Tests
New
src/__tests__/task-retry.test.ts(8 tests):pendingwith failure state cleared andretryCountincremented, and the task becomes dispatchable again (getNext()/pendingCount)MAX_TASK_RETRIESre-queues, further retries returnfalseand the task staysfailedMissionControl.retryTaskpassthrough +task:retriedevent forwardingmission-status-endpoint.test.ts): 404 no mission, 404 unknown task, 409 not-failed, 200 re-queue + broadcastRed/green: all 8 new tests fail on the base commit (
retrydoes not exist) and pass with this change.Locally run:
npx vitest run src/__tests__/task-retry.test.ts src/__tests__/mission-recovery.test.ts src/__tests__/mission-status-endpoint.test.ts src/__tests__/mission-controls.test.ts src/__tests__/index.test.ts— 76 passed, 0 failed (includes the existing wedged-dispatch stall test inindex.test.ts)npx tsc --noEmit— cleannpx eslinton the touched files — identical problem count to the base tree (111, all pre-existing); no new findings