Skip to content

fix(mission): add bounded retry path so a stalled mission can recover - #228

Open
jimmyallinai wants to merge 1 commit into
elder-plinius:mainfrom
jimmyallinai:fix/task-retry-recovery
Open

jimmyallinai wants to merge 1 commit into
elder-plinius:mainfrom
jimmyallinai:fix/task-retry-recovery

Conversation

@jimmyallinai

Copy link
Copy Markdown

Problem

Fixes #223. When a required task in the current phase fails (e.g. the per-dispatch backstop force-resolves a hung dispatch), the mission stalls permanently. POST /api/mission/resume cannot recover it: the failed task is terminal, the next tick re-stalls with the same reason, and there was no task-retry API and no way to clear the failed task.

Root cause

The stall itself is intentional and fail-closed (tick() refuses to advance a phase past a failed required task), but the recovery transition the design calls for was missing: TaskQueue had fail() / complete() / remove() and no way to return a failed task to pending, even though docs/MISSION_RECOVERY.md already prescribes "an explicit bounded retry transition" for failed actions.

What this PR does

  • TaskQueue.retry(taskId) (src/mission/index.ts): resets a failed task to pending — clears the failure result, assignee, and run timestamps, and counts the retry on the task (Task.retryCount, new optional field). Only failed tasks can be retried, at most MAX_TASK_RETRIES (3) times; anything else returns false and changes nothing. Emits a new task:retried event, forwarded by MissionControl (which also gains a retryTask() passthrough).
  • TempestCommand.retryTask(taskId) (src/index.ts): re-queues the task and, only when the mission is paused because of the stall (stallReason set), clears the stall so the next tick dispatches it. A mission the operator paused manually stays paused.
  • POST /api/tasks/:id/retry (src/server.ts): operator-gated endpoint following the existing mission-control patterns — 404 with no active mission / unknown task, 409 when the task is not retryable, otherwise broadcasts task:retried and returns the re-queued task. Also listed in the server startup banner.

The stall-on-failed-required-task behavior is unchanged; this only adds the missing, explicit recovery path. The War Room UI button is left out to keep this PR to the engine/API layer.

Tests

New src/__tests__/task-retry.test.ts (8 tests):

  • retry resets a failed task to pending with failure state cleared and retryCount incremented, and the task becomes dispatchable again (getNext() / pendingCount)
  • retry refuses non-failed tasks (pending/completed) and unknown ids
  • the bound holds: after MAX_TASK_RETRIES re-queues, further retries return false and the task stays failed
  • MissionControl.retryTask passthrough + task:retried event forwarding
  • the HTTP route (extracted and exercised in the style of mission-status-endpoint.test.ts): 404 no mission, 404 unknown task, 409 not-failed, 200 re-queue + broadcast

Red/green: all 8 new tests fail on the base commit (retry does not exist) and pass with this change.

Locally run:

  • npx vitest run src/__tests__/task-retry.test.ts src/__tests__/mission-recovery.test.ts src/__tests__/mission-status-endpoint.test.ts src/__tests__/mission-controls.test.ts src/__tests__/index.test.ts — 76 passed, 0 failed (includes the existing wedged-dispatch stall test in index.test.ts)
  • npx tsc --noEmit — clean
  • npx eslint on the touched files — identical problem count to the base tree (111, all pre-existing); no new findings

@jimmyallinai

Copy link
Copy Markdown
Author

Hi — checking in on this PR, it has been open for a week. I re-verified just now: it is still mergeable with no conflicts, and this repo has no CI configured so there are no failing checks on my side. Happy to adjust the retry limit, the retry endpoint, or the test coverage if you would like anything changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stalled mission is unrecoverable when a required task fails — no retry path

1 participant