Skip to content

Rate-limit mid-task burns retries → hard-skip poisons dependency graph #623

Description

@IBondarenko-iwg

What happened

Mid-task, the Claude usage limit was hit. The agent was cut off before calling task_set_status(done), so the completion check failed. The runner retried immediately, hit the same not-yet-reset limit (in=0 out=0), and exhausted its 2-attempt budget — marking the task skipped with skip_reason = max-retries.

max-retries isn't an intentional-skip reason, so it doesn't satisfy dependencies. Every downstream task stayed permanently blocked. The scheduler kept picking the lowest-priority runnable task — which, with the subtree starved, sat in a later group. Result looks like "out-of-order execution," with no surfaced error explaining why the low-priority foundation never runs.

Instance: t_1sQQIhaP (priority 3) died on the limit and blocked grp-1 p4/5/6 and the full grp-2 chain (p11→…→p18). The workflow process then stopped, so nothing retried it.

What you expected

A transient provider condition (usage limit / rate limit / 429) is not the task's fault and should not:

  • consume the per-task retry budget, or
  • result in a permanent max-retries skip that silently poisons dependents.

Expected instead: detect the usage-limit/rate-limit error class, and either park the task (todo/blocked) and back off until the advertised reset time, or pause the whole workflow with a clear "usage limit reached, resumes at HH:MM" status. The task should resume and complete once the limit clears, and dependents should remain simply blocked (not starved) in the meantime.

Steps to reproduce

  1. Start a start-from-prompt (or any multi-task) workflow.
  2. While it is executing, exhaust the Claude usage limit (or simulate a rate-limit/429 from the provider mid-task).
  3. Observe the task get cut off before task_set_status(done), retry twice against the still-active limit, and get marked skipped with skip_reason = max-retries.
  4. Observe that all tasks depending on it stay todo while the scheduler jumps to higher-priority tasks in other groups, with no surfaced error.

Environment

OS: Windows 11
dotbot: 4.0.1
Runtime: PowerShell 7

Severity

high

Logs / screenshots

Runtime log (.control/logs/dotbot-2026-07-09.jsonl), task t_1sQQIhaP:

08:15:47  session_start  model=claude-opus-4-7
08:15:56  Read pyproject.toml
08:15:57  ERROR "You've hit your limit · resets 3:30pm"
08:15:57  result turns=4 ... time=10.4s        (cut off mid-task)
08:16:00  Completion check failed (attempt 1): still status in-progress
08:16:06  session_start (retry)
08:16:06  ERROR "You've hit your limit · resets 3:30pm"   (turns=1 in=0 out=0, 1.2s)
08:16:08  Completion check failed (attempt 2): still status in-progress
          → skip_reason=max-retries, "Retry budget exhausted after 2 attempt(s)"
08:16:10  proc-b63a53 failed_at → workflow process stopped

Relevant code:

  • Dependency-satisfaction set excludes non-intentional skips (so max-retries blocks dependents): src/runtime/Modules/Dotbot.Process/Dotbot.Process.psm1:594 and the runnable gate :718.
  • Scheduler picks lowest-priority runnable task, so a starved subtree looks like out-of-order execution: :744.

Proposed fix:

  1. Classify usage-limit / rate-limit / 429 as a distinct, transient error — do not count it against the task's retry budget.
  2. On that error, back off until the advertised reset time (parse "resets HH:MM") or park the task as blocked and pause the workflow with a visible "usage limit — resumes at HH:MM" status, rather than retrying immediately against a limit that won't clear for hours.
  3. Never let a max-retries skip silently satisfy/poison dependents without surfacing a visible error or needs-input on the blocked subtree.

Metadata

Metadata

Labels

Projects

Status
In Review

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions