Skip to content

[Feature]: Error Control Config + Circuit Breaker for Agent Execution #46

Description

@vinodvx

Describe the feature

Give users declarative control over how the agent handles failures —fallback models, per-error recovery strategies, and automatic circuitbreaking for repeatedly failing tools and MCP servers. Currently the SDK decides all error behaviour internally with no user control.

Use case

Production agents encounter a range of failures — LLM rate limits, context window exceeded, tool timeouts, MCP server unavailability, unknown tools, malformed tool args, runaway iteration loops. Today the SDK either retries internally or fails the run with no user control over the strategy. This leads to:

  • Token-burning doom loops where a failing tool is retried until context is exhausted
  • No fallback to cheaper/faster models on rate limits
  • No way to skip a non-critical failing tool and continue the run
  • No circuit breaker to stop calling a repeatedly failing MCP server
  • No way to customize behaviour without forking the SDK

Proposed solution

Design principle: error decisions are runtime behaviour, not static config — the right action often depends on live context (attempt count, which tool, elapsed time) that a fixed config value can't express. So error strategies are implemented as hooks, with built-in hook implementations provided for common strategies so most users never write their own.

1. ErrorHooks interface (phase 1 — stateless)

type ErrorHooks interface {
    OnRateLimit(ctx context.Context, info RateLimitInfo) ErrorAction
    OnContextExceeded(ctx context.Context, info ContextExceededInfo) ErrorAction
    OnToolFailure(ctx context.Context, info ToolFailureInfo) ErrorAction
    OnToolNotFound(ctx context.Context, info ToolNotFoundInfo) ErrorAction
    OnModelProviderError(ctx context.Context, info ModelProviderErrorInfo) ErrorAction
    OnMaxIterationsExceeded(ctx context.Context, info MaxIterationsExceededInfo) ErrorAction
    OnInvalidToolArgs(ctx context.Context, info InvalidToolArgsInfo) ErrorAction
}

Each hook returns a shared ErrorAction enum: Retry | Abort | FallbackModel | FallbackTool | InjectPrompt | SkipAndContinue | EscalateHuman. Not every action is valid for every hook — invalid returns are logged and fall back to a safe per-hook default rather than crashing the run.

A DefaultErrorHooks implementation ships with the SDK, preserving current behaviour exactly — zero-config users see no change.

2. Built-in strategy helpers (convenience layer over hooks, not config enums)

sdk.NewAgent(
    sdk.WithErrorHooks(sdk.DefaultErrorHooks{
        OnRateLimit:      sdk.RetryWithBackoff(30 * time.Second),
        OnContextTooLong: sdk.CompactAndRetry,
        OnToolFailed:     sdk.SkipToolAndContinue,
        OnMCPUnavailable: sdk.SkipMCPAndContinue,
        OnLLMFailed:      sdk.FallbackToModel("claude-haiku-4-5"),
    }),
)

Users needing custom logic (e.g. "retry twice, then fallback, but only for this tool") implement ErrorHooks directly instead of using a helper — same interface, no forking required either way.

Strategy helper Behaviour
RetryWithBackoff(d) Retry after duration d
CompactAndRetry Summarize history, retry with smaller context
SkipToolAndContinue Skip failed tool, continue run
SkipMCPAndContinue Skip MCP server, continue with remaining tools
FallbackToModel(name) Switch to fallback LLM
FailRun Stop run, return error to user

3. Circuit breaker — per tool and MCP server

Threshold is static config (checked internally by the SDK's per-run telemetry); the trip action is a hook, same ErrorAction mechanism as above.

sdk.WithErrorControl(sdk.ErrorControlConfig{
    CircuitBreaker: &sdk.CircuitBreakerConfig{
        MaxConsecutiveSameArgs: 3,     // trip after 3 consecutive same-args failures
        PatternWindowSize:      4,     // detect alternating A-B-A-B traps
        ResetAfter:             2 * time.Minute,
    },
})

sdk.WithErrorHooks(sdk.DefaultErrorHooks{
    OnCircuitBreakerTripped: sdk.InjectWarning, // or SkipTool, FailRun, EscalateHuman
})

Circuit breaker tracks consecutive-failure and alternating-pattern signals per tool/MCP server within a run's telemetry. On trip, the SDK short-circuits further calls to that tool and hands off to OnCircuitBreakerTripped.

OnTripped strategy Behaviour
InjectWarning Tell LLM tool is unavailable, continue
SkipTool Remove tool from available tools for this run
FailRun Stop run immediately
EscalateHuman Hand off to human-in-the-loop triage

4. Explicitly out of scope for this issue

MaxToolRetries / MaxLLMRetries / execution timeouts (Temporal-style attempt and wall-clock bounding) are static execution config, not error hooks — tracked separately, already aligned with in-process/Temporal activity config.

Alternatives considered

Strategies as config enum values instead of hooks — considered, rejected. A closed config-driven strategy set can't express context-dependent decisions (e.g. differing behaviour by attempt count) without the SDK growing an ever-larger built-in enum. Hooks with optional built-in helper implementations give the same ergonomic one-liner API for common cases while keeping full custom logic available.

Temporal retry policy only — Temporal handles activity retries but does not cover product-level decisions like fallback models, context compaction, or tool skipping. Not sufficient on its own.

User implements everything in raw hooks, no built-in strategies — possible but requires every user to reimplement the same common patterns. Built-in strategy helpers avoid boilerplate while the underlying hook interface still allows full customization.

Global retry config — single retry count for all errors. Rejected — different errors need different strategies (rate limit needs backoff, context-too-long needs compaction, tool failure needs skip). One-size-fits-all is wrong.

Additional context

  • Circuit breaker pattern: https://martinfowler.com/bliki/CircuitBreaker.html
  • Context compaction reference: Anthropic Claude long context best practices
  • Token-burning doom loop is a known production failure mode in agent systems
  • Builds on existing lifecycle hooks infrastructure (WithHooks) — ErrorHooks is a separate, parallel interface (WithErrorHooks) since lifecycle hooks observe/mutate on the happy path while error hooks make load-bearing control-flow decisions on failure
  • Phased delivery: Phase 1 = stateless error hooks (OnRateLimit, OnContextExceeded, OnToolFailure, OnToolNotFound, OnModelProviderError, OnMaxIterationsExceeded, OnInvalidToolArgs). Phase 2 = circuit breaker (config + OnCircuitBreakerTripped hook)

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions