Skip to content

Fix tool-calling resilience (retry + circuit breaker)#181

Open
imnishantmehra wants to merge 1 commit intolemony-ai:mainfrom
imnishantmehra:fix/tool-calls-use-retry-wrapper
Open

Fix tool-calling resilience (retry + circuit breaker)#181
imnishantmehra wants to merge 1 commit intolemony-ai:mainfrom
imnishantmehra:fix/tool-calls-use-retry-wrapper

Conversation

@imnishantmehra
Copy link
Copy Markdown

Description

Tool-calling flows were bypassing BaseProvider.complete() by calling complete_with_tools() directly in multiple internal paths, which meant tool calls didn’t benefit from the standard retry + circuit breaker + retry-metrics wrapper. This PR routes tool execution through the same resilience layer as normal completions, while remaining compatible with existing provider stubs/mocks used in tests.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Performance improvement

What Changed

  • BaseProvider:
    • Auto-detects tool support when a provider implements complete_with_tools(...).
    • Implements _complete_with_tools_impl(...) to dispatch to complete_with_tools(...) (normalizes messages/system prompt) so tool calls run through _execute_with_retry(...).
    • Passes tool_choice through as-is (supports object/dict-style tool forcing).
  • Internal call sites:
    • Updated tool-call paths in cascadeflow/agent.py, cascadeflow/core/cascade.py, and cascadeflow/streaming/tools.py to go through the retry/CB path when the provider is a real BaseProvider.
    • Keeps compatibility with non-BaseProvider provider stubs by falling back to complete_with_tools(...) where appropriate (test/mocks).
  • Provider config hardening:
    • VLLMProvider normalizes host-only base URLs to include /v1 (prevents misconfig and fixes failing tests).
  • Tests:
    • Added deterministic unit test proving tool calls are retried: tests/test_provider_tool_retry.py.
    • Made a couple provider env tests deterministic via clear=True where needed.

Testing

  • Added new tests
  • pytest tests/test_provider_tool_retry.py -v
  • pytest tests/ -v -m "not integration and not requires_api and not requires_ollama and not requires_vllm"
  • Ran formatting script

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant