Skip to content

feat(core): provider failover chain with a durable route record (#209, 1/3) - #404

Merged
ajianaz merged 1 commit into
developfrom
feat/provider-failover-core
Oct 16, 2026
Merged

ajianaz merged 1 commit into
developfrom
feat/provider-failover-core

Conversation

@ajianaz

@ajianaz ajianaz commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

What

Part 1 of 3 of #209: the failover mechanism in tole-core. Part 2 adds the real-provider integration test, part 3 the env/config wiring and docs.

Why

An unattended mission parks resumable when its provider has an outage, but makes no progress until a human resumes. A reserve provider turns most outages into a non-event, and durable sessions make the switch auditable.

Changes

  • failover.rs (new): FailoverProvider wraps an ordered chain (Link { label, provider }, primary first). On an "unavailable" error (connection failure, timeout, dropped stream, HTTP 5xx or 429) the same step is retried on the next provider; request errors (401/403/404/400, malformed responses) fail loud and never fail over. The answering provider stays active (no flapping); the primary is retried after retry_primary_after (default 5 min). When every provider fails, the last error leads (the turn loop's transient-retry classification reads it) followed by the earlier ones.
  • provider.rs: new default method last_route() (None for a single provider: no behaviour change).
  • turn.rs: the route is stored under the namespaced tole_route key of the step's usage row, next to tole_wire.
  • usage_report.rs: a row holding only tole's own tole_* keys is not a provider step (an empty object still is, the pre-token efficiency: lazy tool registration + tool-result trimming policy #211 contract), so route rows never change step accounting.
  • CHANGELOG.

Testing

  • 6 unit tests: error classification, failover and route report, request errors never reach the reserve, sticky reserve and primary retry after the cooldown, all-fail message order, single-provider transparency; plus a usage-report test for route-only rows.
  • fmt, clippy -D warnings, cargo test --workspace, cargo check -p tole-core --no-default-features pass; cora review --staged clean.
  • Real OpenAiProvider error text and a full turn are covered by the integration test in part 2 (passes locally).

Refs #209

@github-actions

Copy link
Copy Markdown

🔍 Cora AI Code Review

✅ No issues found. Code looks good!


Review powered by cora-code · BYOK · MIT

@ajianaz
ajianaz merged commit 90fde00 into develop Oct 16, 2026
17 checks passed
@ajianaz
ajianaz deleted the feat/provider-failover-core branch October 16, 2026 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant