Skip to content

fix(libsy-llm-client): bound upstream requests with a configurable timeout #277

Description

@gburachas

Problem

The Rust server's HTTP client is built with no request timeout:

reqwest::Client::builder().build()

reqwest applies no overall timeout unless told to, and there is no configuration key to set one.
A request to a hung or unresponsive upstream therefore waits indefinitely — the connection is
held, the caller never gets an answer, and an operator has no way to bound it.

The retry budget does not help: retries fire on failures, and a request that never returns never
fails.

The capability exists elsewhere in the workspace — switchyard-components (the Python-facing path)
takes timeout_secs: Option<f64>, validates it, and passes it to the same reqwest builder. The
Rust server path simply never wires it up. The crate's own tests set .timeout(...) directly, so
the mechanism works; only the configuration is missing.

Proposal

  • timeout_secs: Option<f64> on the client's backend config, exposed as timeout_secs under
    [llm_clients.<name>];
  • applied per attempt via RequestBuilder::timeout, so each retry gets its own budget;
  • validated finite and positive at load time;
  • unset by default, preserving today's behaviour exactly.

Deliberately reuses the seconds-as-f64 shape and the validation wording already used by
switchyard-components, rather than introducing a second convention.

Documented caveat

Because the bound is per attempt, a call that exhausts its retry budget can take up to
(max_retries + 1) x timeout_secs. Stated in the schema reference next to the key.

Scope

6 files, +83/−2. Core change is ~15 lines; the rest is config plumbing, two struct literals, one
test, and the docs row.

Validation

Gate Result
cargo test --workspace 641 passed (baseline 640 + 1)
cargo clippy --workspace --all-targets -- -D warnings clean
cargo fmt --check clean
uv run pytest tests/ unchanged

The new test drives a mock upstream that delays 30s under a 50 ms timeout with retries disabled, and
asserts the call fails as Timeout rather than hanging.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions