Repository navigation
✨ inference translator: opt-in timeouts and bounded 429/5xx retry (default off) - #19156
Conversation
…fault off) Adds governor.inference_resilience (and HIVE_INFERENCE_* env): connect, response-header and idle-stream timeouts, and a bounded retry with exponential backoff, jitter and Retry-After for 429 and 5xx that only runs while no response byte has reached the agent. Metrics and log lines name the route and status only. Every default is off; key resolution and the single 401 retry are unchanged. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. Evidence bundle `#1915690d372f` sha256:5fc7e85476804d8ca0a54a106a0adc6053f09aaacbbfc291cc5efd83a64c7806 — download from the dashboard or `hivectl review evidence #19156`.merged by Hive trusted-author auto-merge: author |
…rence.go Mechanical move of routeWithEntitledModel and entitledModelListed into inference_model_route.go so github_proxy_inference.go stays under the 800-line file-size ratchet. No behavior, name or signature changes. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
TestSleepRatchet counts time.Sleep calls in tests and the PR pushed the total from 195 to 197. Use the existing waitOrDone helper, which waits on the request context or the duration, so behavior is unchanged. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Mutation testinggremlins found 22 surviving mutant(s) in changed files: the code ran under tests, but changing it did not fail any test.
10 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the
ℹ️ 5 mutant(s) on added lines were SKIPPED because gremlins v0.6.0 only honours the first
Verdict: |
Fixes #19057
Summary
The inference translator has no timeout and no retry for rate limits or server errors. This adds opt-in settings for both. With the settings left at their defaults nothing changes.
What changed
governor.inference_resilience(src/crux/config/inference_resilience.go) with named constants for every default and matchingHIVE_INFERENCE_*environment variables. A valid environment value wins over yaml; an invalid one is ignored and named in a startup warning. The field is an omitted pointer by default, so yaml and json output and the config golden are unchanged.Retry-Afterhonored (a value above the max delay skips the retry), default 2 retries with a hard maximum of 5. The retry happens before the handler sees the response, so it can never happen after a byte has reached the agent. The request body replays from the in-memory translated bytes.crux.proxy.inference.retriesandcrux.proxy.inference.timeouts.agent-configuration.mdandai-gateway-evaluation.md, seven settings-parity allowlist entries and a changelog fragment.Tests
Table tests with an httptest upstream cover: defaults unchanged, a stalled upstream cut at the header timeout, an idle stream cut while a slow active stream is not, a 429 with
Retry-Afterretried within the cap, a 5xx retried within the cap, no retry after any byte reached the agent, the cap respected, jitter bounds, and the 401 path untouched, on both the translator and MITM paths, plus a config test.Risks