Skip to content

✨ inference translator: opt-in timeouts and bounded 429/5xx retry (default off) - #19156

Merged
hivecommons-hive[bot] merged 7 commits into
v5from
feat/translator-timeouts-retry
Nov 18, 2026
Merged

hivecommons-hive[bot] merged 7 commits into
v5from
feat/translator-timeouts-retry

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #19057

Summary

The inference translator has no timeout and no retry for rate limits or server errors. This adds opt-in settings for both. With the settings left at their defaults nothing changes.

What changed

  • New config governor.inference_resilience (src/crux/config/inference_resilience.go) with named constants for every default and matching HIVE_INFERENCE_* environment variables. A valid environment value wins over yaml; an invalid one is ignored and named in a startup warning. The field is an omitted pointer by default, so yaml and json output and the config golden are unchanged.
  • Timeouts: connect, response header and idle. The idle timeout only runs while a read is blocked on the upstream, so slow but active streams are never cut.
  • Retry: 429 and 5xx except 501, exponential backoff with jitter, Retry-After honored (a value above the max delay skips the retry), default 2 retries with a hard maximum of 5. The retry happens before the handler sees the response, so it can never happen after a byte has reached the agent. The request body replays from the in-memory translated bytes.
  • Logs carry only backend, status, agent, attempt and delay or kind, never headers or content. Metrics: crux.proxy.inference.retries and crux.proxy.inference.timeouts.
  • Applied in both the translator handler and the MITM handler. Key resolution and the existing single 401 retry are untouched, and the new retry runs before it.
  • Docs updated in agent-configuration.md and ai-gateway-evaluation.md, seven settings-parity allowlist entries and a changelog fragment.

Tests

Table tests with an httptest upstream cover: defaults unchanged, a stalled upstream cut at the header timeout, an idle stream cut while a slow active stream is not, a 429 with Retry-After retried within the cap, a 5xx retried within the cap, no retry after any byte reached the agent, the cap respected, jitter bounds, and the 401 path untouched, on both the translator and MITM paths, plus a config test.

Risks

  • Not run locally; CI is the first verdict for compilation and the new tests.
  • The idle-cut timing tests use an 80 ms margin against a 300 ms timeout and could be sensitive to a loaded CI runner.
  • Header timeout classification relies on the net/http error text "awaiting response headers".

…fault off)

Adds governor.inference_resilience (and HIVE_INFERENCE_* env): connect, response-header and idle-stream timeouts, and a bounded retry with exponential backoff, jitter and Retry-After for 429 and 5xx that only runs while no response byte has reached the agent. Metrics and log lines name the route and status only. Every default is off; key resolution and the single 401 retry are unchanged.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 18, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#19063 ✨ PR gate: fail visibly when the Commit statuses grant is missing same files src/cmd/hive/main.go
#19102 🐛 Let the agent that opened a PR reply on its review threads same files src/cmd/hive/main.go

@hivecommons-hive

hivecommons-hive Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • test (rest 2/4)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Evidence bundle `#1915690d372f` sha256:5fc7e85476804d8ca0a54a106a0adc6053f09aaacbbfc291cc5efd83a64c7806 — download from the dashboard or `hivectl review evidence #19156`.

merged by Hive trusted-author auto-merge: author clubanderson holds role owner and GitHub permission admin on repo; CI green at 90d372f.

…rence.go

Mechanical move of routeWithEntitledModel and entitledModelListed into
inference_model_route.go so github_proxy_inference.go stays under the
800-line file-size ratchet. No behavior, name or signature changes.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
TestSleepRatchet counts time.Sleep calls in tests and the PR pushed the
total from 195 to 197. Use the existing waitOrDone helper, which waits on
the request context or the duration, so behavior is unchanged.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing

gremlins found 22 surviving mutant(s) in changed files: the code ran under tests, but changing it did not fail any test.

File Line Mutation
src/crux/config/inference_resilience.go 121 CONDITIONALS_BOUNDARY
src/crux/config/inference_resilience.go 147 CONDITIONALS_BOUNDARY
src/crux/config/inference_resilience.go 153 ARITHMETIC_BASE
src/crux/config/inference_resilience.go 153 CONDITIONALS_BOUNDARY
src/crux/config/inference_resilience.go 171 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 123 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 123 CONDITIONALS_NEGATION
src/crux/proxy/inference_resilience.go 127 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 139 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 139 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 139 CONDITIONALS_NEGATION
src/crux/proxy/inference_resilience.go 213 CONDITIONALS_NEGATION
src/crux/proxy/inference_resilience.go 242 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 242 CONDITIONALS_NEGATION
src/crux/proxy/inference_resilience.go 245 ARITHMETIC_BASE
src/crux/proxy/inference_resilience.go 268 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 282 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 289 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 292 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 314 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 317 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 341 CONDITIONALS_NEGATION

10 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/crux/proxy/inference_resilience.go 40 ARITHMETIC_BASE
src/crux/proxy/inference_resilience.go 149 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 149 CONDITIONALS_NEGATION
src/crux/proxy/inference_resilience.go 151 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 151 CONDITIONALS_NEGATION
src/crux/proxy/inference_resilience.go 151 CONDITIONALS_NEGATION
src/crux/proxy/inference_resilience.go 250 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_resilience.go 250 CONDITIONALS_NEGATION
src/crux/proxy/inference_resilience.go 293 ARITHMETIC_BASE
src/crux/proxy/inference_resilience.go 366 CONDITIONALS_NEGATION

ℹ️ 5 mutant(s) on added lines were SKIPPED because gremlins v0.6.0 only honours the first lines added lines of each default-context hunk, so non-contiguous edits in one hunk are partly missed. This does not affect the verdict.

File Line Mutation
src/crux/proxy/inference_route.go 317 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_route.go 317 CONDITIONALS_BOUNDARY
src/crux/proxy/inference_route.go 317 CONDITIONALS_NEGATION
src/crux/proxy/inference_route.go 317 CONDITIONALS_NEGATION
src/crux/proxy/inference_route.go 318 CONDITIONALS_NEGATION

Verdict: lived (#17349).

@hivecommons-hive
hivecommons-hive Bot merged commit fc5d41e into v5 Nov 18, 2026
87 of 88 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/translator-timeouts-retry branch November 18, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 inference: configurable timeouts and bounded retry for rate limits and server errors in the translator

1 participant