Skip to content

πŸ› inference: refresh rotated gateway keys and refuse ignored api_key_file paths - #18540

Merged
kubestellar-prow[bot] merged 2 commits into
v5from
fix/18532-litellm-key-resolution
Nov 16, 2026
Merged

kubestellar-prow[bot] merged 2 commits into
v5from
fix/18532-litellm-key-resolution

Conversation

@clubanderson

Copy link
Copy Markdown
Member

What

Two LiteLLM/gateway key bugs, fixed together because they share the key-resolution path.

#18532: the proxy cached the key at route-set time

  • Routes now carry a proxy.InferenceKeyResolver (src/crux/proxy/inference_key_resolver.go). It re-stats the key file at most every inferenceKeyRecheckTTL (3s). It re-resolves when the symlink target, mtime or size changes (a Secret update swaps ..data), and does a full re-resolve every inferenceKeyFullRefreshInterval (30s) so a new source is noticed too.
  • Both entry points (the translator on :18444 and the MITM reroute) use the current key on each request, on a copy of the route. The shared route is never mutated.
  • On an upstream 401, the proxy re-resolves once. If the key changed, it retries the request once with the new key. An auth failure is counted only when the retry also fails. Design choice: a re-resolve that returns the same key is not retried, because resending the same key would just 401 again and double the failed-auth traffic.
  • The inference backend auth failure WARN gains key_source (file:<path>, env:<NAME>, static, none) and key_changed_since_start. Key material is never logged.
  • Wired for litellm and named-gateway routes in cmd/hive. watsonx keeps its static minted IAM token.

#18533: an api_key_file outside the allowed dirs was silently ignored

  • Save path (dashboard/API): litellm, gateways and inference-auth reject an outside path with a 400. The message lists the allowed dirs and tells the operator to mount the Secret there.
  • Boot: warns, does not fail. Existing configs with such a path have been running on their fallback key, and hard-failing would take a working spoke down over a setting that was already ignored. Each offending field gets a WARNING: line.
  • Runtime: one WARN per field and path (configured api_key_file is outside the allowed secret dirs and was skipped), with skipped_path and effective_key_source.
  • API/UI: the LiteLLM section of GET /api/config/governor and each gateway row now include keySource, keySourceDescription (e.g. key from /secrets/litellm_api_key (configured path /secrets-cb/... not allowed)) and apiKeyFileAllowed. The LiteLLM section also includes allowedKeyFileDirs. The LiteLLM tab shows a red note when the configured path is ignored.
  • The audit N8 gate (SecretFilePathAllowed) and its allowed dirs are unchanged.

Tests

  • A key file swapped at runtime (a ..data symlink swap) is picked up after the TTL, on both entry points, with no restart.
  • A 401 leads to a re-resolve and a successful retry, with no failure counted. A second 401 counts exactly one failure, and the WARN has key_source and key_changed_since_start=true. An unchanged key is not retried.
  • Boot validation warns and does not fail on an outside path. The dashboard save paths reject it with the remedy.
  • The runtime warning fires exactly once.
  • The effective source appears in the litellm and gateway API responses.

Docs: src/docs/agent-configuration.md and docs/inference-backends.md. Changelog: changelog.d/fixed-18532-18533-litellm-key-resolution.md.

Fixes #18532
Fixes #18533

cc @joerunde

πŸ€– Generated with Claude Code

…file paths

The inference proxy resolved the LiteLLM/gateway key once, when an agent's
route was set, so a rotated Secret kept 401ing until restart (#18532). Routes
now carry an InferenceKeyResolver that re-stats the key file every few seconds
(symlink target, mtime, size: a Secret update swaps ..data), re-resolves on
change or every 30s, and on an upstream 401 re-resolves once and retries once
with a changed key before counting an auth failure. The auth-failure WARN
gains key_source and key_changed_since_start; no key material is logged.

An api_key_file outside the allowed secret dirs was silently skipped (#18533).
Dashboard/API saves now reject it with a message listing the allowed dirs and
the mount remedy; boot warns (existing configs keep booting on their fallback
key); runtime logs one WARN naming the skipped path and the effective source;
the governor config API and gateway rows expose keySource,
keySourceDescription and apiKeyFileAllowed. The audit N8 gate is unchanged.

Fixes #18532
Fixes #18533

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18536 ✨ feature: choose remote repository write identity same files src/cmd/hive/main.go
src/docs/agent-configuration.md
src/pkg/config/config.go
src/pkg/config/validate.go
src/pkg/dashboard/static/index.html
#18505 ✨ Discover agents declared in a managed repo's .hive/agents/ behind owner approval same files src/cmd/hive/main.go
src/crux/config/agent_runtime.go
src/docs/agent-configuration.md
src/pkg/dashboard/static-src/js/config-tabs.js
src/pkg/dashboard/static/index.html
#18507 ✨ governor: release-feed watcher kicks agents when a tracked upstream dependency releases same files src/cmd/hive/main.go
src/docs/agent-configuration.md
src/pkg/config/config.go
src/pkg/config/validate.go
#18528 ✨ meetings: agent-backend model mode with a warm tool-less Claude Code CLI and Settings picker (part 1 of #18517) same files src/pkg/dashboard/static/index.html
#18542 πŸ› hive never applies hold itself: gates use ready-for-review / decision-needed same files src/cmd/hive/main.go
src/pkg/config/config.go

Keeps agent_runtime.go under the crux file-size ratchet baseline.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing

gremlins found 1 surviving mutant(s) in changed files: the code ran under tests, but changing it did not fail any test.

File Line Mutation
src/crux/proxy/inference_key_resolver.go 134 CONDITIONALS_BOUNDARY

3 mutant(s) in changed files are not reached by any test (NOT COVERED). A maintainer may apply the mutation/acknowledged label to accept this.

File Line Mutation
src/crux/config/secret_file_reporting.go 68 CONDITIONALS_NEGATION
src/crux/proxy/inference_key_resolver.go 25 ARITHMETIC_BASE
src/crux/proxy/inference_key_resolver.go 29 ARITHMETIC_BASE

Verdict: lived (#17349).

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow
kubestellar-prow Bot merged commit 22d3099 into v5 Nov 16, 2026
95 of 97 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/18532-litellm-key-resolution branch November 16, 2026 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

1 participant