Skip to content

[selfreport] inference_backend_auth_failure: inference backend auth failure #19250

Closed as not planned

Description

@hivecommons-hive

A hive hit an unusual runtime error (class inference_backend_auth_failure) and is reporting it so the same defect across the fleet lands in one issue.

Evidence

Field Value
hive hosted-llm-d-llm-d-worklo-wva1
hive name —
spoke hosted-llm-d-llm-d-worklo-wva1
anonymous instance d9d430158d782090
owner(s) @clubanderson, @lionelvillard
hive version 5.309.0-5-gbb85cd0f2
hive commit bb85cd0
cluster vllm-d
mode surge
component backend-auth
scope inference backend auth failure
message inference backend auth failure
count + window 6 in 10m0s
first seen 2026-11-19T11:36:31Z
reported at 2026-11-19T11:37:06Z
repos —
owners —
repo —
ref —
self-recovered no

Affected hives (1):

  • hosted-llm-d-llm-d-worklo-wva1 (instance d9d430158d782090) — open — owner(s): @clubanderson, @lionelvillard — seen 1, last 2026-11-19T11:37:06Z
    seen_count: 1
Diagnostics (scrubbed)
WARN agent=reviewer backend=litellm endpoint=https://ete-litellm.ai-models.vpc.res.ibm.com status=401 key_source=file:/data/secrets/litellm_api_key key_changed_since_start=false

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 19, 2026
  2. hivecommons-hive commented on Nov 19, 2026

    @hivecommons-hive
    ContributorAuthor

    🔓 Claim by @Karibhou released (lease revoked).

    Claim history
    • 2026-11-19 12:22 UTC — claimed by Karibhou (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-19 12:26 UTC — released by Karibhou (contributor; lease revoked)
  3. Karibhou commented on Nov 19, 2026

    @Karibhou
    Contributor

    LiteLLM returned HTTP 401 for a credential resolved from /data/secrets/litellm_api_key. key_changed_since_start=false means the resolved key matched the first key this process saw; it does not show that Hive ignored a replacement.

    The reported build (bb85cd0) and v5 both contain merged PR #18540 (re-read the key after a 401 and retry once if it changed) and merged PR #18557 (classify repeated upstream 401s as an operator credential issue). This report does not show a recurrence of the key-caching defect.

    The spoke owner needs to verify that the configured key is valid for the reported LiteLLM endpoint, replace it at the effective source if needed, and confirm a reviewer inference request succeeds. Please do not share the key here.

    -- hive: backend=omp model=openai-codex/gpt-6.2-sol

  4. added
    stage/045-needs-decisionIssue parked on a person: needs-human or a maintainer decision is owed before the hive picks it up
    hive/already-doneHive contributor found this issue already resolved; remove if work remains
    and removed on Nov 19, 2026
  5. hivecommons-hive commented on Nov 19, 2026

    @hivecommons-hive
    ContributorAuthor

    What to reply

    • /hive approve: go with the recommendation
    • /hive decision <your instructions>: go ahead, following your instructions
    • /hive help: show these again

    If what this issue waited on has been done (a merged fix, or an action taken outside this repository), a maintainer can reply /fixed to close it as completed; Hive clears its park labels.

    Only a maintainer with write access can un-park; everything else is ignored. Full command reference: https://github.com/hivecommons/hive/blob/v5/src/docs/maintainer-commands.md.

  6. clubanderson commented on Nov 19, 2026

    @clubanderson
    Member

    Closing: this is a spoke credential problem, not a Hive defect. The LiteLLM key configured on that spoke is being rejected (401); the running build already re-reads the key file after a 401 (#18540) and reports repeated 401s as an operator credential issue (#18557). The spoke owner needs to install a valid key; no Hive change is needed.

  7. removed
    stage/045-needs-decisionIssue parked on a person: needs-human or a maintainer decision is owed before the hive picks it up
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.hive/already-doneHive contributor found this issue already resolved; remove if work remains

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions