claude-code agent fails on every task: model/protocol mismatch + /v1 double-path against litellm proxy

[Julia-Lex]

Summary

The claude-code agent fails immediately on every task with:

[FAILED] There's an issue with the selected model (dmx-claude-opus-4-7).
It may not exist or you may not have access to it.

The agent drives the real claude CLI, which speaks the native Anthropic Messages API. With the default config it is pointed at an OpenAI-compatible litellm proxy and given an OpenAI-style model name, so the CLI's requests fail before any browser work starts. The misleading "model may not exist" text actually hides two distinct, stacked failures.

Reproduction

bubench run --agent claude-code --data LexBench-Browser --mode single

Executing Claude Code for task 5 (model=dmx-claude-opus-4-7, max_turns=50)
[Claude Code] Done: 1 turns
[FAILED] There's an issue with the selected model (dmx-claude-opus-4-7). It may not exist or you may not have access to it.
[SUMMARY] Total: 1 | Success: 0 | Failed: 1

Environment: claude CLI 2.1.179; ANTHROPIC_BASE_URL=https://litellm.local.lexmount.net/v1; claude-code agent has no active_model, so model resolution falls back to default.model: claude, whose entry is model_type: OPENAI, model_id: dmx-claude-opus-4-7, base_url: $OPENAI_BASE_URL.

Root cause (two stacked problems)

1. ANTHROPIC_BASE_URL must not end in /v1 (this is the actual 404 in the bench run)

The claude CLI appends /v1/messages to ANTHROPIC_BASE_URL itself. When the base already ends in /v1 it requests …/v1/v1/messages and gets a 404, which the CLI surfaces as "model may not exist":

{"is_error": true, "api_error_status": 404, "error": "model_not_found",
 "result": "There's an issue with the selected model (dmx-claude-opus-4-7)..."}

This is the same env var with opposite requirements: OpenAI-protocol agents (browser-use, codex, ...) need the trailing /v1; the native Anthropic CLI must not have it. A direct curl .../v1/messages with the same key/model returns 200, confirming the model and credentials are fine and the failure is purely the doubled path.

browseruse_bench/agents/claude_code.py passes base_url through verbatim:

if base_url:
    env["ANTHROPIC_BASE_URL"] = base_url

2. Every Claude model on the litellm proxy is backed by custom_openai, which rejects the CLI's mandatory params

Once /v1 is stripped, the 404 becomes a 400. The proxy serves all Claude models via an OpenAI-format upstream, and the claude CLI always sends thinking + context_management, which that upstream rejects:

API Error: 400 litellm.UnsupportedParamsError: custom_openai does not support
parameters: ['thinking', 'context_management'], for model=claude-opus-4-7.
To drop these, set litellm.drop_params=True ...

Verified across every Claude model exposed by the proxy — all 400 with the same error:

model	claude CLI result
dmx-claude-opus-4-8	400 UnsupportedParams
dmx-claude-opus-4-8-thinking	400 UnsupportedParams
dmx-claude-sonnet-4-6	400 UnsupportedParams
dmx-claude-opus-4-6	400 UnsupportedParams
openrouter/claude-opus-4.8	400 UnsupportedParams

(A minimal curl to /messages succeeds because it omits these params; the CLI cannot omit them.)

Why this happens by default

config.yaml's default.model is claude, whose entry is an OpenAI-proxy model (model_type: OPENAI, dmx-claude-opus-4-7, base_url: $OPENAI_BASE_URL). Since the claude-code agent block has no active_model, it inherits this OpenAI-protocol entry — but the agent runs the native Anthropic CLI. The model config and the agent's protocol are mismatched.

Proposed fixes

Proxy side (required for #2 — cannot be fixed in this repo alone):

• Enable litellm_settings: drop_params: true on the litellm proxy, or
• Add a litellm model route backed by a native anthropic provider for use by claude-code.

Client side (this repo, addresses #1 and the mismatch):

• Strip a trailing /v1 from base_url before setting ANTHROPIC_BASE_URL in claude_code.py (the CLI appends the API path itself).
• Give the claude-code agent a dedicated active_model that points at a native-Anthropic route (real Anthropic model id + Anthropic-compatible base without /v1), instead of inheriting the OpenAI-proxy claude entry.
• Optionally fail fast with a clearer message when a model_type: OPENAI entry is handed to claude-code.

Acceptance

bubench run --agent claude-code --data LexBench-Browser --mode single reaches a real model call and completes a turn (per the smoke-test guidance in docs_4_codeagent/error-handling-testing.md), instead of erroring on the first turn.