llm_judge: support non-Anthropic judge routes (LiteLLM / OpenAI-compatible) #136

Summary

llm_judge can only reach Anthropic models. Both judge transports are Anthropic-Messages shaped, LiteLLMRoute is explicitly refused, and resolve_evaluation_route pins evaluation back to a Claude backend even when the agent itself runs through a gateway. There is no configuration under which a non-Claude model can grade.

Current behaviour (main @ 0.11.1)

• criteria/llm_judge.py — _invoke_tool_channel matches BedrockRoute / DirectRoute / LiteLLMRoute; the LiteLLM arm returns "llm_judge: evaluation route must be Bedrock/Direct, got LiteLLM".
• models/routing.py — resolve_evaluation_route: "agent_route is LiteLLMRoute → pin evaluation to a constant Claude backend."
• evaluation/judge_bedrock.py — posts "anthropic_version": "bedrock-2023-05-31" with Anthropic-shaped tools / tool_choice, and the reply is parsed by extract_verdict_from_anthropic_response. So even a non-Claude model hosted on Bedrock cannot be used: to_bedrock_model only vendor-qualifies claude-* ids, and the request body wouldn't be understood regardless.
• agent_judge is documented as "built-in kinds only — agent_judge runs a Claude Code sub-agent."

Why this matters

1. Judge-model choice cannot be evaluated in-product. Any comparison has to be run outside the framework, so the results can't be reproduced by coder-eval itself or wired into CI.
2. Self-preference. The agents under test are frequently Claude, and the grader is always Claude. There is currently no way to run a cross-vendor grader to check whether a model family is scoring its own output favourably — a structural gap in evaluation methodology, independent of cost.
3. Cost. Judging is a small share of total run cost, but the per-token ratio between vendors is large enough to matter as judged volume grows or rubrics get richer.

We ran an offline comparison by replaying recorded judge prompts against a non-Claude model, calling the provider directly outside the framework. It reproduced the incumbent grader's pass/fail verdicts on 84 of 85 criteria. Whether that particular model is the right choice is beside the point — the framework currently makes the question unanswerable without bypassing it.

Proposal

Add an OpenAI-compatible judge transport and allow it to be selected:

1. evaluation/judge_openai.py, alongside judge_bedrock.py / judge_anthropic.py — POST /chat/completions with a forced function tool, same retry/backoff contract as the Bedrock invoker.
2. An OpenAI translation of SUBMIT_VERDICT_ANTHROPIC_TOOL (input_schema → function.parameters).
3. A verdict extractor for choices[].message.tool_calls[], mirroring extract_verdict_from_anthropic_response.
4. A real case LiteLLMRoute() arm in _invoke_tool_channel, and relaxing resolve_evaluation_route so a gateway route can serve the judge instead of being pinned to Claude.
5. Usage mapping (prompt_tokens / completion_tokens → TokenUsage).

Two findings from prototyping all of the above, worth encoding in whatever lands:

• The OpenAI surface rejects reasoning_effort when function tools are present on /chat/completions ("Function tools with reasoning_effort are not supported"); only /responses accepts both. "none" is special-cased and does work on /chat/completions.
• Some reasoning models reject temperature values other than their default ("Unsupported value: 'temperature' does not support 0"), so the judge's temperature=0 cannot be assumed portable across providers.

Related, smaller

(a) claude-sonnet-5 cannot be used as a judge at all. judge_bedrock.py puts temperature in the request body unconditionally, and sonnet-5 responds 400 "temperature is deprecated for this model". Setting model: anthropic.claude-sonnet-5 on an llm_judge criterion therefore fails at dispatch on every criterion. Making the field conditional looks like a one-line fix.

(b) There is no way to set the judge model globally. -D override roots are ('agent', 'run_limits', 'sandbox'), and neither ExperimentDefaults nor ExperimentVariant carries a success_criteria field. Changing the judge model across a suite currently means editing every task YAML that declares an llm_judge, or monkey-patching the default on LLMJudgeCriterion.model (which additionally requires TaskDefinition.model_rebuild(force=True), since the criterion union is compiled at import — patching only LLMJudgeCriterion silently has no effect through the task loader). A success_criteria override root, or a judge section in ExperimentDefaults, would turn judge-model A/B into a config change.

Happy to contribute the transport — we have working implementations of (1)–(3) and (5) from the prototype.