What happened (2026-06-11, 08:57–16:04 UTC, all day)
gpt-5.4 via https://llmapi.hk.lexmount.net/v1 returned responses consisting of one complete AgentOutput JSON followed by a newline and a second JSON object on ~20–30% of calls. The two objects had independently mixed formats (compact+compact, pretty+compact, ...) — i.e. two independent generations concatenated by the gateway/upstream aggregation layer, not model repetition.
browser-use's strict pydantic parse fails on these with:
Invalid JSON: trailing characters at line N column 1
Impact on the 6/11 bench runs (/Users/zhu/lexmount/bench-experiment)
| Run |
Backend |
Contamination |
20260611_165250 |
browser-use-cloud |
414/1373 steps (30%), 71/92 tasks |
20260611_184228 |
browserless |
271/1266 steps (21%), 44/92 tasks; task 149 aborted |
20260614_063435 / 20260614_063552 |
lexmount / browserbase (identical LLM config, 6/14) |
zero occurrences |
Retries (max 6) recovered most steps, so no mass aborts — but ~30% of steps burned budget/wall-clock/tokens producing no action, and browser-use usage tracking skips failed calls, so the 6/11 runs' reported token/cost undercount real spend ~1.4x. The 6/11 backend-comparison numbers (browser-use-cloud 53% vs lexmount 65%) are confounded by the incident and must not be used for backend ranking.
Why this issue exists
At the time we could not attribute the failures: the raw failing responses were discarded by the agent. PR #82 (commit 668598b, merged 2026-07-04) fixed that: failed/unparseable LLM calls now capture the raw response + usage into api_logs (llm_failures per step, llm_failures_unmatched.json for orphans). OpenAI-compatible chat.completions path only (Azure responses-API/Anthropic/Google record raw_response=None).
Next time the signature appears
- Confirm in api_logs:
llm_failures[].raw_response should show one complete JSON object followed by a second one after a newline, and the parse error Invalid JSON: trailing characters.
- This is a gateway/upstream aggregation bug — escalate to the gateway admin (llmapi.hk.lexmount.net) with the raw_response excerpt and the time window. Not a bench bug, not model repetition; do not debug bench code.
- Mark any bench runs overlapping the incident window as contaminated for scoring/comparison purposes (steps and cost are inflated, usage undercounted).
Known remaining gap: aggregate cost accounting in result.json metrics still excludes failed calls, so contaminated runs will still undercount spend even with raw capture in place.
🤖 Generated with Claude Code
What happened (2026-06-11, 08:57–16:04 UTC, all day)
gpt-5.4 via
https://llmapi.hk.lexmount.net/v1returned responses consisting of one complete AgentOutput JSON followed by a newline and a second JSON object on ~20–30% of calls. The two objects had independently mixed formats (compact+compact, pretty+compact, ...) — i.e. two independent generations concatenated by the gateway/upstream aggregation layer, not model repetition.browser-use's strict pydantic parse fails on these with:
Impact on the 6/11 bench runs (
/Users/zhu/lexmount/bench-experiment)20260611_16525020260611_18422820260614_063435/20260614_063552Retries (max 6) recovered most steps, so no mass aborts — but ~30% of steps burned budget/wall-clock/tokens producing no action, and browser-use usage tracking skips failed calls, so the 6/11 runs' reported token/cost undercount real spend ~1.4x. The 6/11 backend-comparison numbers (browser-use-cloud 53% vs lexmount 65%) are confounded by the incident and must not be used for backend ranking.
Why this issue exists
At the time we could not attribute the failures: the raw failing responses were discarded by the agent. PR #82 (commit 668598b, merged 2026-07-04) fixed that: failed/unparseable LLM calls now capture the raw response + usage into api_logs (
llm_failuresper step,llm_failures_unmatched.jsonfor orphans). OpenAI-compatible chat.completions path only (Azure responses-API/Anthropic/Google record raw_response=None).Next time the signature appears
llm_failures[].raw_responseshould show one complete JSON object followed by a second one after a newline, and the parse errorInvalid JSON: trailing characters.Known remaining gap: aggregate cost accounting in result.json metrics still excludes failed calls, so contaminated runs will still undercount spend even with raw capture in place.
🤖 Generated with Claude Code