Skip to content

Gateway double-JSON responses (2026-06-11 incident): two concatenated JSON objects per completion — attribute via PR #82 raw capture on next occurrence #86

Description

@Julia-Lex

What happened (2026-06-11, 08:57–16:04 UTC, all day)

gpt-5.4 via https://llmapi.hk.lexmount.net/v1 returned responses consisting of one complete AgentOutput JSON followed by a newline and a second JSON object on ~20–30% of calls. The two objects had independently mixed formats (compact+compact, pretty+compact, ...) — i.e. two independent generations concatenated by the gateway/upstream aggregation layer, not model repetition.

browser-use's strict pydantic parse fails on these with:

Invalid JSON: trailing characters at line N column 1

Impact on the 6/11 bench runs (/Users/zhu/lexmount/bench-experiment)

Run Backend Contamination
20260611_165250 browser-use-cloud 414/1373 steps (30%), 71/92 tasks
20260611_184228 browserless 271/1266 steps (21%), 44/92 tasks; task 149 aborted
20260614_063435 / 20260614_063552 lexmount / browserbase (identical LLM config, 6/14) zero occurrences

Retries (max 6) recovered most steps, so no mass aborts — but ~30% of steps burned budget/wall-clock/tokens producing no action, and browser-use usage tracking skips failed calls, so the 6/11 runs' reported token/cost undercount real spend ~1.4x. The 6/11 backend-comparison numbers (browser-use-cloud 53% vs lexmount 65%) are confounded by the incident and must not be used for backend ranking.

Why this issue exists

At the time we could not attribute the failures: the raw failing responses were discarded by the agent. PR #82 (commit 668598b, merged 2026-07-04) fixed that: failed/unparseable LLM calls now capture the raw response + usage into api_logs (llm_failures per step, llm_failures_unmatched.json for orphans). OpenAI-compatible chat.completions path only (Azure responses-API/Anthropic/Google record raw_response=None).

Next time the signature appears

  1. Confirm in api_logs: llm_failures[].raw_response should show one complete JSON object followed by a second one after a newline, and the parse error Invalid JSON: trailing characters.
  2. This is a gateway/upstream aggregation bug — escalate to the gateway admin (llmapi.hk.lexmount.net) with the raw_response excerpt and the time window. Not a bench bug, not model repetition; do not debug bench code.
  3. Mark any bench runs overlapping the incident window as contaminated for scoring/comparison purposes (steps and cost are inflated, usage undercounted).

Known remaining gap: aggregate cost accounting in result.json metrics still excludes failed calls, so contaminated runs will still undercount spend even with raw capture in place.

🤖 Generated with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions