fix(upstream): report a chat challenge as a retryable overload and stop hammering Qwen during one

[maxff77]

Refs #180.

What #180 is

Since 2026-09-23 11:35Z Qwen answers most chat generations with one frame and nothing else:

ret: ['FAIL_SYS_USER_VALIDATE', 'RGV587_ERROR::SM::哎哟喂,被挤爆啦,请稍后重试']

It happens on a bare 你好: no tools, no upload, one POST to /api/v2/chat/completions, answered in <1 s. Measured on a production deployment (209 accounts, one egress), 2026-09-23 11:35Z → 09-26 15:28Z:

	sends	challenged
before the cutoff (09-22..23)	68	0
after the cutoff, 07:00Z–21:59Z	~300	all
after the cutoff, 22:28Z–06:28Z	the rest	25 passed

• Same accounts pass at night and are challenged by day, so it is not the account.
• chats/new and the history upload/parse succeed on the same account and egress seconds before each challenge, so it is not the context size either.
• The window closes around Beijing's afternoon peak. Qwen's own text says "overloaded, retry later", but the frame's data.url is Aliyun's x5sec slider captcha (…/_____tmd_____/punish?x5secdata=…), and a logged-in browser passes at peak.
• Neither the client headers nor the egress IP decide it. At peak (2026-09-26 20:03–20:45Z, 120 × 你好, breaker off), headers aligned with the current web client (0.3.x: no version, browser timezone, source: h5, with and without Authorization: Bearer) and a direct egress without the proxy were answered 2–3 times in 30 each, the same as the current headers.
• It also arrives as a 200 text/html captcha page instead of the JSON frame (the form fix(images): repair the image/video path (TDZ crash) and stop reporting WAF/captcha as a successful generation #179 handled on the image path).

Rebased onto current main (#176, #178, #179): the challenge is recognised by #179's shared detectWafChallenge (JSON frame and HTML page); this PR only adds the strike count and the wording on top of it.

This PR does not make Qwen accept those requests. It makes the proxy report the refusal truthfully and stop making it worse.

What clients got before

path	before	after
/v1/chat/completions (plain and agent mode, stream and non-stream)	502 upstream_error, no Retry-After / 200 + error frame	503 upstream_unavailable + Retry-After
/v1/messages (stream and non-stream)	500 api_error / 200 + api_error event	529 overloaded_error + Retry-After
/v1/images/*, /v1/videos	502 (after #179), or a 200 whose "video" is a URL from the captcha page (t2v)	503 + Retry-After
any text surface, captcha page as text/html	2–3 sends, then 502 upstream_empty_output / in-stream "empty answer"	529 / 503 + Retry-After, one send

Retry-After is 30 s for a single challenge and up to 59 s while the breaker is open (the Anthropic and OpenAI SDKs ignore a Retry-After of 60 s or more).

Streams commit on Qwen's first valid frame. /v1/messages and the OpenAI agent stream used to write HTTP 200 + message_start (or the role chunk) before reading Qwen, so a challenge could only go out as an in-stream error event, which Claude Code shows as "API error · Retrying in 5s" while ignoring its retry_after. Both now commit on the first upstream frame that passed validation (usually response.created, so thinking still starts at once), on the first ping/heartbeat (cap), or when an attempt ends. A challenge or quota on the first frame is therefore a real status with Retry-After; any other early failure still commits and goes out in the stream as before. One reader, nothing buffered or replayed, one breaker strike per challenge. A challenge after the commit (e.g. on a correction retry) still goes out as an event/frame carrying retry_after.

The message was Agent 上下文可能过大或账号需要验证, which sends users after the wrong cause. It now says what Qwen said:

• 被挤爆啦 → Qwen 上游繁忙，触发风控验证（被挤爆啦），请稍后重试 / Qwen chat challenge: upstream busy, retry later
• a slider captcha (/punish?, Captcha #157) → … captcha required, retry later
• breaker open → … repeated upstream challenges, requests paused, retry later

Agentic clients (Claude Code) retried the 500 within a second. Each retry re-uploaded and re-parsed the history, so the parse rate limiter started answering 529 on top, and the parse budget of the egress was burnt on requests Qwen was going to refuse anyway.

Changes

• describeUpstreamFailure: a chat challenge is overloaded (529 / 503). The plain /v1/chat/completions handlers now build their error through upstreamErrorShape, like agent mode.
• Chat-challenge breaker (upstream-error.js), process-global on purpose (in the measured deployment every account leaves through one egress):
  • closed: each challenge is a strike; an answer (choices) clears strikes;
  • open: after 3 strikes in a row, sendChatRequest and the image/video path answer 529 / 503 before creating a chat or uploading history, for CHAT_CHALLENGE_BREAKER_SECONDS (default 60, 0 disables). An answer from a stream that was already running clears strikes but does not close it;
  • half-open: the first request after the cooldown goes out as the only probe and re-arms the window for everyone else; a challenge to it reopens the breaker, and the first answer from Qwen while it is in flight closes it (from the probe or from an older stream: telling them apart would mean tagging every stream).
  • image/video answers count as answers too (they clear strikes and can close a half-open breaker; t2v as soon as Qwen accepts the task, not after the video is polled to completion);
  • a probe that fails before Qwen answers or challenges it (no account, no chat_id, upload failure, an invalid request) hands the probe to the next request instead of blocking everyone for another window.
  • Logs a warn on open and an info on close. Clock injectable for tests, like the parse breaker.
• Image/video generation posts to the same chat endpoint: it now checks the breaker and answers 503 + Retry-After on the challenge in every form. t2v requests responseType: 'json', so a non-JSON body (the captcha page, or a data:-wrapped frame) arrived as a string that the JSON-only check dropped, and the page's first URL came back as the "video"; string bodies now go through fix(images): repair the image/video path (TDZ crash) and stop reporting WAF/captcha as a successful generation #179's raw-text parser. A challenge with a non-2xx status is counted once, not twice.
• Captcha page on the text path: sendChatRequest reads a 200 text/html body (capped at 256 KiB) and runs it through the same detector before handing on any stream, so it is one strike and a real 529/503 instead of an empty answer after 2–3 sends. A connection cut mid-page is still the challenge if the markers already arrived. Any other HTML body is handed on byte for byte.
• No account cooldown or warning for a challenge (removes recordChallenge / challengeCooldownUntil): it took healthy accounts out of rotation for 5 minutes. The single agent-mode account switch from fix(agent): handle mid-stream quota_limit and upstream_waf_challenge with auto account rotation in runOpenAIAgentTurn #166 stays, but on a chat challenge it keeps the history prefix instead of re-uploading it (Qwen refused before reading it; normal rotation already reuses the prefix across accounts). A quota switch still re-uploads.
• /v1/messages no longer forgets a reused history prefix on a chat challenge, so the client's retry does not re-parse the history.

Not covered:

• user-attached media is uploaded to OSS while the request is built, before sendChatRequest, so a request with images still uploads them while the breaker is open;
• a probe that reaches Qwen and ends with neither an answer nor a challenge (e.g. an empty stream) still costs one window;
• a challenge with a non-2xx status on the streamed image path (t2i/image_edit), or a captcha page cut mid-body there. Never seen: every challenge measured arrived whole, with HTTP 200.

Tests

• npm test gate: PASS, 1225 tests / 137 suites / 0 fail; bun run lint clean; bun run test:bun (real Bun HTTP against a fake upstream) now also asserts a first-frame challenge gives 529/503 + application/json + retry-after on /v1/messages and agent streams, and that the captcha page as text/html is a 529 after one send.

• tests/chat-challenge.test.js (new, 32): wire shape and both messages (also 被挤爆 without RGV587); captcha page as text/html (and cut mid-page) → one strike, no re-POST, no account penalty, other HTML replayed; t2v captcha page / data: frame / non-2xx → 503 with exactly one strike; an answered image closes a half-open breaker and breaks a run of strikes; a probe that fails before asking Qwen (text, or an image_edit without text) hands the probe on; t2v closes a half-open breaker when the task is accepted; every one of these fails with its fix removed; opens on the 3rd strike; an in-flight answer does not close it; exactly one probe after the cooldown, closed by its answer or reopened by its challenge; sendChatRequest refuses without touching the network; OpenAI non-stream 503 upstream_unavailable; /v1/messages keeps the prefix on a challenge and forgets it on other errors; image path 503.

• Updated: agent-account-failover.test.js (no cooldown; challenge switch keeps the prefix key, quota switch drops it), agent-protocol.test.js (503 upstream_unavailable + Retry-After instead of 502). Both reset the process-wide breaker in beforeEach.

Live check (2026-09-26, 17:04–17:28Z, challenge window, one egress)

• Plain 你好: 503 + Retry-After: 30, 30, then 60 when the breaker opened; the next request was refused in 6 ms with no upstream send (4 requests, 3 sends in the log).
• /v1/messages stream while open: real 529 overloaded_error (refused before headers are committed). With the breaker closed: 200 + overloaded_error event with retry_after.
• Half-open: a request while open was refused in 9 ms (Retry-After: 45); after the cooldown, of two concurrent requests one went out as the probe (challenged, breaker reopened for 60 s) and the other was refused in 11 ms.
• Not observed live: a probe being answered and closing the breaker (unit-tested). Qwen did answer one plain request at 17:26Z in the middle of the window, which cleared the strikes as designed.
• Lazy commit (19:55Z, at peak): /v1/messages stream:true → 529 application/json + Retry-After: 30, no event: lines; agent stream → 503 upstream_unavailable + Retry-After: 59; Claude Code waits the Retry-After between retries.
• Not observed live: the text/html form on the text path (Qwen sent only the JSON frame during these checks); covered by unit tests and the Bun smoke.

🤖 Generated with Claude Code